I've successfully installed CubeCart and now in the process of Build Configuring File
part. It's asking me for Database Hostname, Database name, database username/password,
and also database prefixe. Do I go into my control panel and set up a new MySQL?

Yes. And be sure to assign a User to the newly created database, otherwise the script that builds the tables won't have permission to do the CREATE sql command.

What would I type as the Database Hostname and Database name in the build Config File step 3?

I would suggest that you contact your hosting company for this information if you are incapable of obtaining it yourself.

When you create the MyQL database from your hosting provider's control panel (or whatever you choose to use if doing this on your own local server), the hostname will be given to you at that time and the database name could very well be something you specify at the time you create it.