Custom screen resolution for MAC OS X guests in VirtualBox
Step 1 - Selecting the correct resolution
In order to make sure the custom screen resolution works on all versions of VirtualBox, you will need to do a little math to figure out the correct screen resolution first. Let's say you want to run MAC OS X guest in full screen on your 15.6" notebook, then your desired resolution is 1366x768. However, you need to 'round down' both the height and width to the nearest number divisible completely by 8. So for the above said configuration, our screen resolution comes out to be 1360x768 (both numbers completely divisible by 8).
Before moving on, we add the color depth to this number (like 16bit, 32bit), so our correct screen resolution becomes "1360x768x32".
Step 2 - Tuning virtual machine settings
For this step, you will need to either edit the virtual machine configuration file by hand (if you can work with xml files easily), or use the 'vboxmanage' commandline, whichever you find easier. Both methods are described below:
Editing virtual machine configuration file
Make sure the machine is powered off (not in saved state). Open the virtual machine configuration file (which has extension .vbox and located in the folder where your virtual machine resides [you can find this path from VirtualBox settings dialog]), in a text editor, and after the first few lines, you will see the <ExtraData> node. You will need to enter the following two entries inside this node.
<ExtraDataItem name="CustomVideoMode1" value="1360x768x32"/>
<ExtraDataItem name="GUI/CustomVideoMode1" value="1360x768x32"/>
Using vboxmanage commandline
Open a command prompt and navigate to the folder where VirtualBox is installed. Now run the following two commands.
vboxmanage setextradata "MAC OS X" "CustomVideoMode1" "1360x768x32"
vboxmanage setextradata "MAC OS X" "GUI/CustomVideoMode1" "1360x768x32"
In the above commands, "MAC OS X" is the machine name. You should replace it with the correct name of your machine as seen in the VirtualBox interface.
Step 3 - Editing Guest OS boot configuration
After completing the above two steps, start your MAC OS X virtual machine. When you are at the desktop, open a terminal window from Go -> Utilities -> Terminal.
Depending on which text editor you are comfortable with, you will need to edit two files (both of these are xml too, so it's about time you learn about xml if you already haven't). We use the 'pico' editor combined with sudo command here.sudo pico /Library/Preferences/SystemConfiguration/com.apple.boot.plist
Try to find the <key> nodes with data 'Kernel Flags' and 'Graphics Mode'.
You will need to replace the existing <key> and associated <string> node with the following, or add them if they are already not there.
<key>Kernel Flags</key>
<string>"Graphics Mode"="1360x768x32"</string>
<key>Graphics Mode</key>
<string>1360x768x32</string>
When you are done editing, save the file and exit. Now do the exactly same with the following file:
sudo pico /Extra/com.apple.boot.plist
Test your custom screen resolution
Reboot the virtual machine and once you are at the desktop, it should be in the custom screen resolution that you just wanted. Press the VirtualBox HotKey "Host + F" to go fullscreen and enjoy MAC OS X in its entirety.