Posted on Thursday, April 17, 2008 5:30:00 PM
With the release of VMware ESX 3.5 (notice I am not using the word Server anymore abiding by the new naming convention! J) VMware expanded physical memory support on ESX hosts from 64GB with ESX Server 3.0 to 256 GB with ESX 3.5. So now I am at a client site deploying some really awesome HP Proliant DL580 G5 with a 128 GB Ram and ESX 3.5. Once the installation was completed and all the configuration in place I noticed that the available physical memory on this host was 64GB and that the host was able to see 128 GB but 64GB were blocked as if they were in use. See Figure 1
Figure 1
This presented somewhat of a dilemma and I started thinking maybe my license file was corrupted and it still thinks I have a 3.0 host and as a result it was not making more than 64GB available, well that theory quickly went to out the door when I rebuilt the license file. Then I was checking up and down with resource pools and making sure I did not set any crazy limitations, nothing there, google did not get me any results or maybe I was just not using the right keywords because later I found a thread on VMTN that addresses this issue but the title was a bit off. Anyway after spending a couple of hours with VMware support, we identified this as bug number 204347 and VMware supposedly says it has been fixed. I was also told this is only happening with some HP hardware and that a bios from HP would fix this. I did not accept this solution as I was already on the latest bios update from HP but I decided to call HP. We spent few hours on the phone and the tech support found nothing on this.
VMware support provided me with a fix that worked like a charm which is to enable support for 40 Bit MTRR or Memory Type Range Registers. By default this is disabled. To enable it you will need to go to Configuration Tab on your ESX 3.5 host and then Advanced Settings > VMkernel and locate the entry for VMkernel.Boot.force36BitMTRRMask. See Figure 2
Figure 2
After you uncheck this feature you will need to reboot your ESX host and VOILA! You now have access to the full 128 GB of memory. See figure 3.
Figure 3
Now a word of caution before you do this, some hardware systems that do not support this feature may not boot back up correctly after this change is made. In the event that should happen here is a process to back out of this configuration change:
- Boot the host in Safe Mode or Single User Mode
- Edit the file /etc/vmware/esx.conf
- Add the following line at the end of the file /vmkernel/force36BitMTRRMask = "TRUE"
- From a command prompt run esxcfg-boot -b
- Reboot the server
As far as I can tell from installing Update 1 for ESX 3.5 this option is still checked by default which means on some hardware such as the HP DL580 G5 which we were using and as of the writing of this article, this is a setting that needed to be unchecked before the full 128 GB memory was available. If you have had similar experience with different hardware please post and let us know.