Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage (1004424)

Last Updated: 4/14/2015Categories: TroubleshootingTotal Views: 126374Language:
Chinese (Simplified)JapaneseEnglish

2 like this article

Subscribe to get an email
newsletter with new updates

2 people found this helpful Please provide article feedback:

Symptoms

An ESXi/ESX 3.5/4.0 host has more than 4 terabytes of virtual disks (.vmdk files) open.

After virtual machines are failed over by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:

vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.

In the /var/log/messages or /var/log/vmkernel.log file, you see entries similar to:

vmkernel: cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
vmkernel: cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)
cpu4:1959755)WARNING:Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
cpu4:1959755)WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x418009533c50
cpu7:5134)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

cpu29:8350374)WARNING: J3: 1666: Failed to reserve space for journal on 4f8444cf-ae524ea8-a556-5cf3fcdad836 : Out of memory
cpu29:8350374)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 4f8444cf ae524ea8 f35ca556 36d8dafc 0 0 0 0 0 0 0

Adding a VMDK to a virtual machine running on an ESXi/ESX host where VMFS-3 heap is maxed out fails.

When attempting to migrate a virtual machine to another host, you see the error:

Failed to initialize migration at source

When you try to manually power on a migrated virtual machine, you see the error:

The VM failed to resume on the destination during early power on.
Reason: 0 (Cannot allocate memory).
Cannot open the disk 'vmdk_location' or one of the snapshot disks it depends on.

A virtual machine fails to power on and you see this error in the vSphere Client:

An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)

A similar error appears if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which VMFS-3 heap is maxed out.

Cloning a virtual machine using the vmkfstools -i command fails and you see the error:

Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)

Cold storage migration fails with the error:

Could not complete network copy for file

In the /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log file, you see messages similar to:

vmx| FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
vmx| AIOGNRC: Failed to open '/vmfs/volumes/xxxx-flat.vmdk' : Cannot allocate memory (c00000002) (0x2013).
vmx| DISKLIB-VMFS : "/vmfs/volumes/xxxx-flat.vmdk" : failed to open (Cannot allocate memory): AIOMgr_Open failed. Type 3
vmx| DISKLIB-LINK : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
vmx| DISKLIB-CHAIN : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/xxxx.vmdk' with flags 0xa Cannot allocate memory (786441).
vmx| DISK: Cannot open disk "/vmfs/volumes/xxxx.vmdk": Cannot allocate memory (786441).
vmx| Msg_Post: Error
vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/xxxx.vmdk' or one of the snapshot disks it depends on.
vmx| [msg.disk.configureDiskError] Reason: Cannot allocate memory.

You see virtual machines run more slowly than usual or become unresponsive.

The vmkernel stops logging in the appropriate /var/log file of the ESX/ESXi host.

Purpose

This article addresses the issue of running out of heap space on an ESXi/ESX host when a large quantity of virtual disk space is active.

Resolution

Default VMFS heap size

In ESXi and ESX 3.5/4.0, the default heap size is set to 16 MB, allowing a maximum of 4 TB of open virtual disk capacity on a single ESXi/ESX host.

In ESXi/ESX 4.1 and early versions of ESXi 5.x, the default heap size is 80 MB, which allows for 8 TB of open virtual disk capacity on a single ESXi/ESX host.

In ESXi 5.0 Patch 5, the default heap size was increased to 256 MB with a dynamic maximum of 640 MB, which allows for 60 TB of open virtual disk capacity on a single ESXi/ESX host.

Maximum VMFS heap size

In ESXi/ESX 4.x and earlier releases, the maximum heap size is 128 MB. This allows a maximum of 32 TB of open storage per ESXi/ESX host.

In earlier versions of ESXi 5.x, the maximum heap size is 256 MB. Allowing a maximum of 25 TB of open storage per ESXi host.

In ESXi 5.0 Patch 5 and ESXi 5.1 Update 1, the maximum heap size increased to 640 MB.

The net effect of this change is that the ESXi/ESX kernel requires a small amount of additional memory (such as the 256 MB we used to get a maximum of 25 TB in the example above) for the larger heap, but it allows virtual machines with more than 4 TB (ESXi/ESX 4.x) or 8 TB (ESXi 5.0) of virtual disk to be addressed.

Note: ESXi 5.5 includes a much improved heap eviction process so that there is no need for a larger heap size that consumes memory. vSphere 5.5 supports a maximum heap size of 256 MB and enables hosts to access all address space of a 64 TB VMFS datastore.

For more information, see Understanding the VMFS heap eviction process in ESXi 5.5 (2077898) and What's New in VMware vSphere 5.5 Platform.

To adjust the maximum values of VMFS heap:

Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.

Under the Configuration tab, click Advanced Settings.

Click VMFS3.

Update the field in VMFS3.MaxHeapSizeMB.

Reboot the ESXi/ESX host for the changes to take affect.

Minimum VMFS Heap Size

ESXi 5.0 Patch Patch 5 (1024429) and ESXi 5.1 Update 1 (1065491) introduce another option to reserve up to 256 MB of VMFS heap up front for the boot process. This is done by modifying the VMFS3.MinHeapSizeMB option.

To adjust the minimum values of VMFS heap:

Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.

Under the Configuration tab, click Advanced Settings.

Select VMFS3.

Update the field in VMFS3.MinHeapSizeMB.

Reboot the ESXi/ESX host for the changes to take affect.

Command Line Usage

The esxcfg-advcfg command can be used to set/get the maximum heap size for all VMFS file systems.

To check the exiting value, run the command:

esxcfg-advcfg -g /VMFS3/MaxHeapSizeMB

Sample Output:

Value of MaxHeapSizeMB is 256

To change the heap size, run the command:

esxcfg-advcfg -s <heap_value> /VMFS3/MaxHeapSizeMB

For example:

esxcfg-advcfg -s 640 /VMFS3/MaxHeapSizeMB

Note: The advanced configuration parameter, /VMFS3/MaxHeapSizeMB, is used by both VMFS3 and VMFS5 filesystems.

Default, Min, and Max Table

This table outlines the default, minimum and maximum configurable heap size and open VMDK storage for the various builds of ESXi/ESX:

Version/build

Default heap amount

Default allowed open VMDK storage per host

Minimum heap amount

Maximum heap amount

Maximum value for minimum heap size

Maximum open VMDK storage per host

ESXi/ESX 3.5/4.0

16 MB

4 TB

N/A

N/A

N/A

N/A

ESXi/ESX 4.1

80 MB

8 TB

N/A

128 MB

128

32 TB

ESXi 5.0 Update 2 (914586) and earlier

80 MB

8 TB

N/A

256 MB

255

25 TB

ESXi 5.0 Patch 5 (1024429) and later

256 MB

60 TB

256 MB

640 MB

255

60 TB

ESXi 5.1 Patch 1 (914609) and earlier

80 MB

8 TB

N/A

256 MB

255

25 TB

ESXi 5.1 Update 1 (1065491) and later

256 MB

60 TB

256 MB

640 MB

255

60 TB

Notes:

Open VMDK storage refers to any open virtual disks, including thin disks.

Although VMFS3.MinHeapSizeMB can only be set to a maximum value of 255 (256MB), the heap grows to its maximum size (640 MB), as required.

Although the examples here reference VMFS-3, this issue can also affect VMFS-5 datastores. The variables VMFS3.MaxHeapSizeMB and VMFS3.MinHeapSizeMB also apply to VMFS-5 volumes.

For more information, see VMware ESXi 5.1 U1 Release Notes.

Related Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

Correlating build numbers and versions of VMware products
Understanding the VMFS heap eviction process in ESXi 5.5 and later
ESXi 5.5 における VMFS ヒープの扱いについて
まとめて 4 TB または 20 TB の仮想ディスク ストレージを使用する仮想マシンをホストしている場合に ESXi/ESX ホストが VMFS ヒープの警告を報告する
在托管集中使用 4TB 或 20TB 的虚拟磁盘存储的虚拟机时，ESXi/ESX 主机会报告 VMFS 堆警告

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

VMware ESXi 4.1.x Installable

VMware ESXi 4.1.x Embedded

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESXi 3.5.x Installable

VMware ESXi 3.5.x Embedded

VMware ESX Server 3.5.x

VMware ESX Server 3.0.x

VMware ESX 4.1.x

VMware ESX 4.0.x

view all

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant