Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

esxtop command showing high %RDY values for virtual machines running on NUMA enabled ESX/ESXi 4.0 hosts (1037718)

Last Updated: 3/20/2024Categories: How toTotal Views: 7576Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

High %RDY time showing within the esxtop command on the ESX host for virtual machines on a NUMA machine

Under-commitment of CPUs on the ESX host. For example, the number of vCPUs configured for virtual machines on host is less than the number of physical CPUs on the ESX host

Very high imbalance in CPU utilization among nodes

Resolution

This issue has been resolved in ESX/ESXi 4.1 Update 2. For further information please see Resolved Issues section in the ESX/ESXi 4.1 Update 2 Release Notes.

For information pertaining to earlier releases of ESX/ESXi, please see the following:

Note: This article pertains exclusively to ESX/ESXi 4.0. For information about ESX/ESXi 4.1, see esxtop command showing high %RDY values for virtual machines running on NUMA enabled ESX/ESXi 4.1 hosts (1026063)

This occurs due to the architecture of the VMkernel NUMA scheduler and is dependant on the number of CPU cores that exist per NUMA node. If the virtual machine has a higher number of vCPUs than the number of cores in the NUMA node, then the virtual machine is not managed by the NUMA Scheduler and there are no benefits from NUMA locality. These virtual machines can therefore see higher memory access latencies.

This is expected behavior based on the current architecture of the scheduler. However, these points help to limit the impact of the issue:

Ensure that the virtual machine is configured to utilize a number of CPUs less than or equal to the number of cores per NUMA node. In ESX 4.0, if the number of vCPUs exceeds the size of a node, the NUMA scheduler does not manage the virtual machine. As long as the virtual machines are sized such that they are a whole multiple or divisor of the NUMA node size this will help with the number of virtual machines that we can power on and they will benefit from memory locality. For example, if 6 vCPU virtual machines are used when there are 6 cores assigned to each NUMA node, you can run up to at least 8 of those virtual machines (with 100% CPU utilization) without incurring substantial ready times on a 48 core server.

Note: In a DRS cluster, the virtual machines need to be sized appropriately for the whole cluster, as the virtual machines can be migrated between the hosts in the cluster. To make the sizing easier, it is a good idea to have systems with the same NUMA characteristics (mainly the number of cores per NUMA node) in the cluster.

Spread out the large vCPU virtual machines across the environment. The impact is reduced as there is less per host. The administrator should also monitor or utilize DRS to ensure that the distribution of large vCPU virtual machines is constant.

Size the virtual machines appropriately and ensure they have not been configured with an excessive number of vCPUs. If the application running in the system is not going to benefit from having multiple vCPUs, then do not configure them.

Disable NUMA. Disabling NUMA resolves the CPU ready time issue, but should be used as a last resort as the inter-node latencies among certain nodes may be high. Disabling NUMA is discouraged on servers with many NUMA nodes and the latency between the nodes might be significant. NUMA can be disabled by enabling Node Interleaving in the BIOS of the ESX host. High ready times are not seen in this configuration because the scheduler no longer takes NUMA locality into account, which means that there are no restrictions on where the virtual machines can run. When NUMA is enabled, the high ready times are as a result of trying to schedule the virtual machines to their local node to prevent any performance hit from remote locality.

Note: In internal performance evaluations, VMware has observed high performance degradation on disabling NUMA, particularly when memory intensive applications are running in the virtual machines.

Related Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

esxtop command showing high %RDY values for virtual machines running on NUMA enabled ESX/ESXi 4.1 hosts
esxtop command showing high %RDY values for virtual machines running on NUMA enabled ESX/ESXi 4.0 hosts

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESX 4.0.x

Take Our Survey

Support Assistant