Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

VM nvdimm config options for NUMA (78094)

Last Updated: 8/7/2023Categories: InformationalTotal Views: 7309Language:
Chinese (Simplified)JapaneseEnglish

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Details

This article provides best practices for PMem when non-deterministic NVDIMM performance is caused by sub-optimal NUMA locality.

By default, PMem allocation for virtual machines NVDIMMS does not consider NUMA. This can result in the VM running on a different NUMA node than PMEM is allocated from, which causes NVDIMM accesses from the VM to be remote, resulting in sub-optimal performance.

Once PMem has been initially allocated for a VM, it remains on its NUMA node. The virtual machine NUMA clients will be initially placed and migrated according to constrains by the NUMA scheduler, which does not include PMem locality. Baring any contention or other workloads, this might be round-robin on every VM power on.

Solution

To affinitize a virtual machines vCPUs and memory to a physical NUMA node, you can set the VM Advanced Option numa.nodeAffinity = X, where X equals the physical NUMA (starting at 0). Affinitizing the VM to the same node as the PMem allocation will result in optimal performance.

To control the locality of NVDIMM devices, ESXi 6.7 EP14 (15820472) and 7.0b/P01 (16324942) introduced NVDIMM nodeAffinity.

For a host with 2 physical NUMA nodes and a wide VM across both, you might want to set a direct affinity for the vCPUs in vNUMA node 0 to pNUMA node 0, and vice versa for vNUMA node 1. For that, set the VM Advanced Option sched.vcpuX.affinity = Y, where X is the vCPU and Y the PCPU. Note that from 7.0 U2m onwards, ESXi supports per vNUMA node affinity. sched.nodeX.affinity = Z, where X is the vNUMA node and Z the physical NUMA node. To configure the NVDIMM affinity, set nvdimm0:X.nodeAffinity = Z, where X is the device number and Z the PMem node used for allocation (or nodes comma separated if multiple). A VM on ESXi 7.0 U2 or later that is wide over 2 physical NUMA nodes and NVDIMM devices on two PMem NUMA nodes, might have the following configuration for deterministic locality from each NUMA client to the PMem allocation:

sched.node0.affinity = 0
sched.node1.affinity = 1
nvdimm0:0.nodeAffinity = 0
nvdimm0:1.nodeAffinity = 1

As with all manual affinities, care in placement and operation is necessary.

While unrelated to locality, a further optimization for fully deterministic performance is to pre-allocate the entire NVDIMM during VM power on. This prevent runtime allocation overhead when the VM uses an NVDIMM page for the first time. While the VM is powered off, set they VM Advanced Option sched.pmem.prealloc = TRUE.

Related Information

Associate Virtual Machines with Specified NUMA Nodes

Set Advanced Virtual Machine Attributes

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere

Related Versions:

VMware vSphere 7.0.x

VMware vSphere 6.7.x

Take Our Survey

Support Assistant