Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926)

Last Updated: 3/21/2024Categories: TroubleshootingTotal Views: 14644Language:
English

8 like this article

Subscribe to get an email
newsletter with new updates

8 people found this helpful Please provide article feedback:

Purpose

Virtual machines depend on available host resources (CPU, Memory), and the guest operating system consumes those resources. A problem with resource availability or scheduling inside or outside the virtual machine may cause it to become unresponsive.

This article provides steps for using CPU performance metrics to determine whether a guest operating system is actually running, whether the virtual machine monitor (VMM) is running, or whether there is scheduling contention.

Note: This article is part of a series. For more information, see the parent article Troubleshooting a virtual machine that has stopped responding (1007819).

Resolution

Four virtual machine CPU performance metrics can be used together to gain insight into the responsiveness of a virtual machine or its Guest OS:

Run - Amount of time the virtual machine is consuming CPU resources.

Wait - Amount of time the virtual machine is waiting for a VMkernel resource.

Ready - Amount of time the virtual machine was ready to run, waiting in a queue to be scheduled.

Co-Stop - Amount of time a SMP virtual machine was ready to run, but incurred delay due to co-vCPU scheduling contention.

These performance metrics can be reviewed using the Performance tab in the vSphere Client or using the esxtop or resxtop command-line utilities. Choose the most appropriate method for your environment.

Reviewing performance metrics using the vSphere Client

For more information on using custom performance charts, see the Customizing Chart Views section of the Resource Management Guide or the View Advanced Performance Charts section of the vSphere Monitoring and Performance Guide.

Connect to vCenter Server or an ESX/ESXi host using the vSphere Client.

Select the target virtual machine in the inventory.

Click the Performance tab.

Click Chart Options to customize the performance chart.

Under the CPU heading, select Real-time.

Under the Chart Type heading, select Line Graph.

Under the Objects list, select the virtual machine by name.

Under the Counters list, select Co-stop, Run, Ready, and Wait.

Optionally, save the chart settings to make re-use easier.

Click OK.

Make note of the four metrics displayed. Each is measured in milliseconds.

Reviewing performance metrics using esxtop or resxtop

For more information on using esxtop or resxtop, see the Performance Monitoring Utilities: resxtop and esxtop section of the vSphere Monitoring and Performance Guide or Resource Management Guide.

Identify the host on which the unresponsive virtual machine is running:

Open the VMware vSphere Client and connect to your VMware vCenter Server or VirtualCenter server.

Select the virtual machine that is not responding.

Click the Summary tab, and identify the Host: value, indicating the host that has the running virtual machine registered to it.

Open a console session to the ESX/ESXi host where the virtual machine is running, or to the vMA, or to another location where the VMware Command-Line Interface (vCLI) is installed:

For VMware ESX, log in to the service console using SSH or directly at its terminal. For more information, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807).

For VMware ESXi 4.0 and 3.5, use Technical Support Mode. For more information, see Tech Support Mode for Emergency Support (1003677).

For VMware ESXi 4.1, use Technical Support Mode. For more information, see Using Tech Support Mode in ESXi 4.1 (1017910).

Start the esxtop or resxtop command:

esxtop

resxtop --server HostNameOrIPAddress [--username root]

Press c on your keyboard to display the CPU panel.

Press V (uppercase) on your keyboard to display only virtual machines.

Identify the virtual machine by its Name or World ID.

Press f on your keyboard to change the visible fields. Ensure that the CPU State Times are visible:

ID GID NAME NWLD %USED %RUN %SYS %WAIT %RDY %IDLE %OVRLP %CSTP %MLMTD %SWPWT
186 186 VMName 4 2.11 2.08 0.00 397.64 0.25 197.71 0.20 0.00 0.00 0.00

Note: The esxtop fields in ESXi 5.0 include an additional field called %VMWAIT in the CPU view. VMWAIT is only applicable to the vCPU Worlds of a virtual machine. VMWAIT doesn't sum up the idle time. It only explains how much a virtual machine is in the blocked state.

Make note of the four metrics displayed. Each is measured as a percentage of time: %RUN, %WAIT, %RDY, %CSTP.

Interpreting CPU performance metrics

Run, %RUN:

This value represents the percentage of absolute time the virtual machine was running on the system.

If the virtual machine is unresponsive, %RUN may indicate that the guest operating system is busy conducting an operation.

When %RUN is near zero and the virtual machine is unresponsive, it means that the virtual machine is idle, blocked on an operation, or is not scheduled due to resource contention. Look at other values (%WAIT, %RDY, and %CSTP) to identify resource contention.

When %RUN is near the value of the number of vCPUS x 100%, it means that all vCPUs in the virtual machine are busy. This is an indicator that the guest operating system may be stuck in a operational loop. To investigate this issue further, you may need to engage the appropriate operating system vendor for assistance in identifying why the guest operating system is using all of the CPU resources.

If you have engaged the guest operating system vendor, and they have determined that the issue is caused by the VMware Tools or the virtual machine hardware, it may be pertinent to suspend the virtual machine to collect additional diagnostic information.

Wait, %WAIT:

This value represents the percentage of time the virtual machine was waiting for some VMkernel activity to complete (such as I/O) before it can continue.

If the virtual machine is unresponsive and the %WAIT value is proportionally higher than %RUN, %RDY, and %CSTP, then it can indicate that the world is waiting for a VMkernel operation to complete.

You may observe that the %SYS is proportionally higher than %RUN. %SYS represents the percentage of time spent by system services on behalf of the virtual machine.

A high %WAIT value can be a result of a poorly performing storage device where the virtual machine is residing. If you are experiencing storage latency and timeouts, it may trigger these types of symptoms across multiple virtual machines residing in the same LUN, volume, or array depending on the scale of the storage performance issue.

A high %WAIT value can also be triggered by latency to any device in the virtual machine configuration. This can include but is not limited to serial pass-through devices, parallel pass-through parallel , and USB devices. If the device suddenly stops functioning or responding, it can result in these symptoms. A common cause for a high %WAIT value is ISO files that are left mounted in the virtual machine accidentally are either deleted or moved to an alternate location. For more information, see Deleting a datastore from the Datastore inventory results in the error: device or resource busy (1015791).

If there does not appear to be any backing storage or networking infrastructure issue, it may be pertinent to crash the virtual machine to collect additional diagnostic information.

Ready, %RDY:

This value represents the percentage of time that the virtual machine is ready to execute commands, but has not yet been scheduled for CPU time due to contention with other virtual machines.

Compare against the Max-Limited, %MLMTD value. This represents the amount of time that the virtual machine was ready to execute, but has not been scheduled for CPU time because the VMkernel deliberately constrained it. For more information, see the Managing Resource Pools section of the vSphere Monitoring and Performance Guide or Resource Management Guide.

If the virtual machine is unresponsive or very slow and %MLMTD is low, it may indicate that the ESX host has limited CPU time to schedule for this virtual machine.

Co-stop, %CSTP:

This value represents the percentage of time that the virtual machine is ready to execute commands but that it is waiting for the availability of multiple CPUs as the virtual machine is configured to use multiple vCPUs.

If the virtual machine is unresponsive and %CSTP is proportionally high compared to %RUN, it may indicate that the ESX host has limited CPU resources, simultaneously co-schedule all vCPUs in this virtual machine.

Review the usage of virtual machines running with multiple vCPUs on this host. For example, a virtual machine with four vCPUs may need to schedule 4 pCPUs to do an operation. If there are multiple virtual machines configured in this way, it may lead to CPU contention and resource starvation.

Action Plan

When using performance metrics to troubleshoot any issue, capture samples to a persistent location so they can be referred to later. For more information, see Using performance collection tools to gather data for fault analysis (1006797).

Depending on the nature of the performance metrics determined, it may be pertinent to either crash or suspend the virtual machine to collect additional troubleshooting information, or to investigate a resource constraint or other performance issue. For further information, see the Action Plan section of Troubleshooting a virtual machine that has stopped responding (1007819).

If %WAIT is relatively high and the virtual machine is unresponsive, but there are no backing storage or networking infrastructure problems, this indicates that the virtual machine may be blocked on some stuck operation. For more information, see Crashing a virtual machine on ESX/ESXi to collect diagnostic information (2005715).

If %RUN is relatively high and the virtual machine is unresponsive, this indicates that the guest operating system or virtual machine monitor is running very hot, possible indicating a runaway process. For more information, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831).

Otherwise, review the Hyperthreaded Core Sharing Modes section from the vSphere Resource Management Guide and the Performance Tuning Best Practices document to adjust your virtual machine distribution based on CPU contention, and review Impact of virtual machine memory and CPU resource limits (1033115).

Related Information

For more information, see:

Interpreting esxtop 3.5 Statistics

Interpreting esxtop 4.1 Statistics

Tech Support Mode for Emergency Support
Unable to connect to an ESX host using Secure Shell (SSH)
Using performance collection tools to gather data for fault analysis
Troubleshooting a virtual machine that has stopped responding
Deleting a datastore from the Datastore Inventory results in the error: device or resource busy
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
Impact of virtual machine memory and CPU resource limits
Troubleshooting ESX/ESXi virtual machine performance issues
Crashing a virtual machine on ESX/ESXi to collect diagnostic information
Suspending a virtual machine on ESX/ESXi to collect diagnostic information
応答しなくなった仮想マシンのトラブルシューティング： VMM とゲスト CPU 使用量の比較
对已停止响应的虚拟机进行故障排除：VMM 与客户机的 CPU 使用情况比较

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

Take Our Survey

Support Assistant