Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Troubleshooting a virtual machine that has stopped responding (1007819)

Last Updated: 10/24/2023Categories: TroubleshootingTotal Views: 142359Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglish

67 like this article

Subscribe to get an email
newsletter with new updates

67 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Symptoms

Virtual machines can become unresponsive/freeze/hang in the same way as a physical server for various reasons, such as:

Virtual machines are unresponsive to your actions or appear to be frozen/hang.

Tasks performed on the virtual machine fail, timeout, or do not start.

Virtual machine does not produce network or disk traffic.

Virtual machine does not allow access via RDP, vCenter Server virtual machine console screen, or other connection methods.

Virtual machines are unreachable over the network.

Suspending the virtual machine through vmdumper/vm-support/GUI does not initiate the task and fails . (2005831)

Activity of the VMware tools inside the guest rules out the virtual machine networking or MKS (Mouse,Keyboard,Screen) issues . (749640)

The VSCSI I/O warning counters in the esxtop indicates inactivity.

Virtual machines may report an invalid state

Crashing a virtual machine on ESX/ESXi to collect diagnostic information. (2005715)

The screen is frozen, and no actions are possible.

The affected virtual machines have a black console with an error.

You see the error:

Virtual machine screen is black and does not refresh.

You may see one or more of the following errors:

/init: /init: 151: Syntax error: 0xforce=panic
Kernel panic - not syncing: Attempted to kill init!
PAGE_FAULT_IN_NONPAGED_AREA
Error codes 6005, 6008 in Windows Event Viewer.

Purpose

This article provides steps to isolate possible causes of a vSphere virtual machine becoming unresponsive.

An unresponsive virtual machine does not respond to any connection attempts and may be unable to respond to any attempts to power cycle it. There are a variety of reasons a virtual machine can end up in an unresponsive state. This article enables you to identify and resolve these common causes, and, when resolved, return the virtual machine to an operational state.

It is possible to hard power off a virtual machine without troubleshooting the cause, but this will prevent collection and analysis of information which could assist with determining the root cause of the outage. For more information about shutting down the virtual machine, see Powering off a virtual machine on an ESXi host (1014165) and Powering off an unresponsive virtual machine on an ESX host (1004340).

This article assumes that the issue is currently occurring. If troubleshooting an issue that occurred in the past, some required information may be unavailable.

Cause

These situation can be caused by:

Virtual Machine backups and schedules may create heavy I/O load related tasks, may lead to unresponsive issues.

Virtual machine may become unresponsive when the hard disk runs out of space.

Guest Operating system on the Virtual machine may cause VM to appear unresponsive.

The virtual machine disk  controllers not set as per best practices.

If your network contains a firewall between the ESXi/ESX host and the vSphere Client.

Resolution

The services a virtual machine provides may become unresponsive or unreachable due to a number of causes, including problems with the applications or guest OS within the virtual machine, problems with the virtual machine monitor or virtual devices, resource contention on the host, or issues with underlying storage or networking infrastructure.

If the guest OS is producing any activity, it is successfully running. In this case, unresponsiveness is likely due to a connectivity problem or resource contention, or is specific to a higher-level component such as an application or service running within the guest OS.

Validate the scope

It is important to have accurate symptoms and an understanding of the scope of a problem. To confirm the scope of the problem, work through these checks:

Confirm that the virtual machine is actually unresponsive. It is possible that the virtual machine is not responding via one interface, but is functioning correctly on others. For more information on testing whether a virtual machine is genuinely unresponsive, see Confirming whether virtual machine is unresponsive (1007802).

If a virtual machine is responsive, but performing poorly, see Troubleshooting ESX virtual machine performance issues (2001003).

Verify that the virtual machine is powered on. If the virtual machine has been powered off unexpectedly, power it back on and then troubleshoot the cause of the unexpected shutdown. For more information, see:

Powering on an ESX/ESXi host's virtual machine (1003738)

Determining why a virtual machine was powered off or restarted (1019064)

Note: If a virtual machine is powered off and cannot be powered back on, see Troubleshooting a virtual machine that is unable to power on (2001005).

Determine whether this issue is affecting multiple virtual machines or just one. If multiple virtual machines are affected, consider the similarities between the affected virtual machines when attempting to narrow the potential scope. In particular, focus on shared infrastructure which the group of affected virtual machines depend on, and whether all virtual machines depending on that common infrastructure are affected. For more information, see Assessing commonalities of an outage affecting multiple virtual machines (1019000).

Determine whether the guest OS is responsive to interaction at the virtual machine console. If an issue has been isolated to the guest OS or applications within the virtual machine, and the guest OS is responsive at the console, interact with the guest OS at the console to address the problem. For more information, see Troubleshooting virtual machine network connection issues (1003893).

Determine whether the guest OS or its application services are responsive to interaction via the network. If the guest OS or services respond to network communication but the console is unresponsive or non-functional, see Troubleshooting virtual machine console and MKS issues in the vSphere Client (749640) or Ensuring that a virtual machine is not inaccessible due to a VMware vCenter or VirtualCenter issue (1007808).

Determine whether the guest OS has reported any critical errors to the console, and is sitting in a halted state. For more information, see Identifying critical Guest OS failures within virtual machines (1003999).

Determine whether the ESX/ESXi host is unresponsive too. If the host is unresponsive as well, the scope is larger than initially assumed. For more information, see Determining why an ESX/ESXi host does not respond to user interaction at the console (1017135).

Identify the cause

At this point, you have established that one or more virtual machines are unresponsive at both the virtual console and via the network. The host itself is responsive. A problem may exist with resource accessibility or contention, or with underlying storage or networking infrastructure.

To identify the cause:

Determine whether the problem is triggered by an operation or task being performed on the virtual machine. For example, snapshot and vMotion operations both stun a virtual machine for brief periods of time while memory state is copied across the network or to disk. For more information, see Taking a snapshot with virtual machine memory stuns the virtual machine while the memory is written to disk (1013163).

Some common configuration errors can lead to a virtual machine becoming unresponsive, such as while waiting for a resource. Review the virtual machine and host configuration. For more information, see:

Common ESX/ESXi host configuration issues which can cause virtual machines to become unresponsive (1007813)

Common ESX/ESXi virtual machine configuration issues which can cause virtual machines to become unresponsive (1007814)

Virtual machines depend on functional backing infrastructure. If there is an issue with the backing storage or networking infrastructure which the virtual machine depends on, the virtual hardware which a virtual machine presents to the guest OS may be impacted. Address the underlying storage or networking issue. For more information, see:

ESX Server virtual machines stop responding due to shared storage connectivity issues (1004144)

Verifying that ESX/ESXi virtual machine storage is accessible (1003751)

Troubleshooting virtual machine network connection issues (1003893)

Virtual machines depend on available host resources (CPU, Memory), and the guest OS consumes those resources. A problem with resource availability or scheduling inside or outside the virtual machine may cause it to become unresponsive. The virtual machine may also be blocking on unavailable resources or spinning at 100% vCPU utilization. For more information, see Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison (1017926).

Action Plan

At this point, you have established that the host running the virtual machine(s) is both responsive and not encountering any shared storage or networking infrastructure issues. The guest OS has not failed with a critical error, but remains unresponsive at the virtual machine console and via the network.

Take action to recover or collect information about the unresponsive virtual machine based on the architectural layer which is suspect:

If an issue has been isolated to the guest OS, or the %RUN is relatively high, but the virtual machine monitor is functioning correctly, move investigation to within the virtual machine's guest OS or applications. A guest OS can become unresponsive inside a virtual machine in the same way it can on physical hardware. For more information, see Troubleshooting unresponsive guest operating system issues (1007818).

Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797).

Attempt to manually induce a panic of the kernel inside the guest OS to collect additional information about its internal state. For more information, see:

If useful diagnostic information is produced by the guest OS in response to one of these events, engage the guest OS vendor to investigate further.

Crashing a virtual machine on ESX/ESXi to collect diagnostic information (2005715)

Microsoft article 927069: How to generate a complete crash dump file or a kernel crash dump by using an NMI on a Windows-based system

Linux Documentation Project article: Magic SysRq key

If step 2 does not produce useful information, suspend the virtual machine to collect information about its internal state and open a case with VMware Support. For more information, see:

Note: If the virtual machine cannot be suspended because another management task is in progress, see Collecting information about tasks in VMware ESX and ESXi (1013003) and Restarting the Management agents on an ESX or ESXi Server (1003490). If attempts to suspend the virtual machine fail and no management task appears to be present, skip to the next section and attempt to crash the virtual machine.

Suspend the virtual machine and collect the suspend state files.

For virtual machines with hardware versions up until 10, this will be the file with the suffix .vmss.

For virtual machines with hardware version 11 or newer, there will be 2 files, one with the suffix .vmss, and another one with the suffix .vmem. Collect both.

For more information, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831).

Collect logs from the host running the virtual machine. For more information see Collecting diagnostic information for VMware products (1008524).

Power the virtual machine back on, then reset it.

Engage VMware Support, providing the information collected in steps 1, 3a and 3b. For more information, see How to File a Support Request.

If an issue has been isolated to the virtual machine monitor, or the %WAIT is relatively high, or attempts to suspend the virtual machine have failed, collect performance data and forcefully crash the virtual machine to collect additional information about its internal state.

Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797).

Crash the virtual machine to collect information about its internal state. For more information, see Crashing a virtual machine on ESX/ESXi to collect diagnostic information (2005715).

Note: If attempts to crash the virtual machine fail, skip to the next section and attempt to crash the host.

Engage VMware Support, providing the information collected in steps 1 and 2. For more information, see How to File a Support Request.

If an issue has been isolated to the virtual machine monitor, but attempts to suspend or crash the virtual machine fail, this reflects a problem with the VMkernel. Collect a log bundle from the host, evacuate all unaffected virtual machines from the host, and use an NMI to intentionally generate a purple diagnostic screen.

Collect performance data while the problem is happening. For more information, see Using performance collection tools to gather data for fault analysis (1006797).

Move all unaffected virtual machines off of the host using vMotion. If possible, use Maintenance Mode to prevent additional virtual machines from being started on the host.

Configure the host to panic on receiving a non-maskable interrupt, and then issue an NMI to trigger a panic. For more information, see Using hardware NMI facilities to troubleshoot unresponsive hosts (1014767).

After the host has generated a purple diagnostic screen and completed dump of diagnostic information, take a screenshot or photograph of the console and restart the host.

Collect diagnostic information from the host. For more information, Collecting diagnostic information from an ESX or ESXi host that experiences a purple diagnostic screen (1004128).

Engage VMware Support, providing the information collected in steps 1, 4 and 5. For more information, see How to File a Support Request.

Related Information

Restarting the Management agents in ESXi
Powering on an ESX/ESXi host's virtual machine
Verifying that ESX/ESXi virtual machine storage is accessible
Troubleshooting virtual machine network connection issues
Identifying critical Guest OS failures within virtual machines
Collecting diagnostic information from an ESX or ESXi host that experiences a purple diagnostic screen
ESX Server virtual machines stop responding due to shared storage connectivity issues
Powering off an unresponsive virtual machine on an ESX host
Using performance collection tools to gather data for fault analysis
Confirming whether a virtual machine is unresponsive
Ensuring that a virtual machine is not inaccessible due to a VMware vCenter or VirtualCenter issue
Common ESX/ESXi host configuration issues which can cause virtual machines to become unresponsive
Common ESX/ESXi virtual machine configuration issues that can cause virtual machines to become unresponsive
Collecting diagnostic information for VMware products
Collecting information about tasks in VMware ESXi/ESX
Virtual machine becomes unresponsive or inactive when taking a snapshot
Unable to power off the virtual machine in an ESXi host
Using hardware NMI facilities to troubleshoot unresponsive hosts
Determining why an ESX/ESXi host does not respond to user interaction at the console
Troubleshooting a virtual machine that has stopped responding: VMM and Guest CPU usage comparison
Assessing commonalities of an outage affecting multiple virtual machines
Determining why a virtual machine was powered off or restarted
Troubleshooting ESX/ESXi virtual machine performance issues
Troubleshooting a virtual machine that is unable to power on
Crashing a virtual machine on ESX/ESXi to collect diagnostic information
仮想マシンが応答しなくなった場合のトラブルシューティング
Solução de problemas em uma máquina virtual que parou de responder
Resolución de problemas de una máquina virtual que no responde
对已停止响应的虚拟机进行故障排除
Troubleshooting virtual machine console and MKS issues in the vSphere Client

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglish

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

Related Versions:

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant