Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Virtual machines stop responding when any LUN on the host is in an all-paths-down (APD) condition (1016626)

Last Updated: 10/9/2016Categories: Staff PicksTotal Views: 78918Language:
Chinese (Simplified)JapaneseFrenchSpanishEnglish

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Details

The issue addressed in this article occurs when a rescan is issued while an all-paths-down state exists for any LUN in the vCenter Server cluster. Therefore, a virtual machine on one LUN stops responding (temporarily or permanently) because a different LUN in the vCenter Server cluster is in an all-paths-down (APD) state.

These symptoms may indicate you have an all-paths-down state:

You may see intermittent Request timed out (out of 6 or 7 successful pings) while trying to ping the virtual machine.

When powering on a virtual machine with a raw device mapping (RDM) the progress bar stops at 50% and the virtual machine console becomes unresponsive at the VMware splash/loading/BIOS screen.

In vSphere 4.0, virtual machines drop packets intermittently.

Virtual machines drop ping packets.

Network becomes temporarily unresponsive.

In the vmkernel.log file, you see entries similar to:

NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa._______________" - failed to issue command due to Not found (APD)
NMP: nmp_DeviceUpdatePathStates: Activated path "NULL" for NMP device "naa.__________________".

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

All-Paths-Down State (APD)
The all-paths-down state is a condition where no working path exists to a storage device (LUN). These situations can cause an all-paths-down state to a LUN:

A hardware failure (permanent or transient)

Removal of a LUN

Rescanning
This article is concerned with the type of rescanning that can cause virtual machines on other LUNs to become unresponsive:

Configuration changes in ESXi/ESX 4.x involving VMFS (Virtual Machine File System)

Configuration of VMFS that involves creating, deleting, or increasing datastores can cause an automated rescan. vCenter Server issues a vCenter Server-wide rescan as part of the workflow to discover storage changes. This rescan allows automatic discovery to maintain a consistent view of storage across all hosts in the cluster. For example, to perform a rescan in ESX 4.0, use this command:

esxcfg-rescan -d vmhba#

Note: The command to perform a rescan on the vmhba using various methods, such as RCLI, vMA, and PowerCLI varies. For appropriate commands, refer the documentation.

Removal of a LUN

The removal of a LUN through an array-based administration action followed by a manual rescan can cause virtual machines on other LUNs to also become unresponsive.

Solution

The issue is resolved in ESXi/ESX 4.1 Update 1 and the fix has also been included with ESXi 5.0.

This issue is resolved in the patch release for ESX 4.0. For more information see, VMware ESX 4.0, Patch ESX400-200912401-BG: Updates vmkernel, vmklinux, tools, CIM, and perftools (1016291).

Notes:

With ESXi/ESX 4.1 Update 1 and ESXi/ESX 4.0 Update 3, you do not need to modify the advanced setting. Virtual machines that are not associated with the APD volume(s) do not become unresponsive on a rescan.

When unpresenting a LUN containing a datastore, follow the instructions in Removing a LUN containing a datastore from VMware ESXi/ESX 4.x (1029786). If the issue still persists, contact VMware Technical Support.

Workaround
ESXi/ESX 4.x can list all of the LUNs it detects, as well as the state of these LUNs. If none of the paths to a storage device are in the ACTIVE state, then ESXi/ESX considers the device to be in an all-paths-down state. If an all-paths-down state does exist, then this is likely the issue causing LUNs to be unresponsive, either for a limited period of time or permanently, when a rescan occurs. For more information, see Identifying disks when working with VMware ESX (1014953).

If virtual machines are not responding on an ESXi/ESX 4.0 host, determine if an all-paths-down condition exists by running the command:

# esxcfg-mpath --list-paths --device device_naa | grep state

or

# esxcfg-mpath --list-paths --device device_mpx | grep state

Where:

device_naa is the Network Addressing Authority (NAA) unique address for the full storage device

device_mpx is the identifier if a NAA ID is not available

Note: For information about using the command line with ESXi, see Tech Support Mode for Emergency Support (1003677).

Starting with ESXi/ESX 4.0 Update 1, you can set an advanced configuration option on all hosts in the vCenter Server cluster to reduce rescan times and to prevent virtual machines from not responding. By default this option is disabled.

Caution: Not every all-paths-down condition is permanent. Some all-paths-down conditions, such as those that occur briefly during a network re-configuration, are transient. Enabling this option can cause devices in a transient all-paths-down state to become unavailable. VMware recommends disabling this option after the rescan operation completes.

To enable this option, run the command:

# esxcfg-advcfg -s 1 /VMFS3/FailVolumeOpenIfAPD

To disable and reset to the default value without requiring downtime, run the command:

# esxcfg-advcfg -s 0 /VMFS3/FailVolumeOpenIfAPD

To check the value of this option, run the command:

# esxcfg-advcfg -g /VMFS3/FailVolumeOpenIfAPD

To minimize the amount of time that the virtual machines are unresponsive, apply these patches:

VMware ESX 4.0, Patch ESX400-200912401-BG: Updates vmkernel, vmklinux, tools, CIM, and perftools (1016291)

VMware ESXi 4.0, Patch ESXi400-200912401-BG: Updates Firmware (1016295)

Note: This does not apply for ESXi/ESX 4.0 Update 2 and 4.1 because the patch is integrated in these versions.

With ESX 4.1 Update 1 and ESX 4.0 Update 3, you no longer have to make the modification to the advanced setting. Virtual machines that are not associated with the APD Volume(s) do not become unresponsive upon a rescan.

For more information, see:

VMware ESX 4.0, Patch ESX400-200912401-BG: Updates vmkernel, vmklinux, tools, CIM, and perftools (1016291)

Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0 (2004684)

Additional Information
For translated versions of this article, see:

Español: Las máquinas virtuales dejan de responder cuando alguna unidad de almacenamiento LUN en el host está en estado APD (all-paths-down) (1031207)

Português:

日本語: ホストのいずれかの LUN が all-paths-down 状態 （APD） になった場合、仮想マシンが応答を停止する (1033174)

Français: Les machines virtuelles ne répondent plus quand aucun LUN sur l'hôte est en état all-paths-down (APD) (1038305)

Deutsch: Virtuelle Maschinen antworten nicht mehr, wenn eine LUN des Host im all-path-down (APD) Modus ist (2000677)

简体中文: 主机上的任意 LUN 处于全部路径异常 (APD) 状况时，虚拟机停止响应 (2078036)

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseFrenchSpanishEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

VMware ESXi 4.1.x Installable

VMware ESXi 4.1.x Embedded

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESX 4.1.x

VMware ESX 4.0.x

view all

Take Our Survey

Support Assistant