Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Active/Passive or ALUA based storage devices see APD events during storage controller fail-over on ESXi 6.7 & 7.0 hosts (67006)

Last Updated: 4/18/2024Categories: TroubleshootingTotal Views: 51464Language:
Chinese (Simplified)JapaneseEnglish

16 like this article

Subscribe to get an email
newsletter with new updates

16 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Details

Symptoms:
ESXi 6.7 & 7.0 hosts with active/passive or ALUA based storage devices may see premature APD events during storage controller fail-over scenarios

Purpose:
If you are using ESXi 6.7 & 7.0 hosts with a ALUA based storage array managed by the VMware NMP stack (NFS arrays are excluded), then it is recommended to disable action_OnRetryErrors.

Solution

Use one of these methods to resolve the issue:

Disable OnRetryErrors using an SATP claim rule (Preferred method)
Disable OnRetryErrors for existing devices (quick fix - non-preferred method)

Disable OnRetryErrors using an SATP claim rule (Preferred method)

Create an SATP claim rule to change the default behavior of action_OnRetryErrors to "off". This setting will need to be applied to every ESXi 6.7 host that has ALUA based storage array Luns mapped to it.

Add the claimrule with an option to disable OnRetryErrors.

esxcli storage nmp satp rule add -V COMPELNT -P VMW_PSP_RR -s VMW_SATP_ALUA -o disable_action_OnRetryErrors

Reload the claim rules to enforce the change:

esxcli storage core claimrule load

Note: The change will take effect immediately for any new LUNs presented but requires a reboot in order to reclaim existing storage devices with the new ruleset.

List the claimrule table to confirm the changes are there:

esxcli storage nmp satp rule list |grep -i comp

Example output:

VMW_SATP_ALUA    COMPELNT disable_action_OnRetryErrors  user   VMW_PSP_RR

To verify the setting against a device, capture the naa id of the device and run this command:

esxcli storage nmp device list | grep -A2 naa.deviceIDhere

For example:

Device Display Name: COMPELNT Fibre Channel Disk (naa.60000000000000000000000000000000)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=61445,TPG_state=AO}{TPG_id=61446,TPG_state=AO}}

Disable OnRetryErrors for existing devices (quick fix - non-preferred method)

Disable this setting for existing devices on a live host without a reboot.

Run the following to get a list of all ALUA based storage array and validate the current setting.

esxcli storage nmp device list | grep -A2 COMPELNT

Example output:

Device Display Name: COMPELNT Fibre Channel Disk (naa.60000000000000000000000000000000)

Storage Array Type: VMW_SATP_ALUA

Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=61445,TPG_state=AO}{TPG_id=61446,TPG_state=AO}}

Extract the naa device names where appropriate and run the following to change OnRetryErrors to "off" on a per-device per-host basis.

esxcli storage nmp satp generic deviceconfig set -c disable_action_OnRetryErrors -d naa.xxx

Once all devices are configured, go back and perform step 2a to validate the change worked.

Repeat steps 1-3 for the remaining hosts.

Note: No reboot is required for these changes to take effect. additionally, the SATP claimrule workaround will overwrite this one upon reboot.

Related Information

A feature called "action_OnRetryErrors" was introduced at the storage array type plugin (SATP) level of ESXi 6.0 to enable automatic path failover capabilities in response to a failure of health probing on a storage path. This feature is disabled by default in ESXi 6.0 and 6.5 but in ESXi 6.7 the default SATP setting for "action_OnRetryErrors" was changed from "off" to "on" for all presentation models (EXPLICIT ALUA ONLY/IMPLICIT ALUA ONLY/EXPLICIT AND IMPLICIT ALUA). But we saw issues with this setting in case of IMPLICIT ALUA ONLY targets, and hence in ESXi 6.7P01 (ESXi670-201912001) and later versions, "action_OnRetryErrors" is disabled for IMPLICIT ALUA ONLY targets and remains enabled for other types.

This change is detectable at the device level by running the following command

esxcli storage nmp device list

Example output:
Device Display Name: COMPELNT Fibre Channel Disk (naa.60000000000000000000000000000000)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=on; {TPG_id=61445,TPG_state=AO}{TPG_id=61446,TPG_state=AO}} <==
Path Selection Policy: VMW_PSP_MRU
Path Selection Policy Device Config: Current Path=vmhba3:C0:T1:L1
Path Selection Policy Device Custom Config:
Working Paths: vmhba3:C0:T1:L1
Is USB: false

Before we dig into how this change impacts failover behavior we first need to understand how a path is marked as "dead" or "unavailable". See Understanding how paths to a storage/LUN device are marked as Dead

action_onRetryErrors

This setting determines how ESXi will react to a VMK_STORAGE_RETRY_OPERATION response received from a failed path health probing attempt. If "action_OnRetryErrors" setting is enabled, when a VMK_STORAGE_RETRY_OPERATION is received in response to a failed health probing attempt, the path is then marked dead and a failover is initiated after a fixed number of retries. If the setting is disabled, ESXi considers VMK_STORAGE_RETRY_OPERATION as transient and retries IO on same path.

Impact

If there is another path available to receive IO then this failover behavior change won't have an impact. However in the case of some ALUA based arrays such as Dell EMC SC Series ("Compellent") or even general active/passive array configurations, this can be problematic during specific failover scenarios such as a controller failover during a firmware upgrade, or just general controller failover testing as it may take several seconds for the remaining controller to recover ownership of devices previously handled by the degraded controller. Because of the quick failover practice, there may not be a path available for the IO to resume on resulting in a premature APD scenario.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

Related Versions:

VMware vSphere ESXi 7.x

VMware vSphere ESXi 6.7

Take Our Survey

Support Assistant