Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

EMC VMAX and PowerMAX arrays experience VMFS corruption when using VAAI XCOPY on vSphere 6.x (74595)

Last Updated: 9/29/2023Categories: InformationalTotal Views: 29466Language:
Chinese (Simplified)JapaneseSpanishEnglish

31 like this article

Subscribe to get an email
newsletter with new updates

31 people found this helpful Please provide article feedback:

Symptoms

Following a Storage VMotion or VM cloning operation that utilizes the VAAI XCOPY feature, the destination Datastore may become corrupt and be taken offline.

Recognition of mismatch in devID on the destination datastore:

2019-08-12T10:56:01.122Z cpu42:662296)LVM: 4120: [naa.60000970000297900266533030303732:1] devID mismatch:
2019-08-12T10:56:01.122Z cpu42:662296)LVM: 4121: Cached: <naa.60000970000297900266533030303732:1>
2019-08-12T10:56:01.122Z cpu42:662296)LVM: 4122: Read: 5d09cd35-e6dd0eb0-c1e8-98f2b3ef28e2

Next, you will observe the VMFS volume being forced offline similar to:

2019-08-12T10:56:01.122Z cpu42:662296)LVM: 6493: Forcing APD unregistration of devID 5d09cdc6-7682d4ee-4050-98f2b3ef28e2 in state 1.
2019-08-12T10:56:01.122Z cpu42:662296)LVM: 5866: Could not open device , vol [5d09cdc6-605944ea-e5a7-98f2b3ef28e2, 5d09cdc6-605944ea-e5a7-98f2b3ef28e2, 1]: Device does not contain a logical volume
2019-08-12T10:56:01.125Z cpu42:662296)Vol3: 3155: Failed to get object 28 type 1 uuid 5d09cdcc-f6be88c4-5a31-98f2b3ef28e2 FD 0 gen 0 :No filesystem on the device
2019-08-12T10:56:01.125Z cpu42:662296)WARNING: Fil3: 1389: Failed to reserve volume f530 28 1 5d09cdcc f6be88c4 f2985a31 e228efb3 0 0 0 0 0 0 0

You may see the following messages on hosts in the cluster that did not initiate the clone/migration similar to:

2019-08-12T11:28:21.515Z cpu18:65844)WARNING: Vol3: 2422: FS 5d09ce55-a64b1c48-f404-98f2b3ef28e2 uuid change detected, possibly got re-formatted by other host, new uuid 5d09cd3b-6c6db68e-a32c-98f2b3ef28e2

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Impact / Risks

VMFS corruption

Resolution

VMware and Dell|EMC engineering teams have worked closely to develop fixes that will prevent the datastore from being overwritten during an XCopy session. It is recommended to load both the Dell microcode fix and the ESXi patch before re-enabling XCOPY from the host or array.

Patches for ESXi 6.5 and 6.7 have been created that intercepts all attempts to overwrite block 0 on EMC Symmetrix based storage arrays (VMAX/PowerMAX). With the fix, if the array detects an XCOPY attempt to overwrite block 0, the array will reject this attempt and return a custom SCSI sense code to the host. If the ESXi hosts have the patches installed, it will interpret this rejection and fail the XCOPY operation entirely instead of reverting to the software datamover and the XCOPY operation will need to be attempted again.

The Dell EMC fix was released in February 2020 to address this corruption issue. As of May 4, 2020, Dell released microcode enhancement 105521 which supersedes enhancement 104495. This fix is referenced in Dell's DTA 537000: https://www.dell.com/support/kbdoc/en-us/000002467

VMware has released patches for ESXi 6.5 & 6.7 that protects and reports attempts to alter the VMFS metadata region:

ESXi 6.7: https://docs.vmware.com/en/VMware-vSphere/6.7/rn/esxi670-202004002.html

ESXi 6.5: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/esxi650-202007001.html

Workaround

Until you are able to install the fix from Dell EMC, the workaround is to disable the VAAI XCOPY feature on the ESXi hosts.

To disable hardware accelerated move using the vSphere Web Client, follow the below procedure:

Browse to the host in the vSphere Web Client navigator.

Click the Configure tab.

Under System, click Advanced System Settings.

Change the value for DataMover.HardwareAcceleratedMove to 0 (disabled):

For more information, see Disabling Hardware Accelerated Move (XCOPY) in ESXi (2146567).

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseSpanishEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

Related Versions:

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

Take Our Survey

Support Assistant