Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Skip to main content

My Subscriptions

Loading

Knowledge Base

Investigating virtual machine file locks on ESXi hosts (10051)

Last Updated: 1/12/2022Categories: TroubleshootingTotal Views: 238863Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglishGerman

136 like this article

Subscribe to get an email
newsletter with new updates

136 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Details

Powering on the virtual machine results in the power on task remaining at 95% indefinitely.

Adding an existing virtual machine disk (VMDK) to a powered on virtual machine fails with the error:

Failed to add disk scsi0:1. Failed to power on scsi0:1

Cannot power on the virtual machine after deploying it from a template.

Powering on a virtual machine fails with an error:

Unable to open Swap File

Unable to access a file since it is locked

Unable to access a file filename since it is locked

Unable to access Virtual machine configuration

In the /var/log/vmkernel log file, you see entries similar to:

WARNING: World: VM xxxx: xxx: Failed to open swap file path: Lock was not free
WARNING: World: VM xxxx: xxx: Failed to initialize swap file path

When opening a console to the virtual machine, you may receive the error:

Error connecting to path/virtual machine/vmx because the VMX is not started

The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI.

Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to:

cat: can't open '[name of vm].vmx': Invalid argument

Solution

The Purpose of File Locking

To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

VMNAME.vswp

DISKNAME-flat.vmdk

DISKNAME-ITERATION-delta.vmdk

VMNAME.vmx

VMNAME.vmxf

vmware.log

Initial Quick Test

Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host help the lock.

Identifying the Locked File

Power on the virtual machine, this process should fail and display an error message. Make note of the this error.

Connect to the ESXi host the virtual machine is on with an SSH session

Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.

vmfsfilelockinfo takes these parameters:

File to be tested

Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

For example:

vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u [email-masked]

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.10 with user [email-masked]
Password: xXxXxXxXxXx
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.178
Searching on Host 192.168.1.179
Searching on Host 192.168.1.180
MAC Address : xx:xx:xx:xx:xx:xx

Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

Total time taken : 0.27 seconds.

Note: During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

mode 0 = no lock

mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)

mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)

mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)

To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:

For example:

lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

Cartel | World name | Type | fd | Description
36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk

This shows that the file is locked by a virtual machine having Cartel ID 36202.

Now display the list of active Cartel IDs by with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
Display Name: Alternate_VM27
Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
………

The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.

If no processes are shown, the following script can search for VMs with the locked vmdk mounted.

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/"  '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Removing the Lock

Power off the virtual machine holding the lock

Reboot the ESXi host that holds the lock

Removing the .lck file (NFS only)

Power down the virtual machine and remove lck-#### files in the virtual machines directory.

Caution: Only remove the lck files of a powered off virtual machine

Note: VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.

Verify the integrity of the virtual machine configuration file (.vmx)

For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity

For related information, see

"Cannot open the disk" errors powering on a VM .

"An error occurred while consolidating disks:" deleting one or more snapshots or consolidate disks

Powering off an unresponsive virtual machine on an ESXi host

Opening a Support Request

If the problem persists after completing the steps in this article:

Collecting diagnostic information for VMware products .

Investigating virtual machine file locks on ESXi

Related Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Troubleshooting virtual machine power on task hangs

Failed to power on virtual machine

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglishGerman

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware vCenter Server

Related Versions:

VMware vSphere ESXi 8.0.x

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vCenter Server 8.0.x

VMware vCenter Server 7.0.x

VMware vCenter Server 6.7.x

VMware vCenter Server 6.5.x

VMware vCenter Server 6.0.x

view all

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Cookie Preference Center

General Information on cookies

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Please confirm your settings by refreshing the page.

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Please confirm your settings by refreshing the page.

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Support Assistant