Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Snapshot removal task stops at 99% (1007566)

Last Updated: 2/9/2021Categories: InformationalTotal Views: 34020Language:
Chinese (Simplified)JapaneseEnglishGerman

15 like this article

Subscribe to get an email
newsletter with new updates

15 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Symptoms

The snapshot removal task hang at 95% or 99%.

You cannot observe the progress when you delete a snapshot.

Note: For additional symptoms, see the Additional Information section.

Purpose

This article provides steps to monitor snapshot removal on ESXi using the watch command.

Cause

Note: Time involved to commit snapshots is environmental and subjective.

To estimate the time it will take to remove snapshots, see Estimate the time required to consolidate virtual machine snapshots

The consolidation/Removal task hanging at 99% is in most cases related to:

Virtual Machine snapshot size on disk (Delta or sesparse file size).

Storage performance.

Network connection interrupted between ESXI host and vCenter.

Size of the Virtual Machine.

The virtual machine may become unresponsive due to the stun process integrated with the Consolidation process. The stun/freeze happens to accommodate the changes to be written back to the base disk from the Delta disks.  At times if the virtual machine is generating a lot of I/O or the underlying storage is experiencing latency, the stun time can increase.

Impact / Risks

The is no way to stop or cancel the consolidation process without risking data corruption. Do not attempt to create a new snapshot on the consolidating virtual machine.

Resolution

Finding consolidation tasks that are at 99%
Monitor using watch in ESXi 6.x/ 7.x
For 5.x, see Monitor snapshot deletion using esxtop command (2146232)
Monitor using vim-cmd in ESXi

Finding consolidation tasks that are at 99%

Log in as root to the ESXi host using SSH.

Navigate to the virtual machine directory containing vmdk virtual disk files. Run the cd command to change the current directory.

For example:

cd /vmfs/volumes/Datastore_name/Virtual_Machine_name/

Run less testdev02.vmx | grep vmdk to list vmdk (disks) of the VM in the vmx (configuration file of VM):

For example:

less testdev02.vmx |grep vmdk
scsi0:0.fileName = "testdev02.vmdk"
scsi0:1.fileName = "testdev02_1-000003.vmdk"
scsi0:2.fileName = "testdev02_2-000002.vmdk"

Run this command to list the snapshot files in the directory:

less testdev02.vmx | grep *sesparse.vmdk

For VMFS5 volume: less testdev02.vmx | grep *delta.vmdk

Listing the snapshot of the VM.

-rw-------    1 root     root       18.2G Oct 29 10:04 testdev02_2-000001-sesparse.vmdk
-rw-------    1 root     root      943.6G Oct 29 10:09 testdev02_1-000002-sesparse.vmdk
-rw-------    1 root     root        1.8T Oct 29 10:09 testdev02_1-000001-sesparse.vmdk
-rw-------    1 root     root        1.7G Oct 29 20:03 testdev02_2-000002-sesparse.vmdk
-rw-------    1 root     root        3.1T Oct 29 20:03 testdev02_1-000003-sesparse.vmdk

Note: Sesparse is a default format for all delta disks on the VMFS6 datastores. On VMFS5, Sesparse is used for virtual disks of the size 2 TB and larger.

In this example we have snapshots approximately 6 TB and this could be the main reason for the progress of the task. The process will continue to show 99% until the snapshot removal task completes and VM is using the flat files of the VMDK.

Find the consolidation task with this command: vim-cmd vimsvc/task_list

For example:

[root@ESXi:~] vim-cmd vimsvc/task_list
(ManagedObjectReference) [
'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
'vim.Task:haTask-ha-host-vim.host.ImageConfigManager.installDate-321689752',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.start-321689774',
'vim.Task:haTask-ha-host-vim.host.ServiceSystem.refresh-321689779'
]

View the details of the consolidation with this command: vim-cmd vimsvc/task_info <task_name_from_step_6>

[root@ESXi:~] vim-cmd vimsvc/task_info haTask-91-vim.vm.Snapshot.remove-321615207
(vim.TaskInfo) {
key = "haTask-91-vim.vm.Snapshot.remove-321615207",
task = 'vim.Task:haTask-91-vim.vm.Snapshot.remove-321615207',
description = (vmodl.LocalizableMessage) null,
name = "vim.vm.Snapshot.remove",
descriptionId = "vm.Snapshot.remove",
entity = 'vim.VirtualMachine:81',
entityName = "TESTDEV",
locked = <unset>,
state = "running",
cancelled = false,
cancelable = true,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 99,
reason = (vim.TaskReasonUser) {
userName = "vpxuser"
},
queueTime = "2019-10-29T10:08:21.403596Z",
startTime = "2019-10-29T10:08:21.404185Z",
completeTime = <unset>,
eventChainId = -1078812089,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
activationId = <unset>
}

Monitor using watch in ESXi 6.x/7.x

To monitor the VMDK snapshot and base disks which are currently being updated, run this watch command:

This command refreshes the file referred by the on going task every 2 seconds.

#   watch -d 'ls -luth | grep -E "delta|flat|sesparse"'

where:

-d highlights the differences between successive updates

In addition, run this command to monitor the time stamp update of the base disks to confirm if the process is working. The result lists the files by their modification date.

ls -lrth |grep -E "flat|delta|sesparse"

Notes:

In ESX/ESXi 4.0 Update 2 and later, the process works differently in that the data in snapshots (deltas) are written directly to the base disk (flat).

If there are more than 10 snapshots, run this command to monitor the snapshot commit process and to prevent the screen from filling with many files:

#while true;do date;ls -lht *vmdk|head -10;echo ________;sleep 3;done

You can quit the consolidation process monitoring by pressing Ctrl + C.

Related Information

VMware Skyline Health Diagnostics for vSphere - FAQ
You experience these additional symptoms:

The hostd process does not respond.

Storage vMotion migration stays at 18% for a long time.

Best practices for using snapshots in the vSphere environment
Tech Support Mode for Emergency Support
Consolidating snapshots in ESX/ESXi 3.x and 4.x
Identifying disks when working with VMware ESXi/ESX
Understanding VM snapshots in ESXi / ESX
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
Editing files on an ESX host using vi or nano
ESXi/ESX でのスナップショットの削除タスクが 99% で停止する
在 ESXi/ESX 中，快照删除任务在进行到 99% 时停止
How to monitor snapshot deletion using the vim-cmd command
How to monitor snapshot deletion using esxtop command
Aufgabe zum Entfernen eines Snapshots wird in ESXi/ESX bei 99 % beendet

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglishGerman

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant