Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Understanding and troubleshooting vMotion (1003734)

Last Updated: 3/9/2021Categories: TroubleshootingTotal Views: 299495Language:
Chinese (Simplified)KoreanJapaneseEnglish

20 like this article

Subscribe to get an email
newsletter with new updates

20 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Symptoms

vMotion stuck.

vMotion fails at 10%

vMotion times out

In vCenter Server, you see these errors similar to:

Migration will cause the virtual machine's configuration to be modified to preserve the CPU feature requirements for it's guest operating system.
Operation timed out

A general system error occurred:

Failed waiting for data. Error 16. Invalid argument

A general system error occurred: failed to look up vMotion destination resource pool object

Purpose

This article provides information on troubleshooting the VMware vMotion process.

Resolution

Introduction

When performing a live migration of a virtual machine from one ESXi host to another, vMotion consist of these steps:

vMotion request is sent to the vCenter Server

vCenter Server sends the vMotion request to the destination ESXi host

vCenter Server computes the specifications of the virtual machine to migrate

vCenter Server sends the vMotion request to the source ESXi host to prepare the virtual machine for migration

vCenter Server initiates the destination virtual machine

vCenter Server initiates the source virtual machine

vCenter Server switches the virtual machine's ESXi host from the source to destination

vCenter Server completes the vMotion task

Understanding these tasks can assist in the troubleshooting and remediation of vMotion failures. Below is a summary of each step and common issues you may experience.

vMotion request is sent to the vCenter Server

During this stage, a call is sent to vCenter Server requesting the live migration of a virtual machine to another host. This call may be issued through the VMware vSphere Web Client, VMware vSphere Client or through an API call.

vCenter Server sends the vMotion request to the destination ESXi host

During this stage, a request is sent to the destination ESXi host by vCenter Server to notify the host for an incoming vMotion. This step also validates if the host can receive a vMotion. If a vMotion is allowed on the host, the host replies to the request allowing the vMotion to continue. If the host is not configured for vMotion, the host replies to the request disallowing the vMotion, resulting in a vMotion failure.

Common issues:

Ensure that vMotion is enabled on all ESX/ESXi hosts. For more information, see Enabling vMotion and Fault tolerance logging (1036145).

Determine if resetting the Migrate.Enabled setting on both the source and destination ESX or ESXi hosts addresses the vMotion failure. For more information, see vMotion fails at 10% with the error: A general system error occurred: Migration failed while copying data, Broken Pipe (1013150).

Verify that VMkernel network connectivity exists and there is no packet loss. For more information, see vMotion fails with network errors.

Verify that Name Resolution is valid on the host. For more information, see Identifying issues with and setting up name resolution on ESX/ESXi Server (1003735).

Verify if the ESXi/ESX host can be reconnected or if reconnecting the ESX/ESXi host resolves the issue. For more information, see Changing an ESXi or ESX host's connection status in vCenter Server (1003480).

Verify that you do not have two or more virtual machine swap files in the virtual machine directory. For more information, see vMotion fails at 14% with the error: Operation Timed out (2054100).

If you are migrating a virtual machine to or from a host running VMware ESXi below version 5.5 Update 2, see vMotion fails at 14% with the error: Timed out waiting for migration start request (2068817).

vCenter Server computes the specifications of the virtual machine to migrate

During this stage, details of the virtual machine are queried to notify the source and destination hosts of the vMotion task details. This may include any Fault Tolerance settings, disk size, vMotion IP address streams, and the source and destination virtual machine configuration file locations.

Common issues during this stage are:

Verify that the virtual machine is not configured to use a device that is not valid on the target host. For more information, see Troubleshooting migration compatibility error: Device is a connected device with a remote backing (1003780).

vCenter Server sends the vMotion request to the source ESXi host to prepare the virtual machine for migration

During this stage, a request is made to the source ESXi host by vCenter Server to notify the host for an incoming vMotion. This step validates if the host can send a vMotion. If a vMotion is allowed on the host, the host replies to the request allowing the vMotion to continue. If the host is not configured for vMotion, the host will reply to the request disallowing the vMotion, resulting in a vMotion failure.

Once the vMotion task has been validated, the configuration file for the virtual machine is placed into read-only mode and closed with a 90 second protection timer. This prevents changes to the virtual machine while the vMotion task is in progress.

Common issue during this stage are:

Ensure that vMotion is enabled on all ESX/ESXi hosts. For more information, see Enabling vMotion and Fault tolerance logging (1036145).

Determine if resetting the Migrate.Enabled setting on both the source and destination ESX or ESXi hosts addresses the vMotion failure. For more information, see vMotion fails at 10% with the error: A general system error occurred: Migration failed while copying data, Broken Pipe (1013150).

Verify that VMkernel network connectivity exists and there is no packet loss. For more information, see vMotion fails with network error or Performing vMotion fails at 14% despite vmkping succeeding from source to target IP addres.

Verify that Name Resolution is valid on the host. For more information, see Identifying issues with and setting up name resolution on ESX/ESXi Server (1003735).

Verify if the ESXi/ESX host can be reconnected or if reconnecting the ESX/ESXi host resolves the issue. For more information, see Changing an ESXi or ESX host's connection status in vCenter Server (1003480).

vCenter Server initiates the destination host virtual machine

During this stage, the destination host creates, registers and powers on a new virtual machine. The virtual machine is powered on to a state that allows the virtual machine to consume resources and prepares it to receive the virtual machine state from the source host. During this time a world ID is generated that is sent to the source host as the target virtual machine for the vMotion.

Common issues:

Verify that the required disk space is available. For more information, see Investigating disk space on an ESX or ESXi host (1003564).

Verify that time is synchronized across environment. For more information, see Verifying time synchronization across an ESX/ESXi host environment (1003736).

Verify that hostd is not spiking the console. For more information, see Checking for resource starvation of the ESX Service Console (1003496).

Verify that valid limits are set for the virtual machine being vMotioned. For more information, see VMware vMotion fails if target host does not meet reservation requirements (1003791).

Verify if the ESXi/ESX host can be reconnected or if reconnecting the ESX/ESXi host resolves the issue. For more information, see KB article Changing an ESXi or ESX host's connection status in vCenter Server (1003480).

vCenter Server initiates the source host virtual machine

During this stage, the source host begins to migrate the memory and running state of the source virtual machine to the destination virtual machine. This information is transferred using VMkernel ports configured for vMotion. Additional resources are allocated for the destination virtual machine and additional helper worlds are created. The memory of the source virtual machine is transferred using checkpoints.

After the memory and virtual machine state is completed, a stun of the source virtual machine occurs to copy any remaining changes that occurred during the last checkpoint copy. Once this is complete the destination virtual machine resume as the primary machine for the virtual machine that is being migrated.

In cases where vMotion memory pre-copy cannot converge, it leads to vMotion failure. To prevent this, SDPS (Stun During Page Send) intentionally stuns (slows down) the vCPUs to keep the virtual machine's memory modification rate below the vMotion network transmit rate to force memory pre-copy convergence. The duration of a stun is in the order of 10's of micro secs. SDPS comprises of many such small stuns to vCPUs spaced across time in a fashion that the impact of the stun is not noticeable by the Guest Operating System or its application in most cases. But there are cases where total amount of SDPS stun time can lead to the guest operating system or application running slower than expected. It may also affect the time on the guest OS.

Common issues:

If Jumbo Frames are enabled (MTU of 9000) (9000 -8 bytes (ICMP header) -20 bytes (IP header) for a total of 8972), ensure that vmkping is using the command:

vmkping -d -s 8972 destinationIPaddress

Note: You may experience problems with the trunk between two physical switches that have been misconfigured to an MTU of 1500.

Verify that valid limits are set for the virtual machine being vMotioned. For more information, see VMware vMotion fails if target host does not meet reservation requirements (1003791).

Verify the virtual hardware is not out of date. For more information, see vMotion fails at 82% with the hostd log error: Source detected that destination failed to resume (1006052).

This issue may be caused by SAN configuration. Specifically, this issue may occur if zoning is set up differently on different servers in the same cluster.

Verify and ensure that the log.rotateSize parameter in the virtual machine's configuration file is not set to a very low value. For more information, see vMotion fails at 10% with the error: Operation timed out (2007343).

If you are migrating a 64-bit virtual machine, verify that the VT option is enabled on both the source and destination host. For more information, see vMotion fails at 90% with the error: A general system error occurred: failed to resume on destination message (1008735).

Verify that there are no issues with the shared storage or networking. For more information, see Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts (1003659) and vMotion fails with the error: Migration to host <> failed with error (1030267).

If you are using NFS storage, verify if the VMFS volume containing the VMDK file of a virtual machine being migrated is on an NFS datastore and the datastore is not mounted differently on both the source and destination. For more information, see vMotion fails on a NFS datastore (1023230).

If you are using VMware vShield Endpoint, verify the vShield Endpoint LKM is installed on the ESX/ESXi hosts to which you are trying to vMotion the virtual machine. For more information, see Trying to vMotion or power on a virtual machine being protected by vShield Endpoint fails (1030463).

vCenter Server switches the virtual machine's ESXi host from the source to destination

During this stage, the virtual machine is running on the destination ESXi host. vCenter Server will update to reflect this change by changing the virtual machines host and pool to the destination host. The source virtual machine is powered down and unregistered from the source host. Any resources that were in use are released back to the host.

vCenter Server completes the vMotion task

During this stage, the vMotion task is marked as complete.

Common issues during this stage are:

Verify that Console OS network connectivity exists. For more information, see Testing network connectivity with the ping command (1003486).

Related Information

VMware Skyline Health Diagnostics for vSphere - FAQ
vMotion fails with network errors
Testing network connectivity with the ping command
Restarting the Management agents in ESXi
Checking for resource starvation of the ESX Service Console
Investigating disk space on an ESX or ESXi host
Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts
Collecting diagnostic information in a VMware Virtual Infrastructure Environment
Testing VMkernel network connectivity with the vmkping command
Identifying issues with and setting up name resolution on ESX/ESXi Server
Verifying time synchronization across an ESX/ESXi host environment
Troubleshooting migration compatibility error: Device is a connected device with a remote backing
VMware vMotion fails if target host does not meet reservation requirements
vMotion fails at 82% with the hostd log error: "Source detected that destination failed to resume"
vMotion fails at 90% with the error: A general system error occurred: failed to resume on destination message
VMotion fails after a third-party security tool performs a port scan of the ESX/ESXi hosts
VMotion fails on a NFS datastore
vMotion fails with the error: Migration to host <> failed with error
Performing vMotion or powering on a virtual machine being protected by vShield Endpoint fails
vMotion fails at 10% with the error: Operation timed out
Performing vMotion fails at 14% despite vmkping succeeding from source to target IP address
vMotion fails at 14% with the error: Operation Timed out
vMotion fails at 14% with the error: Timed out waiting for migration start request
VMware vMotion の理解とトラブルシューティング
了解 VMware vMotion 和故障排除

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)KoreanJapaneseEnglish

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware vCenter Server

Related Versions:

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant