Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Interpreting an ESX/ESXi host purple diagnostic screen (1004250)

Last Updated: 8/12/2016Categories: InformationalTotal Views: 48354Language:
Chinese (Simplified)JapaneseEnglishGerman

34 like this article

Subscribe to get an email
newsletter with new updates

34 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Purpose

This article provides information to decode ESX/ESXi host purple screen errors.

An ESX/ESXi purple screen error appears similar to:

Note: This article uses the information in this purple screen as an example.

Resolution

What is the VMkernel?

The VMkernel is the operating system core of ESX/ESXi. The kernel handles resource scheduling and device IO. Device IO is handled by the VMware network and storage stacks, which serves as a layer between the virtual file system, network devices and the device drivers that control physical devices.

Interpreting the purple diagnostic screen

If the VMkernel experiences an error, the error appears in a purple diagnostic screen. The purple diagnostic screen looks similar to:

VMware ESX Server [Releasebuild-98103
PCPU 1 locked up. Failed to ack TLB invalidate.
frame=0x3a37d98 ip=0x625e94 cr2=0x0 cr3=0x40c66000 cr4=0x16c
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3a37ef4 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff
*0:1037/helper1-4 1:1107/vmm0:Fagi 2:1121/vmware-vm 3:1122/mks:Franc
0x3a37ef4:[0x625e94]Panic+0x17 stack: 0x833ab4, 0x3a37f10, 0x3a37f48
0x3a37f04:[0x625e94]Panic+0x17 stack: 0x833ab4, 0x1, 0x14a03a0
0x3a37f48:[0x64bfa4]TLBDoInvalidate+0x38f stack: 0x3a37f54, 0x40, 0x2
0x3a37f70:[0x66da4d]XMapForceFlush+0x64 stack: 0x0, 0x4d3a, 0x0
0x3a37fac:[0x652b8b]helpFunc+0x2d2 stack: 0x1, 0x14a4580, 0x0
0x3a37ffc:[0x750902]CpuSched_StartWorld+0x109 stack: 0x0, 0x0, 0x0
0x3a38000:[0x0]blk_dev+0xfd76461f stack: 0x0, 0x0, 0x0
VMK uptime: 7:05:43:45.014 TSC: 1751259712918392
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1...using slot 1 of 1... log

Here is a breakdown of each section of the above purple diagnostic screen:

The Product and Build:

VMware ESX Server [Releasebuild-98103]

This section of the purple diagnostic screen identifies the product and build that has experienced the error. In this example, the product is VMware ESX Server build 98103.

The Error Message:

PCPU 1 locked up. Failed to ack TLB invalidate

This section of the purple diagnostic screen identifies the reported error message. There are only a finite number of error messages that can be reported. These error messages are discussed in this article.

The CPU Registers:

frame=0x3a37d98 ip=0x625e94 cr2=0x0 cr3=0x40c66000 cr4=0x16c
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
ebp=0x3a37ef4 esi=0xffffffff edi=0xffffffff err=-1 eflags=0xffffffff

These are the values that were in the physical CPU registers at the time of the error. The information in these registers may vary greatly between VMkernel errors. These registers can only be used internally when debugging a core dump of the VMkernel error. For more information about these registers, see http://www.intel.com/products/processor/manuals/ for Intel and http://support.amd.com/us/psearch/Pages/psearch.aspx for AMD. At the AMD site, search for the Architecture Programmer's manual for your specific processor type.

Note: The preceding links were correct as of March 28, 2013. If you find the links to be broken, provide feedback on the article and a VMware employee will update the article as necessary.

The Physical CPU:

*0:1037/helper1-4 1:1107/vmm0:Fagi 2:1121/vmware-vm 3:1122/mks:Franc

This section of the purple diagnostic screen identifies the physical CPU that was running instructions during the VMkernel error. In the example, the * beside the 0 indicates that physical CPU 0 was running an operation at the time of the failure. In newer versions of ESX, instead of including an *, the preceding letters CPU are included. For example, if the same error as the above were to occur in newer versions of VMware ESX, the same line appears as:

CPU0:1037/helper1-4 cpu1:1107/vmm0:Fagi cpu2:1121/vmware-vm cpu3:1122/mks:Franc.

This section of the purple diagnostic screen also describes the world (process) that was running on the CPU at the time of the error. In the above example, the userworld running was helper1-4.

Note: The name of the process may be truncated.

The Stack Trace:

0x3a37ef4:[0x625e94]Panic+0x17 stack: 0x833ab4, 0x3a37f10, 0x3a37f48
0x3a37f04:[0x625e94]Panic+0x17 stack: 0x833ab4, 0x1, 0x14a03a0
0x3a37f48:[0x64bfa4]TLBDoInvalidate+0x38f stack: 0x3a37f54, 0x40, 0x2
0x3a37f70:[0x66da4d]XMapForceFlush+0x64 stack: 0x0, 0x4d3a, 0x0
0x3a37fac:[0x652b8b]helpFunc+0x2d2 stack: 0x1, 0x14a4580, 0x0
0x3a37ffc:[0x750902]CpuSched_StartWorld+0x109 stack: 0x0, 0x0, 0x0
0x3a38000:[0x0]blk_dev+0xfd76461f stack: 0x0, 0x0, 0x0

The stack represents what the VMkernel was doing at the time of the error. In this example, it was trying to clear memory page tables (TLB). This information is a vital tool in the diagnosis of purple screen errors by evaluating the actions of the kernel at the time of the error.

The Uptime:

VMK uptime: 7:05:43:45.014 TSC: 1751259712918392

This section indicates how long a server is running since the last boot. In this example, the ESX host was running for 7 days, 5 hours, 43 minutes and 45.014 seconds. The TSC value is the number of CPU clock cycles that have elapsed since the server was started.

The Core Dump:

Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1...using slot 1 of 1... log

This section of the purple diagnostic screen indicates that the contents of the VMkernel memory are being copied to the vmkcore partition.

Using the error message of the purple diagnostic screen to troubleshoot a vmkernel error

The VMkernel error message generated by the purple screen can be used to identify the cause of the issue. The number of error messages that can be produced are finite. This is a list of known VMkernel error messages.

Type: Console Oops
Example Error: COS Error: Oops
Description: An ESX host fails and causes a purple screen when there is a Service Console oops. Unlike most purple screen errors, it is not triggered by the VMkernel. Instead the error is triggered by the Service Console and occurs at the Linux level. These purple screen errors contain additional information from the Linux kernel. For more information about Console Oops, see Understanding an "Oops" purple diagnostic screen (1006802).

Type: Lost Heartbeat
Example Error: Lost Heartbeat
Description: The ESX VMkernel and the Service Console Linux kernel run at the same time on ESX. The Service Console Linux kernel runs a process called vmnixhbd, which heartbeats the VMkernel as long as it is able to allocate and free a page of memory. If no heartbeats are received before a timeout period of 30 minutes, the VMkernel triggers a COS Panic and a purple diagnostics screen that mentions a Lost Heartbeat. For more information on Lost Heatbeats, see Understanding a "Lost Heartbeat" purple diagnostic screen (1009525) .

Type: Assert
Example Error: ASSERT bora/vmkernel/main/pframe_int.h:527
Description: Assert errors are software errors, because they are related to assumptions on which the program is based. This type of purple screen error is primarily caused by software issues. For more information on the assert error message, see Understanding ASSERT and NOT_IMPLEMENTED purple diagnostic screens (1019956).

Type: Not Implemented
Example Error: NOT_IMPLEMENTED /build/mts/release/bora-84374/bora/vmkernel/main/util.c:83
Description: A not implemented error message occurs when the code encounters a situation that it was not designed to handle. For more information, see Understanding ASSERT and NOT_IMPLEMENTED purple diagnostic screens (1019956).

Type: Spin count exceeded / Possible deadlock
Example Error: Spin count exceeded (iplLock) - possible deadlock
Description: A VMware ESX host may report a Spin count exceeded and possible deadlock in a purple diagnostic screen when a thread is attempting to execute in the critical section of code. Since it was trying to enter the critical section, the thread needed to poll a mutex for a lock prior to executing the code by conducting a spinlock operation. The thread continues to poll the mutex during the spinlock operation, but there is a certain limit of how many times it polls the mutex. For more information on Spin count exceeded errors, see Understanding a "Spin count exceeded" purple diagnostic screen (1020105).

Type: Failed to ack TLB invalidate
Example Error: PCPU 1 locked up. Failed to ack TLB invalidate.
Description: Physical CPUs fail when trying to clear memory page tables. For more information, see Understanding a Failed to ack TLB invalidate purple diagnostic screen (1020214).

A purple diagnostic screen can also come in the form of an Exception. An Exception Handler is a computer hardware mechanism designed to handle some condition that changes the normal flow of execution (Division by Zero, Page Fault, etc). There is no trace from handlers, so you need logging to determine if handler faulted (or single step debugging). This is a list of common exceptions:

Type: Exception 13 (General Protection Fault)
Example Error: #GP Exception(13) in world 4130:helper13-0 @ 0x41803399e303
Description: A general protection fault (Exception 13) occurs under one of the following circumstances: the page being requested does not belong to the program requesting it (and not mapped in program memory), or the program does not have rights to perform a read or write operation on the page. For more information on Exception 13 or Page Fault, see Understanding Exception 13 and Exception 14 purple diagnostic screen events (1020181).

Type: Exception 14 (Page Fault)
Example Error: #PF Exception type 14 in world 136:helper0-0 @ 0x4a8e6e
Description: A page fault (Exception 14) occurs when the page being requested has not been successfully loaded into memory. For more information on Exception 14 or Page Fault, see Understanding Exception 13 and Exception 14 purple diagnostic screen events (1020181).

Type: Exception 18 (Machine Check Exception)
Example Error: Machine Check Exception: Unable to continue
Example Error: Hardware (Machine) Error
Description: A Machine Check Exception (MCE) is generated by the hardware and reported by the host. Consult your hardware vendor in the event of an MCE. By evaluating the information presented, it is possible to identify the individual component reporting the error. For more information on MCE, see Decoding Machine Check Exception (MCE) output after a purple screen error (1005184).

If your VMware ESX or ESXi host experiences an error similar to one of these that does not point you to a general article, search for the error message and stack trace information within the Knowledge Base. If the error has not been documented within the Knowledge Base, collect the diagnostic information from the VMware ESX host and submit a support request.

For more information, see:

Collecting diagnostic information for VMware products (1008524)

How to Submit a Support Request

Using the pattern analysis to troubleshoot multiple vmkernel errors on the same ESX host

In the event that you experience multiple purple diagnostic screens from the same VMware ESX host, you can use the sample of multiple purple diagnostic screens to determine the likeliness of an issue being related to hardware or software. This can be done by identifying patterns in these sections of the purple diagnostic screen:

The error message and the stack trace:

If the error message and stack vary greatly between vmkernel errors, this indicates that software is not always hitting the same error. Although inconclusive, this may indicate a hardware issue.

If the error message and the stack are always identical between vmkernel errors, this indicates that software is always hitting the same error. Although inconclusive, this may indicate a software issue.

For more information about the error message you are experiencing, refer to the above section about the specific error message.

The physical CPU:

If the physical CPU value remains the same across multiple vmkernel errors, this indicates that the software is always failing on the same physical CPU. Although inconclusive, this may indicate a CPU issue.

For more information, see Determining if virtual machine and ESX host unresponsiveness is caused by hardware issues (1003560).

The world:

If the world value remains the same across multiple VMkernel errors, this indicates that the vmkernel is failing when receiving instructions from the same world. Although inconclusive, this may indicate a world is sending instructions that may be triggering the VMkernel error.

Related Information

This is a complete list of exceptions:

Exception Type 0 #DE: Divide Error

Exception Type 1 #DB: Debug Exception

Exception Type 2 NMI: Non-Maskable Interrupt

Exception Type 3 #BP: Breakpoint Exception

Exception Type 4 #OF: Overflow (INTO instruction)

Exception Type 5 #BR: Bounds check (BOUND instruction)

Exception Type 6 #UD: Invalid Opcode

Exception Type 7 #NM: Coprocessor not available

Exception Type 8 #DF: Double Fault

Exception Type 10 #TS: Invalid TSS

Exception Type 11 #NP: Segment Not Present

Exception Type 12 #SS: Stack Segment Fault

Exception Type 13 #GP: General Protection Fault

Exception Type 14 #PF: Page Fault

Exception Type 16 #MF: Coprocessor error

Exception Type 17 #AC: Alignment Check

Exception Type 18 #MC: Machine Check Exception

Exception Type 19 #XF: SIMD Floating-Point Exception

Exception Type 20-31: Reserved

Exception Type 32-255: User-defined (clock scheduler)

For more information about these Exceptions, see the Call and Return Operation for Interrupt or Exception Handling Procedures section in Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture and Chapter 6: Interrupts and Exception Handling in Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A.

Note: The preceding links were correct as of October 2, 2015. If you find a link is broken, provide feedback and a VMware employee will update the link.

There is a known issue when upgrading to vCenter Server 5.0, with ESXi 4.0 Updated 2 hosts. For more information, see ESXi 4.0 hosts may experience a purple screen after vCenter Server is upgraded to 5.0 (2007269).

Determining if virtual machine and ESX host unresponsiveness is caused by hardware issues
Decoding Machine Check Exception (MCE) output after a purple screen error
Understanding an "Oops" purple diagnostic screen
Collecting diagnostic information for VMware products
Understanding a "Lost Heartbeat" purple diagnostic screen
Understanding ASSERT and NOT_IMPLEMENTED purple diagnostic screens
Understanding a "Spin count exceeded" purple diagnostic screen
Understanding Exception 13 and Exception 14 purple diagnostic screen events in ESX 3.x/4.x and ESXi 3.x/4.x/5.x/6.x
Understanding a "Failed to ack TLB invalidate" purple diagnostic screen
ESX/ESXi ホストのパープル スクリーンの解析
ESXi 4.0 hosts may experience a purple screen after vCenter Server is upgraded to 5.0
解释 ESX/ESXi 主机紫色诊断屏幕
VMware 製品の診断情報の収集
IPv6 を有効化した ESXi 5.5 ホストが停止し、紫色の診断画面に find_pfxlist_reachable_router と表示される
Interpretieren eines violetten Diagnosebildschirms des ESX/ESXi-Hosts

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglishGerman

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

VMware ESXi 4.1.x Installable

VMware ESXi 4.1.x Embedded

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESXi 3.5.x Installable

VMware ESXi 3.5.x Embedded

VMware ESX Server 3.5.x

VMware ESX Server 3.0.x

VMware ESX Server 2.5.x

VMware ESX 4.1.x

VMware ESX 4.0.x

view all

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant