Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Understanding a "Lost Heartbeat" purple diagnostic screen (1009525)

Last Updated: 3/20/2024Categories: TroubleshootingTotal Views: 43228Language:
JapaneseEnglish

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Symptoms

An ESX host fails with a purple diagnostic screen

The purple diagnostic screen contains this text:

Lost Heartbeat

The purple diagnostics screen appears similar to:

Note: Most of the information in a purple diagnostics screen is boilerplate and does not reflect a specific root cause. Unique information contained within the purple diagnostics screen is highlighted in the following example.

VMware ESX Server [Releasebuild-123630]
Lost heartbeat
frame=0x1f4fa34 ip=0xc014b306 cr2=0xb6bc5054 cr3=0x34401000 cr4=0x6f0
es=0xc02a0068 ds=0x690068 fs=0x0 gs=0x0
eax=0x2dc94 ebx=0xc1609228 ecx=0xc1bb9938 edx=0xc0396c80
ebp=0x2e3d2 esi=0xc160920c edi=0x2dc95 err=0 eflags=0x0
*0:1024/console 1:1367/vmware-vm 2:1125/vcpu-0:VM 3:1351/vmware-vm
4:1384/mks:VM123 5:1029/idle5 6:1321/mks:VM456 7:1166/mks:VM789
8:1120/vmm0:VMAB 9:1150/vmm0:VMCD 10:1215/vmware-vm 11:1035/idle11
12:1237/Worker#0: 13:1380/vmm0:VMEF 14:1211/vmware-vm 15:1170/mks:VMGHI
@BlueScreen: Lost heartbeat
0x2e3d2:[0xc014b306]blk_dev+0xbd8524a5 stack: 0x0, 0x0, 0x0
VMK uptime: 42:14:03:58.308 TSC: 11010722567976450
42:14:02:29.347 cpu0:1024)VMNIX: ALERT: HB: 365: Lost heartbeat (comm=vmnixhbd pid=20 t=29 to=30 clt=1).
42:14:03:29.341 cpu0:1024)VMNIX: ALERT: HB: 365: Lost heartbeat (comm=VMap pid=9173 t=30 to=30 clt=1).
42:14:03:29.508 cpu0:1024)Host: 4781: COS Error: Lost heartbeat
Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log
42:14:03:29.517 cpu7:1047)Dump: 354: Dumping core to '/volumes/4533e089-1dfcdd21-f1d3-001a140ebdca/cos-core-esxhostname'

The cos-core file is written to a datastore, usually named /vmfs/volumes/UUID/cos-core-hostname. For example:

VMware ESX Server [Releasebuild-123630]
Dumping console os core. This may take up to an hour.

Note: While writing the Service Console (COS) coredump, the VMFS filesystems remain active and open files remain locked. This process can take up to an hour. Check the named datastore for the cos-core file. If you engage VMware Tech Support, provide this file in addition to normal logs.

A stack trace from the COS log typically mentions processes that are not necessarily responsible for the crash. The stack call trace usually describes memory operations such as swapping or rebalancing. For an example of a stack trace from the Service Console, see Decoding the purple screens - Service Console OOPS/Panic (1006802).

Note: A lot of the stack trace is boilerplate. Most lost heartbeat purple diagnostic screens reference memory operations (such as refill_inactive_zone, rebalance_inactive do_try_to_free_pages alloc_pages).

If you encounter a purple diagnostic screen that does not match the symptoms above, see Interpreting an ESX host purple diagnostic screen (1004250).

Purpose

This article provides a high-level overview of the causes of a Lost Heartbeat purple diagnostic screen and provides information to help you identify the underlying cause.

A Lost Heartbeat purple diagnostic screen is a symptom of a larger issue. Determining the underlying cause of the purple diagnostic screen is non-trivial. Neither the vmkernel log or Linux OOPS messages log reflect the underlying cause. To prevent re-occurrence of the outage, the underlying root cause must be identified.

Resolution

The ESX VMkernel and the Service Console Linux kernel run at the same time on ESX. The Service Console Linux kernel runs a process called vmnixhbd, which heartbeats the VMkernel as long as it is able to allocate and free a page of memory. If no heartbeats are received before a timeout period of 30 minutes, the VMkernel triggers a COS Oops and a purple diagnostics screen that mentions a Lost Heartbeat. This timeout period is not configurable.

There are numerous possible reasons why a VMkernel does not receive a heartbeat. The Service Console becoming unresponsive is the most common cause. An unresponsive Service Console is usually caused by a memory or CPU contention in the Service Console. This could be caused by a memory leak in a process, third party software with a large memory footprint, or excessive numbers of processes running on the Service Console. To prevent this, be aware of things that can contribute to memory exhaustion. Other possible causes include a failed heartbeat daemon, a broken heartbeat mechanism, RPC-hang, or a VMkernel process thread scheduling issue.

Historical information from /var/log/vmksummary

The /var/log/vmksummary* files are written to once an hour. They record the amount of swap consumed and the highest three memory consumers in the Service Console at that time. The memory footprint of these top-three processes are measured in kilobytes (KB). This provides both the consumption in the hour before the purple diagnostic screen and a historical trend for these metrics. This information is always included in a vm-support log bundle, regardless of whether a Service Console coredump is available. Memory leaks can be slow and are reflected in this file. However, if memory usage spikes rapidly within an hour, this log may not have a record of it.

Here is an example of a /var/log/vmksummary file:

Nov 13 03:01:35 esxhostname logger: (1258099291) hb: vmk loaded, 11795301.25, 11795288.970, 73, 153875, 153875, 567468032, VMap-583524, vmware-h-82812, webAcces-27332
Nov 13 04:01:45 esxhostname logger: (1258102905) hb: vmk loaded, 11798915.64, 11798902.253, 73, 153875, 153875, 567468032, VMap-586196, vmware-h-83180, webAcces-27292
Nov 13 07:13:58 esxhostname vmkhalt: (1258114439) Starting system…

Where:

11798915.64 is the uptime of the host in seconds. In this example, the host was up for 136 days.

73 is the number of virtual machines running. In this example, 73 virtual machines were affected by the failure.

567468032 is the bytes of swap consumed. In this example, 541MB of swap was consumed.

VMap-586196 is the amount of virtual memory (measured in KB) that the VMap service (which is part of VMware HA) was consuming. In this example, the VMap service was consuming 572MB of virtual memory.

vmware-h-83180 is the amount of virtual memory (measured in KB) that vmware-hostd (which is a core management agent). In this example, vmware-hostd was consuming 81MB of virtual memory.

webAcces-27292 is the amount of virtual memory (measured in KB) that webAccess (which is used for browser-based virtual management) was consuming. In this example, webAccess was using 2 6MB of virtual memory.

In this example, at the last hour, the VMap component of VMware HA was consuming an excessive amount of memory and a lot of swap was consumed. Looking backward in the vmksummary log file shows a gradual increase in the memory footprint of the VMap process. This is suggestive of a memory leak.

If the vmksummary log does not show an obvious cause, the memory consumption may have spiked from normal usage to very high within the hour, or it may not have been a single process that caused the problem.

The COS coredump

The Lost Heartbeat purple diagnostic screen writes two coredumps to disk:

The file vmkernel-zdump-111309.07.13.1 contains both a VMkernel coredump and log file, and is named after the date and time of the next successful startup. It is placed in /root/ or /var/core/, and rotated to /root/old_cores/ after collection.

The file cos-core-esxhostname.123.core.gz contains the Service Console coredump and is named after the ESX host. It is placed in /root/old_cores/ or /var/core/.

During startup, the ESX host moves the coredumps to their final location. It is possible that the move operation can fail, so both locations should be checked as analysis cannot proceed without these files.

Through analysis of the cos-coredump file, VMware Tech Support may be able to determine what processes were running at the time of the crash and the memory footprint of each. This information is not historic, but is reflective of state at the time the COS World was stunned for writing the coredump.

If the information in /var/log/vmksummary is not conclusive, engage Tech Support and provide logs and both coredump files. For more information, see Collecting diagnostic information for VMware ESX/ESXi (653).

Memory Allocation to the Service Console

Contrast the information from /var/log/vmksummary with the actual memory allocations (RAM and Swap) made for the Service Console. On a working system, look at /proc/meminfo. This is included in the vm-support log bundle. Here is an example of actual memory allocations in /proc/meminfo:

MemTotal:799732 kB
HighTotal:0 kB
LowTotal:799732 kB
SwapTotal:554168 kB

In this example, there is 554168KB (541MB) of swap and 799732KB (780MB) of RAM assigned to the Service Console. These values reflect configured maximums. The consumption information here (free/used) is post-reboot and is not reflective of the state of the system at the time of the outage.

Comparing these values with information known from /var/log/vmksummary, you can determine that 100% of swap was consumed (567468032 bytes or 541MB of 554168KB or 541MB) and that 69% of RAM was consumed by a single process (586196KB or 572MB of 799732KB or 780MB).

Action Plan

If you experience a Lost Heartbeat purple diagnostic screen, consider performing these options:

In a given incident, the process may be revealed as a component shipped with ESX/VirtualCenter (cimserver, snmpd, vmware-hostd, vmap, etc) or a third-party agent. If there is a memory leak, increasing the amount of memory assigned to the Service Console does not typically prevent memory exhaustion and the purple diagnostic screen, but rather only delays it. If memory usage is generally static but under stress, VMware recommends increasing the amount of RAM or swap assigned to the Service Console. For more information, see Increasing the amount of RAM assigned to the ESX Server service console (1003501).

Once the cause of the memory exhaustion is determined, the next steps to be taken depend on the process at fault. If third-party software is the cause, determine where it came from and consider removing it as per the Third Party Hardware and Software Support Policy. If a VMware component is responsible for leaking memory, engage VMware Technical Support for further investigation. For more information, see How to Submit a Support Request.

Note: Workarounds typically involve disabling the component that is causing the memory leak.

In anticipation of a re-occurrence, run top in batch mode to gather historic process CPU and memory consumption statistics within the Service Console. For more information, see Using performance collection tools to gather data for fault analysis (1006797).

If you do not have a cos-core file, ensure that the Misc.CosCoreFile advanced configuration option is valid. For more information, see Configuring an ESX host to capture a Service Console coredump (1032962).

Related Information

If you have a "Lost Heartbeat" purple diagnostic screen that exactly matches the symptoms outlined in Pegasus (cimserver) memory leaks reported in ESX 3.5 Update 2 and later (1009607), follow the directions in that article.

Note: If the error has not been documented within the Knowledge Base, collect diagnostic information from the ESX host and submit a support request. For more information, see Collecting Diagnostic Information for VMware Products (1008524) and How to Submit a Support Request.

「ハートビート消失」による紫色の診断画面について
Collecting diagnostic information for VMware ESX/ESXi

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
JapaneseEnglish

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Como solucionar problemas de conflito de endereço IP em máquinas virtuais (1008177)

KB
•
Resolución de problemas de conflictos de direcciones IP de máquinas virtuales (1008177)

KB
•
Download and Install VMware Fusion (2014097)

KB
•
无法手动更改虚拟机的 MAC 地址 (2007042)

Results 1-5 of 5

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware ESX Server

Related Versions:

VMware ESX Server 3.5.x

VMware ESX Server 3.0.x

VMware ESX 4.1.x

VMware ESX 4.0.x

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant