Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

vCenter reported "Host cannot communicate with other hosts" however there was never a network partition (66984)

Last Updated: 2/24/2023Categories: AlertsTotal Views: 8010Language:
Chinese (Simplified)JapaneseEnglish

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Symptoms

vCenter reported an alarm "One host cannot communicate with other hosts and after a while the error disappeared from the vSAN health plugin.

From the vCenter server: - vmware-vsan-health-service-991.log; you may see vCenter could not fetch vmk information used for vSAN/unicast on Host:<HOSTNAME> during unicast check:

2019-01-02T04:58:38.509Z WARNING vsan-health[100e4ede-0e4b-11e9] [VsanVcClusterHealthSystemImpl::_GetConsistentConfigTest] Vmknic not present on host <hostname>.cityofmarion.lan, skip testing unicast

From the vCenter health report file: we see that health plugin reported vmknic on the host-1808 had issues howvever it also reported that "Cluster partition" state was green/good.

2019-01-02T04:58:40.55Z INFO vsan-health[healthThread-CloudHealthSender-22884] [VsanHealthSummaryLogUtil::PrintHealthResult] Cluster COM vxRail Overall Health : red
Group network health : red
Test hostdisconnected health : green
Test hostconnectivity health : red
HostsWithCommunicationIssues: Host
(Host-88320),
Test clusterpartition health : green
Test vsanvmknic health : red
HostsWithNoVsanVmknicPresent: Host​​
     (Host-88320),
Test matchingsubnet health : yellow
VsanIpSubnetConfigurations: Host IpSubnet(S)
(Host-88316, 10.3.1.0/24), (Host-106228, 10.3.1.0/24), (Host-88318, 10.3.1.0/24), (Host-106233, 10.3.1.0/24), (Host-88312, 10.3.1.0/24), (Host-88320, ''),
(Host-106239, 10.3.1.0/24), (Host-88314, 10.3.1.0/24),
Test smallping health : green
Test largeping health : green

You may not see any host being partitioned from clomd.log; hostd and vpxa.log. Clomd may never report any change state or node drop count.

From vsanmgmt.0 you may see "cmmds-tool" was reporting "Cannot allocate memory"

2019-01-02T04:58:38Z VSANMGMTSVC: ERROR vsanperfsvc[117160c8-0e4b-11e9] [VsanStretchedClusterSystemImpl::GetStretchedClusterInfoFromCmmds] Failed to get stretched cluster info from cmmds: Running cmd ['/bin/cmmds-tool', 'find', '--format=python', '-t', 'NODE'] with error /bin/cmmds-tool: error while loading shared libraries: libpthread.so.0: failed to map segment from shared object: Cannot allocate memory
2019-01-02T04:58:38Z VSANMGMTSVC: ERROR vsanperfsvc[117160c8-0e4b-11e9] [VsanStretchedClusterSystemImpl::GetStretchedClusterInfoFromCmmds] Running cmd ['/bin/cmmds-tool', 'find', '--format=python', '-t', 'NODE'] with
error /bin/cmmds-tool: error while loading shared libraries: libpthread.so.0: failed to map segment from shared object: Cannot allocate memory Traceback (most recent call last): File "/build/mts/release/bora-10390
117/bora/build/esxvsan/release/vsan/usr/lib/vmware/vsan/perfsvc/VsanStretchedClusterSystemImpl.py", line 349, in GetStretchedClusterInfoFromCmmds File "/usr/lib/vmware/hostd/hmo/VsanInternalSystem.py", line 69, in
_cmmds_find raise Exception('Running cmd %s with error %s' % (cmd, err)) Exception: Running cmd ['/bin/cmmds-tool', 'find', '--format=python', '-t', 'NODE'] with error /bin/cmmds-tool: error while loading shared
libraries: libpthread.so.0: failed to map segment from share
2019-01-02T04:58:38Z VSANMGMTSVC: d object: Cannot allocate memory

vmkwarning: reports cmmds-tool reporting " Admission check failed for memory resource

2019-01-02T04:59:59.847Z cpu16:64944763)WARNING: User: 4530: cmmds-tool: Error in initial cartel setup: Failed late cartel initialization: Admission check failed for memory resource

From vsansystem.log: cmmds module in inaccessible state and busy. This is due to out of memory on cmmds module as reported in the above events.

Further inspecting the vsansystem.log, we see cmmds reporting memory exhaustion hence it failed to get membership count.

Hence the node count was reported as "nodeCount: 0" and that caused the network health alarm on the vCenter. This is a false positive alert where there was no network partition.

2019-01-02T04:58:32.462Z info vsansystem[A2E0BEB700] [Originator@6876 sub=Libs opID=0d939440-0e4b-11e9] ModuleImpl Refresh: Module became inaccessible: cmmds
2019-01-02T04:58:32.464Z error vsansystem[A2E0BEB700] [Originator@6876 sub=VsanSystemProvider opID=0d939440-0e4b-11e9] Error querying host status: Unable to load module /usr/lib/vmware/vmkmod/cmmds: Busy
2019-01-02T04:58:33.292Z info vsansystem[A2E001A700] [Originator@6876 sub=VsanSystemProvider opID=CMMDSAccessUpdate-a6b0] Timer fired: posting access generation update
2019-01-02T04:58:33.292Z info vsansystem[A2E001A700] [Originator@6876 sub=VsanSystemProvider opID=CMMDSAccessUpdate-a6b0] Posting VSAN access generation update (genNo: '1143', batched: 20, now: 1899377715332, lastUpdate: 1899316714310)
2019-01-02T04:58:33.293Z info vsansystem[A2E01E0700] [Originator@6876 sub=Libs opID=CMMDSAccessUpdate-a6b0] ModuleImpl Refresh: Module became inaccessible: cmmds
2019-01-02T04:58:33.294Z error vsansystem[A2E01E0700] [Originator@6876 sub=VsanSystemProvider opID=CMMDSAccessUpdate-a6b0] Error retrieving membership list: Unable to load module /usr/lib/vmware/vmkmod/cmmds: Busy
2019-01-02T04:58:33.294Z info vsansystem[A2E01E0700] [Originator@6876 sub=VsanSystemProvider opID=CMMDSAccessUpdate-a6b0] Complete, nodeCount: 0, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
--> membershipList = <unset>,
--> diskIssues = <unset>,
--> accessGenNo = 1143
--> }

++ vsanmgmt: memory was being reported on vsanperfsvc much before the issue window and during the window

2019-01-02T03:35:38Z VSANMGMTSVC: INFO vsanperfsvc[MainThread] [statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=152.132
MB, eMinPeak=155.648MB, rMinPeak=157.820MB MEMORY PRESSURE
2019-01-02T03:35:38Z VSANMGMTSVC: INFO vsanperfsvc[MainThread] [statsdaemon::_logDaemonMemoryStats] 'python.59873341' memory stats:
eMin=152.132MB, eMinPeak=155.284MB, rMinPeak=155.916MB MEMORY PRESSURE

Cause

Partition and vSAN health warning are all related to out of memory of vSAN management daemon.

Resolution

Out of Memory on vSAN management daemon is now fixed in 6.5 U2 Patch 03.

Workaround

At the moment, there is no workaround for this. This a false positive error which may report network partition.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
vSphere Update Manager がホストをスキャンまたは修正できない：エラー「vCenter Server がホストのアップグレード エージェントを展開できません (vCenter cannot deploy Host upgrade agent to host)」 (67018)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSAN

Related Versions:

VMware vSAN 6.6.x

Take Our Survey

Support Assistant