Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Network Partition Of vSAN Node (74601)

Last Updated: 7/23/2023Categories: TroubleshootingTotal Views: 4887Language:
English

2 like this article

Subscribe to get an email
newsletter with new updates

2 people found this helpful Please provide article feedback:

Symptoms

The issue has been seen in 2 Node Stretched cluster of vSAN

ESXi version 6.7 u2. Cluster is partitioned with Witness able to communicate over ping to only one Data Node.

The 2 Data nodes are able to communicate with each other and show in "vsan cluster get " command

The issue related to NIC card ( physical )

Purpose

How to isolate the issue with the NIC

Cause

The NIC was in a " hung state "  ( With the latest driver  )

Resolution

Packets get dropped upon ping to VSAN vmkernel.

NODE2# vmkping -I vmk2 192.168.1.111 -c 1000
PING 192.168.1.111 (192.168.1.111): 56 data bytes
64 bytes from 192.168.1.111: icmp_seq=2 ttl=64 time=0.133 ms
64 bytes from 192.168.1.111: icmp_seq=3 ttl=64 time=0.111 ms
64 bytes from 192.168.1.111: icmp_seq=4 ttl=64 time=0.129 ms
64 bytes from 192.168.1.111: icmp_seq=5 ttl=64 time=0.133 ms
64 bytes from 192.168.1.111: icmp_seq=6 ttl=64 time=0.137 ms
64 bytes from 192.168.1.111: icmp_seq=7 ttl=64 time=0.140 ms
64 bytes from 192.168.1.111: icmp_seq=8 ttl=64 time=0.141 ms
64 bytes from 192.168.1.111: icmp_seq=9 ttl=64 time=0.127 ms
64 bytes from 192.168.1.111: icmp_seq=10 ttl=64 time=0.139 ms
64 bytes from 192.168.1.111: icmp_seq=11 ttl=64 time=0.087 ms<======= Sequence missed
64 bytes from 192.168.1.111: icmp_seq=37 ttl=64 time=0.137 ms<======= Sequence missed
64 bytes from 192.168.1.111: icmp_seq=38 ttl=64 time=0.151 ms

Packet capture shows UDP traffic is working but  We have seen the "sequence 11 is followed by sequence 37"

# pktcap-uw --uplink vmnic4 --dir 0 --stage 1 --proto 0x11 -o -| tcpdump-uw -r - -nne   >> Run this command on one of the data node where uplink 4 is used for vSAN vmkernel.

----- Output of the above command is as below  -----

The Stage is Post.
The session filter IP protocol is 0x11.
pktcap: The output file is -.
pktcap: No server port specifed, select 21248 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 21248.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Accept...
pktcap: Vsock connection from port 1029 cid 2.
07:39:15.068063 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 178: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 136
07:39:16.068090 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 178: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 136
07:39:17.068136 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 258: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 216
07:39:17.068162 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 186: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 144
07:39:18.068157 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 258: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 216
07:39:18.068186 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 186: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 144
07:39:19.068208 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:20.068203 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:21.068238 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:22.068288 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:23.068326 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:24.068347 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:25.068365 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:26.068417 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 242: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 200
07:39:27.068432 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 466: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 424
07:39:28.068511 00:50:56:6a:c1:90 > 00:50:56:61:cb:93, ethertype IPv4 (0x0800), length 466: 192.168.1.222.12321 > 192.168.1.111.12321: UDP, length 424

The same packet capture with ICMP filter shows more drops:

# pktcap-uw --uplink vmnic5 --dir 0 --stage 0 --proto 0x01 -o -|tcpdump-uw -r - -nne
The name of the uplink is vmnic5.
The Stage is Pre.
The session filter IP protocol is 0x01.
pktcap: The output file is -.
pktcap: No server port specifed, select 42606 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 42606.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Accept...
pktcap: Vsock connection from port 1026 cid 2.

7:45:06.559790 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 98, length 64
07:45:07.561992 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 99, length 64
07:45:08.562521 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 100, length 64
07:45:09.564725 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 101, length 64
07:45:10.566928 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 102, length 64
07:45:11.569107 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 103, length 64
07:45:27.598571 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 >  192.168.1.222: ICMP echo request, id 36438, seq 119, length 64 <======== show sequence missed again.
07:45:28.600526 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 120, length 64
07:45:29.602738 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 121, length 64
07:45:30.604959 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 122, length 64
07:45:31.607195 00:50:56:61:cb:93 > 00:50:56:6a:c1:90, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.222: ICMP echo request, id 36438, seq 123, length 64

NODE2# esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2019-09-03T07:02:40Z
Local Node UUID: 5cc8c87f-1ad4-5768-e1fe-20040ff07c0e
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5cc8c87f-1ad4-5768-e1fe-20040ff07c0e
Sub-Cluster Backup UUID:
Sub-Cluster UUID: 52694080-22bb-7ced-95fb-01b8e5a92f67
Sub-Cluster Membership Entry Revision: 0
Sub-Cluster Member Count: 1
Sub-Cluster Member UUIDs: 5cc8c87f-1ad4-5768-e1fe-20040ff07c0e
Sub-Cluster Member HostNames: NODE2
Sub-Cluster Membership UUID: 01106e5d-52e2-c75d-907e-20040ff07c0e
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: de0bb183-3c7e-418b-b652-258770e89e01 12 2019-08-19T09:12:12.1

NODE2# esxcli network ip interface ipv4 get
Name  IPv4 Address   IPv4 Netmask     IPv4 Broadcast  Address Type  Gateway        DHCP DNS
----  -------------  ---------------  --------------  ------------  -------------  --------
vmk0  10.12.132.247  255.255.255.224  10.12.132.255   STATIC        10.12.132.225     false
vmk2  192.168.1.222  255.255.255.0    192.168.1.255   STATIC        0.0.0.0           false
vmk3  192.168.2.20   255.255.255.0    192.168.2.255   STATIC        0.0.0.0           false

NODE1# esxcli network ip interface ipv4 get
Name  IPv4 Address   IPv4 Netmask     IPv4 Broadcast  Address Type  Gateway        DHCP DNS
----  -------------  ---------------  --------------  ------------  -------------  --------
vmk0  10.12.132.246  255.255.255.224  10.12.132.255   STATIC        10.12.132.225     false
vmk2  192.168.1.111  255.255.255.0    192.168.1.255   STATIC        0.0.0.0           false
vmk3  192.168.2.10   255.255.255.0    192.168.2.255   STATIC        0.0.0.0           false

Isolating 1 NIC shows 100 % packet loss:

NODE2# esxcli network nic list
Name    PCI Device    Driver   Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ------------  -----------  -----  ------  -----------------  ----  -----------------------------------------------------------------
vmnic0  0000:18:00.0  ntg3     Up            Up            1000  Full    20:04:0f:f0:7c:0c  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1  0000:18:00.1  ntg3     Up            Down             0  Half    20:04:0f:f0:7c:0d  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:19:00.0  ntg3     Up            Up            1000  Full    20:04:0f:f0:7c:0e  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:19:00.1  ntg3     Up            Down             0  Half    20:04:0f:f0:7c:0f  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:87:00.0  qedentv  Down          Down             0  Half    34:80:0d:0f:0a:2c  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic5  0000:87:00.1  qedentv  Up            Up           10000  Full    34:80:0d:0f:0a:2d  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
NODE2#vmkping -I vmk2 192.168.1.111 -c 100 -i 0.005
PING 192.168.1.111 (192.168.1.111): 56 data bytes

--- 192.168.1.111 ping statistics ---
100 packets transmitted, 0 packets received, 100% packet loss

Bringing up other NIC and making faulty down show packet is not lost by verifying it on esxtop command and selecting option "n"  to see association between NIC and vmkernel port.

NODE2# esxcli network nic up -n vmnic4

NODE2#esxcli network nic list
Name    PCI Device    Driver   Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ------------  -----------  -----  ------  -----------------  ----  -----------------------------------------------------------------
vmnic0  0000:18:00.0  ntg3     Up            Up            1000  Full    20:04:0f:f0:7c:0c  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1  0000:18:00.1  ntg3     Up            Down             0  Half    20:04:0f:f0:7c:0d  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:19:00.0  ntg3     Up            Up            1000  Full    20:04:0f:f0:7c:0e  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:19:00.1  ntg3     Up            Down             0  Half    20:04:0f:f0:7c:0f  1500  Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:87:00.0  qedentv  Up            Up           10000  Full    34:80:0d:0f:0a:2c  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter
vmnic5  0000:87:00.1  qedentv  Up            Up           10000  Full    34:80:0d:0f:0a:2d  1500  QLogic Corp. QLogic FastLinQ QL41xxx 1/10/25 GbE Ethernet Adapter

NODE2#esxcli network nic down -n vmnic5

NODE02# vmkping -I vmk2 192.168.1.111 -c 100 -i 0.005
PING 192.168.1.111 (192.168.1.111): 56 data bytes
64 bytes from 192.168.1.111: icmp_seq=0 ttl=64 time=0.148 ms
64 bytes from 192.168.1.111: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 192.168.1.111: icmp_seq=2 ttl=64 time=0.066 ms
64 bytes from 192.168.1.111: icmp_seq=3 ttl=64 time=0.072 ms
64 bytes from 192.168.1.111: icmp_seq=4 ttl=64 time=0.068 ms
64 bytes from 192.168.1.111: icmp_seq=5 ttl=64 time=0.061 ms

NIC was using latest driver.

NODE2#vmkload_mod -s qedentv
vmkload_mod module information
input file: /usr/lib/vmware/vmkmod/qedentv
Version: 3.9.31.2-1OEM.670.0.0.8169922
Build Type: release
License: QLogic_Proprietary
Required name-spaces:
com.vmware.vmkapi#v2_5_0_0
Parameters:

Workaround

Isolated the faulty NIC in the standard switch with the working NIC. Select "load balancing" setting to "Route based on originating port ID"

Moving the faulty NIC to standby.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

KB
•
誤ったアラームが VMware ESXi 6.5 および VMware ESXi 6.7 で実行している IPMI センサーからトリガーされる (76611)

KB
•
ESXi を 6.7 にアップグレードした後で Linux 仮想マシンが再起動する (70813)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSAN

Related Versions:

VMware vSAN 6.x

Take Our Survey

Support Assistant