Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

vSAN -- Maintenance -- Simultaneous Host Reboots / Cluster Full Shutdown -- Risk for Data Unavailability (60424)

Last Updated: 4/5/2024Categories: InformationalTotal Views: 53602Language:
Chinese (Simplified)JapaneseEnglish

14 like this article

Subscribe to get an email
newsletter with new updates

14 people found this helpful Please provide article feedback:

Symptoms

Performing cluster-level maintenance on a vSAN cluster using the "No Action/No data evacuation" maintenance mode feature followed by a reboot, may result in data unavailability post-maintenance in the event that a failure occurs during cluster start up or if the host is in a vSAN Decom state.

Examples of Issues/Failures :
- Disk(s) failures
- Any other hardware issues
- Host(s) are unable to join the Cluster, e.g. due to Network issues

The issue does not exist when :
- Using any other Maintenance Mode options than "No Action/No data evacuation"
- Rebooting the vSAN hosts by execution of "Rolling Reboots" (e.g. only putting a single vSAN host in Maintenance Mode at a time)

Note:
All VMs should be gracefully powered down prior to cluster-wide maintenance in this fashion, including vCenter Server.
Please disable vSphere HA and set vSphere DRS to Manual for the vSAN cluster if the vCenter Server is running out of the vSAN cluster and cannot be powered off.

Cause

Several vSAN Components of the same vSAN Object (= VM data) may be flagged as "Stale" as a result of placing multiple vSAN hosts in "No Action/No data evacuation" Maintenance Mode, followed by simultaneously rebooting multiple vSAN hosts.

Any failures/connection issues as described above, might result vSAN objects unable to switch from "Absent Stale" state back to a healthy state and therefore becomes unavailable.
This might result in some VM data become unavailable (= e.g. VMDKs which cannot be brought online).

Impact / Risks

As described above, there is an increased risk of Data unavailability when using vSAN Maintenance Mode "No Action/No data evacuation" on multiple vSAN hosts at the same time, followed by a simultaneous reboot of multiple vSAN hosts.

Resolution

General: To lower the risk of occurrence:
Before starting to place any vSAN host in Maintenance Mode and prior to shutting down all VMs :

- Please ensure that all VMs are compliant with their Storage Policy to ensure healthy and fully implemented data redundancy.
See here on how to Check Compliance for a VM Storage Policy

- Please ensure that your vSAN is healthy and no hardware failures exist.

You can validate this by running retest on the Skyline Health check:

See here on how to run vSAN Health check

To avoid issues:
- For vSAN hosts, 7.0U3 use Shutting Down and Restarting the vSAN Cluster

- For vSAN hosts 6.7 Update 3 (and newer up to 7.0U2) only:
A built-in tool can be used: See Knowledge Base article Using a built-in tool to perform simultaneous reboot of all hosts in the vSAN cluster

- For vSAN hosts below 6.7U3 use the Workaround

If the issue has occurred:
In the event the described issue occurs after the vSAN hosts were rebooted, affected vSAN objects may be recoverable by VMware Support.

Workaround

To be executed prior to placing any vSAN hosts in Maintenance Mode and rebooting them.

Before starting:
- Please ensure that all VMs are compliant with their Storage Policy to ensure healthy and fully implemented data redundancy.
See here on how to check (select installed Build):
Check Compliance for a VM Storage Policy

- Please ensure that your vSAN is healthy and no hardware failures exist.
You can validate this by running retest on the Skyline Health check:
See here on how to run (select installed Build):
Check vSAN Health

For vSAN Hosts below 6.7 Update 3:

Part 1:

Note:
The following steps need to be executed on all hosts in the vSAN cluster prior rebooting them simultanously.
The hosts do not need to be in Maintenance Mode to complete these steps :

1.  Make sure all the hosts in the cluster are NTP time synchronized.
For more information, see Configuring Network Time Protocol (NTP) on an ESXi host using the vSphere Web Client.

Note: It is very important that all the hosts need to be NTP time synchronized for this workaround to work.

2.  Download the scripts attached to this article (pre_reboot.sh & post_reboot.sh) to the vSAN host.
Place them on a persistent storage location except vSAN Datastore, so that they are available after reboot.
If there is no persistent storage other than the vSAN datastore available, have the scripts re-upload to the hosts.

Assign needed execute permission to run the scripts by using below commands via SSH/Putty (remove "$" prior running):

$ chmod +x pre_reboot.sh
$ chmod +x post_reboot.sh

3.) Take a backup of the vSAN network configuration by saving the output of the command "esxcli vsan network list".
Place the file on a persistent storage other than vSAN Datastore.
Example on how to save the output in a txt file:
esxcli vsan network list > vsan_network_list_backup.txt

Example of output on a regular Cluster:
In the following example "vmk0" is the VmkNic interface used for vSAN and the traffic type is "vsan".
Note: Depending on your configuration you may have more than one VmkNic interface configured for vSAN.

[root@sc-rdops-vm06-dhcp-174-97:~] esxcli vsan network list
Interface
VmkNic Name: vmk0
IP Protocol: IP
Interface UUID: e162115c-a0ed-26ab-fff3-02000ca1a627
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan

Example of output on a Stretched Cluster:
In addition to the output above, you might see a dedicated VmkNic configured for communication with Witness host. Here  "vmk1" is the VmkNic interface used for communication with Witness host (= traffic type: witness).

VmkNic Name: vmk1
IP Protocol: IP
Interface UUID: b2b43659-9ccf-018e-fcb8-a0369fdefe94
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: witness

4.  Based on Step 3:
Modify pre_reboot.sh file to disable vSAN traffic on all the related VmkNics found in Step 3 above by performing the following.
For each of the VmkNics, add the following command to the pre_reboot.sh file:
esxcli vsan network ip remove -i <VMkNic Name>

Example:
For the example configuration listed in Step 3 following commands need to be added to pre_reboot.sh script.
esxcli vsan network ip remove -i vmk0
esxcli vsan network ip remove -i vmk1

5.  Based on Step 3:
Modify post_reboot.sh file to re-enable vSAN traffic on all the related VmkNics found in Step 3 above by performing the following.
For each of the VmkNics, add the following command to the post_reboot.sh file.
esxcli vsan network ip add -i <VMkNic Name> -T=<Traffic Type>

Example:
For the example configuration listed in Step 3 following commands need to be added to post_reboot.sh script.
esxcli vsan network ip add -i vmk0 -T=vsan
esxcli vsan network ip add -i vmk1 -T=witness

6.  Create a CRON job that runs the "pre_reboot.sh" script at the exact same time on all of the vSAN hosts of the related cluster.
Note: Select a time in the future where all vSAN hosts will be available to run the script at the same time.

6.1) Create a backup of the current Crontab file:
Crontab file: /var/spool/cron/crontabs/root
Copy it to another non-vSAN Location.
Example:
cp /var/spool/cron/crontabs/root /vmfs/volumes/datastore1/root_crontab.BKP

6.2) Edit the current Crontab file:
Crontab file: /var/spool/cron/crontabs/root
Open it via command: vi /var/spool/cron/crontabs/root

Add the CRON job at the end of this file:
Example:
Note: The format to be used in the Crontab file is: #min hour day mon dow command
If we want "pre_reboot.sh" to be run on all hosts simultaneously on Dec 15, 20:30 UTC, we need to add the following line on each vSAN Host:
30   20   15  12  *   /vmfs/volumes/datastore/pre_reboot.sh

6.3) Stop and restart the current running CROND daemon on the vSAN host by
running these two commands  (remove "$" prior running):
$ kill -HUP $(cat /var/run/crond.pid)
$ /usr/lib/vmware/busybox/bin/busybox crond

7.  Repeat Steps 2-6 on the next vSAN host

8.  After the CRON job has been run on all vSAN Hosts (e.g. on Dec 15, 20:30 UTC), verify that all vSAN hosts "Node State" is "MASTER" via:
esxcli vsan cluster get | grep "Local Node State"

9.  Proceed with placing all vSAN hosts in Maintenance Mode with "No Action" via executing the following command on each host (remove "$" prior running):
$ esxcli system maintenanceMode set -e true -m noAction

10.  Proceed with rebooting of all vSAN hosts

Part 2:

Note:
The following steps need to be executed on all hosts in the vSAN cluster once all the vSAN hosts are back from the Reboot (= back online).

11.  Verify on all vSAN hosts that Node State is "MASTER" via:
esxcli vsan cluster get | grep "Local Node State"

12.  Check and ensure that all the vSAN disks are showing up as mounted i.e. with CMMDS status as "true" on each vSAN host  :
esxcli vsan storage list | grep "In CMMDS"
In CMMDS: true
In CMMDS: true
In CMMDS: true
In CMMDS: true

If you are seeing one more disks entries with "false", the host might be still initializing the related disk(s) or has encountered a disk issue.
You can check on any disk Issues by e.g. running the vSAN Healthcheck.
See here on how to run (select installed Build):
Check vSAN Health

13.  On all vSAN hosts: Exit the Maintenance Mode via below command :
esxcli system maintenanceMode set -e false

14.  Verify that all hosts are out of vSAN Maintenance Mode via:
esxcli vsan cluster get | grep "Maintenance Mode State"
Maintenance Mode State: OFF

15.  On each vSAN host: Ensure that the changes made to "post_reboot.sh" prior to the reboot are not lost (= Step 5).

16. Create a CRON job that runs the "post_reboot.sh" script to re-enable vSAN traffic on the VmkNic interfaces.
Note: Select a time in the future where all vSAN hosts will be available to run the script at the same time.

16.1) Create a backup of the current Crontab file:
Crontab file: /var/spool/cron/crontabs/root
Copy it to another non-vSAN Location.
Example:
cp /var/spool/cron/crontabs/root /vmfs/volumes/datastore1/root_crontab_Post_Reboot.BKP2

16.2) Edit the current Crontab file:
Crontab file: /var/spool/cron/crontabs/root
Open it via command: vi /var/spool/cron/crontabs/root
Remove the entry made via Part 1 - 6.2.
Add the new CRON job at the end of this file.
Example:
Note: The format to be used in the Crontab file is: #min hour day mon dow command
If we want "post_reboot.sh" to be run on all hosts simultaneously on Dec 15, 21:00 UTC, then we need to add the following line on each vSAN host:
00   21   15  12  *   /vmfs/volumes/datastore/post_reboot.sh

16.3) Stop and restart the current running CROND daemon on the vSAN host by
running these two commands  (remove "$" prior running):
$ kill -HUP $(cat /var/run/crond.pid)
$ /usr/lib/vmware/busybox/bin/busybox crond

17.  After the new CRON job has been run on all vSAN hosts (e.g. on Dec 15, 20:30 UTC),
verify that all objects are healthy (= no inaccessible Objects).
For that to be true the output of the following command needs to be empty:
cmmds-tool find -f python | grep -C5 CONFIG_STATUS | grep content | grep -v "state....7\|state....15"

18.  Edit the current Crontab file:
Crontab file: /var/spool/cron/crontabs/root
Open it via command: vi /var/spool/cron/crontabs/root
Remove the entry made via Part 2 - 16.2.

19.  Stop and restart the current running CROND daemon on the vSAN host by
running these two commands  (remove "$" prior running):
$ kill -HUP $(cat /var/run/crond.pid)
$ /usr/lib/vmware/busybox/bin/busybox crond

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

post_reboot.sh

pre_reboot.sh

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSAN

Related Versions:

VMware vSAN 8.0.x

VMware vSAN 7.0.x

VMware vSAN 6.x

VMware vSAN 6.7.x

VMware vSAN 6.6.x

VMware vSAN 6.5.x

VMware vSAN 6.2.x

VMware vSAN 6.1.x

VMware vSAN 6.0.x

view all

Take Our Survey

Support Assistant