Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Using a built-in tool to perform a simultaneous shutdown/reboot of all hosts in the vSAN cluster (70650)

Last Updated: 9/12/2023Categories: How toTotal Views: 38041Language:
Chinese (Simplified)Portuguese (Brazil)JapaneseSpanishEnglish

18 like this article

Subscribe to get an email
newsletter with new updates

18 people found this helpful Please provide article feedback:

Purpose

This article provides information on how to safely perform a simultaneous shutdown/reboot of all hosts in the vSAN cluster. With ESXi 6.7 U3 (and newer), a built-in tool can be used to avoid the issue mentioned in Knowledge Base article A simultaneous reboot of all hosts in the vSAN cluster may result in data unavailability after a single failure.

Note: This solution is only applicable to a vSAN cluster where all hosts are on 6.7 U3 or later in a normal healthy cluster.

Resolution

Before starting the process of cluster-level shutdown/reboot of the hosts follow the steps below:

1) Check vSAN Health to confirm the cluster is healthy.
2) Disable HA to prevent the cluster registering hosts as failed.
3) Power down all VMs in the cluster running in the vSAN cluster. Unless vCenter Server is running on the cluster. If vCenter Server is hosted in the vSAN cluster, do not power off the vCenter Server VM.
4) For vSphere 7.0 U1 and later, to enable vCLS retreat mode. See vSphere Cluster Services (vCLS) in vSphere 7.0 U1.

Note: Ensure that all vCLS VMs have been completely removed from the cluster before moving on to the next step. This can be checked by selecting the vSAN Cluster > VMs Tab.

5) If vCenter Server is hosted in the vSAN cluster, Power off the vCenter Server VM. The vSphere Client becomes unavailable.
6) Run command 'esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates' on all hosts in the cluster.
7) Log in to one of the hosts in the cluster (except witness host).
8) Run the command below on only one host in the cluster (if it is run on multiple hosts concurrently then it may cause a race condition causing unexpected results)

python /usr/lib/vmware/vsan/bin/reboot_helper.py prepare

9) Wait until the command returns and prints 'Cluster preparation is done'.

Notes:
If an error occurred at this step, resolve the issue based on the error message and try step #3 again.
If there are unhealthy/disconnected hosts in the cluster, recover/remove them and retry.
Cluster will be fully partitioned after this step - this is expected behavior

10) Put all hosts into maintenance mode with 'No Action' mode.
11) Proceed with the reboot/shutdown.
12) When all hosts are back from the reboot/shutdown, exit all hosts from maintenance mode.

Note: If any host(s) failed to come up, manually recover the host(s) or move the bad host(s) out of the vSAN cluster.

13) Log in to one of the hosts in the cluster (except witness host).
14) Run the below command on only one host in the cluster (if it is run on multiple hosts concurrently then it may cause a race condition causing unexpected results)

python /usr/lib/vmware/vsan/bin/reboot_helper.py recover

15) Wait until the command returns and prints 'Cluster reboot/power-on is completed successfully!'
16) Run command 'esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates' on all hosts in the cluster.
17) For vSphere 7.0 U1 and later, If the vCenter Server is hosted in the vSAN cluster, wait for the vCenter Server VM to be powered up and running. To disable vCLS retreat mode. See vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1.
18) Re-enable HA.

Notes:
If there are unhealthy/disconnected hosts in the cluster, recover/remove them from vSAN cluster. Retry the above commands only after the vSAN health shows all available hosts are green state.

If the environment is a 3 node vSAN cluster, reboot_helper.py recover command will not work in 1 host failure situation.
In this case, Administrator should do following,
1. Temporary remove failure host information from unicastagent list
2. Add the host after running reboot_helper.py recover.

Below are commands to remove and add a host to a vSAN cluster, (please do not run the commands below if you are unaware/not familiar with them and open a ticket with VMware GSS vSAN team for help )
esxcli vsan cluster unicastagent remove -a <IP Address> -t node -u <NodeUuid>
esxcli vsan cluster unicastagent add -t node -u <NodeUuid> -U true -a <IP Address> -p 12321

If IPv6 settings on ESXi are not enabled, the following error may be logged when running reboot_helper.py
This is normal behavior and this error has no impact.

ERROR:root:Error to run _getIPRouteListFromEsxCLI

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)Portuguese (Brazil)JapaneseSpanishEnglish

Attachments

Additional Resources

KB
•
vRealize Automation postgres データベース ebs_event テーブルのパージ間隔を調整する方法 (70759)

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSAN

Related Versions:

VMware vSAN 7.0.x

VMware vSAN 6.7.x

Take Our Survey

Support Assistant