Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Failed or non-responsive NSX Controller VMs can safely be destroyed and redeployed via Web Client with no harm to VMware Cloud Foundation (56520)

Last Updated: 7/10/2020Categories: InformationalTotal Views: 5090Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

You see a failed workflow in the SDDC Manager UI and a failed or non responsive NSX Controller VM is the cause.

A NSX Controller VM is observed in a disconnected state from the controller cluster or failed state when navigating to Home >> Networking and Security in the web client.   Rebooting the controller does not resolve the issue with the controller.

All other attempts to resolve the NSX Controller VM issue fail.  See  NSX Controller Cluster Failures.

Purpose

Failed NSX Controller VM's can be manually deleted and redeployed quickly without impacting the VMware Cloud Foundation environment.

Resolution

Verify if you can resolve the issue as described in NSX Controller Cluster Failures.

Note:  Do not attempt the resolution while executing a SDDC manager workflows or workload domain expansions, deployments, etc unless workflows are in a failed state and no longer running.

The resolution is a 2 part process:  Delete the controller then Redeploy.

Delete an NSX Controller

You can delete an NSX Controller forcefully or gracefully. Graceful removal procedure checks for the following conditions before removing the node:

About this task

There is no current NSX Controller node upgrade operation.

The controller cluster is healthy, and a controller cluster API request can be processed.

The host state, as obtained from the vCenter Server inventory, shows connected and powered on.

This is not the last controller node.

Forceful removal procedure does not check the above mentioned conditions before removing the controller node.

Things to remember while deleting controllers:

Do not attempt to delete the controller VM before deleting it through the vSphere Web Client UI or API. When the UI is not usable, use the DELETE /2.0/vdn/controller/{controllerId} API to delete the controller.

After deletion of a node, ensure that the existing cluster stays stable.

When deleting all the nodes in a cluster, the last remaining node must be deleted using the Forcefully remove the controller option. Always verify that the controller VM is deleted successfully. If not, manually power down the VM and delete the controller VM using the UI.

If the delete operation fails, it means that the VM could not get deleted. In such case, invoke controller delete through UI with the Forcefully remove the controller option. For API, set the forceRemoval parameter to true. After forceful removal, manually power down the VM and delete the controller VM using the UI.

Since a multi-node cluster can only sustain one failure, deletion counts as a failure. The deleted node must be redeployed before another failure occurs.

For Cross-vCenter NSX environment:

Deleting the controller VM or powering it off directly in vCenter Server is not a supported operation. The Status column displays Out of sync status.

If controller deletion succeeds only partially, and an entry is left behind in the NSX Manager database in a Cross-vCenter NSX environment, use the DELETE api/2.0/vdn/controller/external API.

If the controller was imported through the NSX Manager API, use the removeExternalControllerReference API with the forceRemoval option.

When deleting a controller, NSX requests to delete a controller VM via vCenter Server using the Managed Object ID (MOID) of the VM. If vCenter Server cannot find VM by its MOID, NSX reports failure for the controller delete request and stops the operation.
If the Forcefully Delete option is selected, NSX do not stop the controller delete operation and will clear the controller's information. NSX also update all the hosts to no longer trust the deleted controller. However, if the controller VM is still active and running with a different MOID, it still has credentials to participate as a member of the controller cluster. Under this scenario, any logical switch or router that is assigned to this controller node will not function properly because the ESXi hosts no longer trust the deleted controller.

To delete the NSX Controller, perform the following procedure:

Procedure

Log in to the vSphere Web Client.

Click Networking & Security, and then click Installation.

In the NSX Controller nodes section, click the affected controller and take screen shots/print-screens of the NSX Controller Details screen (IP, port group, Cluster/Resource Pool, datastore, Host IP or name) or write down the configuration information for later reference.

Under Management, select the controller that you want to delete.

Click the Delete (x) icon.

Select either Delete or Forcefully Delete.

When you select the Forcefully Delete option, the controller gets deleted forcefully and not gracefully. This option ignores any failures and clears the data from database. You should verify that any possible failures are taken care of manually. You must confirm that the controller VM is successfully deleted. If not, you must delete it through vCenter Server.
Note: If you are deleting the last controller in the cluster, you must select the Forcefully Delete option to remove the last controller node. When there are no controllers in the system, the hosts are operating in what is called "headless" mode. New VMs or vMotioned VMs will have networking issues until new controllers are deployed and the synchronization is completed.

If you do not select this , the controller gets deleted gracefully.

Click Yes. Graceful controller deletion uses the following sequence:

Power off the node.

Check the cluster health.

If the cluster is not healthy, power on the controller, and fail the removal request.

If the cluster is healthy, remove the controller VM, and release the IP address of the node.

Remove the controller VM's identity from the cluster.
The selected controller is deleted.

Re-synchronize the controller state by clicking Actions > Update Controller State.

Redeploy an NSX Controller

Log in to the vSphere Web Client.

From Networking & Security, click Installation > Management.

Deploy a new NSX Controller node by clicking the Add Node (+) icon.

In the Add Controller dialog box, select the datacenter on which you are adding the nodes, and configure the controller settings.

Select the appropriate cluster.

Select a Host in the cluster and storage.

Select the distributed port-group.

Select the IP pool from which IP addresses are to be assigned to the node.

Click OK, wait for installation to complete, and ensure the node have a status of Normal.

Resynchronize the controller state by clicking Actions > Update Controller State.

Update Controller State pushes the current VXLAN and Distributed Logical Router configuration (including Universal Objects in a Cross-vCenter NSX deployment) from NSX Manager to the controller cluster.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Cloud Foundation

Related Versions:

VMware Cloud Foundation 2.3.x

VMware Cloud Foundation 2.2.x

Take Our Survey

Support Assistant