Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Replacing a faulty NSX-T manager node in a VCF environment (78967)

Last Updated: 10/28/2023Categories: How toTotal Views: 4153Language:
English

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Symptoms

The NSX cluster's health will be in degraded state when there is a faulty NSX manager node and can block several VCF operations.

Purpose

This article provides procedures for replacing a single node from a backup in a 3-node NSX-T Manager cluster within a VCF environment.

Resolution

Basic Info

Tested on VCF version 3.9.1 & 4.0

Applies to VCF 4.x

Procedure to replace one NSX-T manager node in 3-node cluster.

In this procedure, the 3-node NSX-T manager cluster has a single node down. In this example, there are 3x NSX-T manager nodes in the MGMT cluster:

172.17.110.22 vi1nsxmanager.dellrack1.vmware.corp ----> vip

172.17.110.23 vi1nsxmanager1.dellrack1.vmware.corp

172.17.110.24 vi1nsxmanager2.dellrack1.vmware.corp

172.17.110.25 vi1nsxmanager3.dellrack1.vmware.corp

Assume that vi1nsxmanager2 has gone down and must be replaced. You must obtain the following information for the node that needs a replacement.

2.1 Prerequisites

You must obtain the following information as prerequisites for the unrepairable NSX-T manager:

VM Name

FQDN hostname

IP address, netmask & gateway

DNS, NTP servers

admin, audit, root user passwords

If you do not know the admin and root passwords, follow the instructions in the password chapter of the VMware Cloud Foundation Operations and Administration guide to retrieve them from the SDDC Manager inventory. If you want to change these two passwords, should do so after restoring the NSX-T manager VMs and using the SDDC Manager password update function.

For the NSX-T manager cluster

NSX-T manager VM size. When you deploy a NSX-T Manager OVA, you must specify the node size.  To determine the size, log into the management domain vCenter, navigate to the summary page for one of the NSX-T manager VMs for this NSX-T instance, and expand the VM hardware pane. Compare the memory size and number of cpus of that VM to the NSX Manager VM Resource Requirements section in the the following product documents, and from the table, determine the size to use in step 2.2.4

NSX-T 3.0: Please refer to NSX Manager VM and Host Transport Node System Requirements.

NSX-T 2.5: Please refer to NSX Manager VM and Host Transport Node System Requirements.

For one operational NSX-T Manager:

FQDN hostname and IP address

Note: Make sure you download the OVA image before you start and check the md5sum of the OVA file as well. Section 2.2.4 lists the procedures for determining the specific OVA to download.

2.2 Procedure

2.2.1. Power off faulty NSX-T manager VM

From the vSphere Web Client > Hosts and Clusters, select vi1nsxmanager2 VM

Select Power Off

2.2.2. Delete faulty NSX-T manager VM

From vSphere Web Client > Hosts and Clusters, select vi1nsxmanager2 VM

Select Delete from disk

If you want to be safe, you can rename it instead, but remember to delete it once procedure is completed

2.2.3. Delete faulty NSX-T manager node from NSX-T manager cluster

From NSX-T manager UI > System > Overview, select 172.17.110.24, click the wheel icon, click Delete.

If the faulty NSX-T manager is the primary node, the wheel icon will not be available, delete it using the CLI as described below.

Note: Steps differ based on if the node is primary or secondary for deleting the manager

2.2.3.1. Delete a secondary node

The NSX-T manager UI will report error: compute manager could not find....

Click the wheel icon, there is a new option available, click Force Delete

2.2.3.2. Delete a primary node

First you must obtain the UUID of the faulty NSX-T manager, SSH into one operational NSX-T Manager

ssh admin@vi1nsxmanager3

Issue the get cluster status command, record the UUID of the faulty NSX-T manager

get cluster status

Issue the detach command:

detach node <uuid>

For example: detach node 77a01dab-58f1-4a86-8134-0bfc3e0c40d9

2.2.3.3. Wait for deletion complete

From NSX-T manager UI > Home > Dashboard > System, wait until 2-node NSX-T manager cluster has green stable status

2.2.3.4. Verification

After removing one node from the cluster and before adding a new one, make sure you run "get cluster status" command and verify the services are UP on the nodes.

2.2.4. Add new NSX-T manager node

The new node cannot be added from the NSX-T manager UI. The NSX-T manager VMs are in the MGMT cluster, but the NSX-T manager cluster only knows about the VI WLD vCenter, not the MGMT domain vCenter, so the add nodes wizard will not allow adding a node to the MGMT cluster.

From NSX-T manager UI > System > Overview, record the NSX version and build number of the NSX-T manager nodes

For example: 2.4.2.1.0.14374085

Download the exact build from customerconnect.vmware.com. e.g. nsx-unified-appliance-2.4.2.1.0.14374085.ova

From vSphere Web Client > Hosts and Clusters > SDDC-Cluster1 > Mgmt-ResourcePool, select Deploy OVF template...

Select local file, select the OVA file

nsx-unified-appliance-2.4.2.1.0.14374085.ova, click Next

Input the VM Name, vi1nsxmanager2

Select the SDDC-Datacenter, click Next

Select the SDDC-Cluster1, select the Mgmt-ResourcePool, click Next

Choose the correct size as determined in Section 2 above, click Next

Select the vSAN datastore, For example: sfo01-m01-vsan, click Next

Select the MGMT portgroup, For example: SDDC-DPortGroup-Mgmt, click Next

Input the information for the faulty NSX-T manager node

For example:
fqdn=vi1nsxmanager2.dellrack1.vmware.corp
role=nsx-manager-nsx-controller
gateway=172.17.110.1
ipv4=172.17.110.24
netmask=255.255.255.0
dns=172.17.110.251
domain=dellrack1.vmware.corp
ntp=172.17.110.251
ssh=enabled (checked)
allowroot=disabled (unchecked)

Review and click Finish

Wait for task completion

Power On the VM

2.2.5. Join the new NSX-T manager node to the cluster

Lookup the IP address of the NSX-T manager node nslookup vi1nsxmanager1

You can also ping vi1nsxmanager1 and it will display the IP address

SSH into an operational NSX-T manager node

ssh admin@vi1nsxmanager1

Get the cluster ID

get cluster config | find Id:

Get the API thumbprint

get certificate api thumbprint

SSH into the new NSX-T manager node

ssh admin@vi1nsxmanager2

join the new node to the cluster

join <vi1nsxmanager1 ip> cluster-id <uuid> thumbprint <thumbprint> username admin

For example:
join 172.17.110.23 cluster-id 3ca96913-7d42-4cce-a69c-365a7c52b545 thumbprint dd35cf8826bcb9cd6bd21deddb81a7447cc726fcfa393d71781d492a3302ca1e username admin

wait for 5 minutes for join to complete

2.2.6. Wait for addition complete

From NSX-T manager UI > Home > Dashboard > System, wait until 3-node NSX-T manager cluster has a green stable status.

2.2.7. Reassign old VMCA or external CA-signed certificate to the new NSX-T manager node.

2.2.7.1 If the certificate exists follow the below mentioned steps:

From NSX-T manager UI > System > Certificates > Certificates, check the VMCA/external CA-signed certificate for the faulty NSX-T manager node.

For example:
Issued By=CA
Issued To=vi1nsxmanager2.dellrack1.vmware.corp

Note that you should only check the checkbox, but do not open the detail screen

Hover the mouse over the ID field, the second column, a pop-up appears with the ID of the certificate

Record the certificate ID number, note this is the second column

For the next step, you require to send a POST request to the new NSX-T manager node, you can either use Postman in Windows or curl in linux, the example here will use curl in Linux.

Issue the POST request to the new NSX-T manager node to assign the old certificate to the new node

curl -H 'Accept: application/json' -H 'Content-Type: application/json'\
--insecure -u 'admin:<admin password>' -X POST\
'https://<new nsx-t mgr fqdn or ip>/api/v1/node/services/http?action=apply_certificate&certificate_id=<certificate id>'

For example:
curl -H 'Accept: application/json' -H 'Content-Type: application/json'\
--insecure -u 'admin:VMware123!VMware123!' -X POST\
'https://vi1nsxmanager2.dellrack1.vmware.corp/api/v1/node/services\/http?action=apply_certificate&certificate_id=24781ed5-7721-49bb-801d- cc8a4415d60e'

The curl command does not return a response, since the http server is restarted as part of the command.

From the vSphere Web Client > Host and Clusters, select the new NSX-T manager VM, click Restart Guest OS.

Wait 5 minutes for the node to reboot.

Specific to VCF 4.0: If assigning the certificate fails because the certificate revocation list (CRL) could not be verified, please follow the steps in KB 78794 to address the problem. If you decide to disable CRL checking in order to assign the certificate, re-enable CRL checking once the certificate has been assigned.

2.2.7.2 If the certificate does not exits, please follow Replace Expired or Self-signed NSX-T Manager Certificates with VMCA-Signed Certificates for more information.

2.2.8. Wait for reboot to complete

From NSX-T manager UI > Home > Dashboard > System, wait until 3-node NSX-T manager cluster has a green stable status

2.2.9. Verify VMCA cert of the new NSX-T manager node

This step requires openssl

Issue the openssl command to retrieve the certificate of the new NSX-T manager node

For example:
echo | openssl s_client -no_ign_eof -showcerts -connect \
vi1nsxmanager2.dellrack1.vmware.corp:443 > nsx2.pem

Issue the openssl command to display the certificate

For example:
openssl x509 -in nsx2.pem -noout -text | more

Verify the Issuer is psc-1

Verify the Subject CN=vi1nsxmanager2.dellrack1.vmware.corp

2.2.10. Refresh SDDC Manager SSH Key Store for VCF 4.0

This step is specific to VCF 4.0, as a final step, you need to update the SSH keys SDDC Manager saves for the NSX-T managers. VMware offers a script that automates this process. Please follow the Refresh SDDC Manager SSH Keys procedure documented in the KB 79004.

2.2.11 Update VM Anti-Affinity Rule

When Cloud Foundation deploys NSX-T Manager, it creates a VM anti-affinity rule to prevent the VMs of the same NSX-T Manager cluster from running on the same host. In this step, you need to add the newly deployed replacement VM to the rule for this NSX-T Manager cluster.
Log in to the management domain vCenter Server, and select Menu > Hosts and Clusters.
In the Navigator pane, select the management cluster
Select Configure > VM/Host Rules.
Finally, add the VM to the correct "separate virtual machine" rule. The rule for the management-domain NSX-T Manager cluster is named anti-affinity-rule-nsxt, while the rule for workload domains has the form "<NSXT Mgr VIP FQDN> - NSX-T Managers Anti Affinity Rule". Once you locate the rule, click edit, and add the newly deployed VM (e.g., vi1nsxmanager2) to it.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
将 Cloud Director 9.5 迁移到 10.0 的过程中，出现“无法读取或写入配置文件: 密钥库已被篡改，或密码不正确”错误 (78924)

KB
•
Download and Install VMware Fusion (2014097)

KB
•
VMware Update Manager の診断情報の収集 (1003693)

KB
•
VMware Intelligent Hub 20.04 for iOS リリースのお知らせ (78889)

Results 1-5 of 5

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Cloud Foundation

Related Versions:

VMware NSX-T Data Center 2.5.x

VMware NSX-T Data Center

VMware Cloud Foundation 4.x

VMware Cloud Foundation 3.9.x

VMware Cloud Foundation 3.8.x

Take Our Survey

Support Assistant