Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

(403 Forbidden)" error while accessing NSX Intelligence (76583)

Last Updated: 5/20/2020Categories: TroubleshootingTotal Views: 8889Language:
Chinese (Simplified)JapaneseEnglish

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

While accessing NSX Intelligence 1.0.0 from Discover & Take Action page in Plan & Troubleshoot tab in NSX-T Datacenter 2.5 using CA signed certificates, you see these errors similar to:

The application server is unable to fulfill your request due to insufficient privileges. You do not have the privileges to access NSX Intelligence. (403 Forbidden)

OR

Failed to load requested application. Please try refreshing the browser or contact support if the problem persists.

In the /var/log/kafka/server.log file of the NSX Intelligence appliance, you see entries similar to:

[2019-12-18 23:00:05,259] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR1_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-12-18 23:00:05,645] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR2_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-12-18 23:00:05,763] INFO [SocketServer brokerId=0] Failed authentication with /<NSX_MGR3_IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs due to an incomplete or incorrect certificate exchange with NSX Intelligence appliance when NSX Manager uses CA-signed certificates.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.0.0 with NSX Intelligence 1.1.0, available at VMware Downloads.

Workaround

To work around this issue if the NSX-T Manager is using the CA signed certificate:

For NSX-T Data Center 2.5.0 with NSX Intelligence 1.0.0

Step 1

Contact VMware Technical Support and reference this KB article (76583) requesting the FTP link to download the WAR files (proton and policy). For more information on contacting VMware support, see How to file a Support Request in Customer Connect (2006985).

Step 2

(a) Download nsxapi.war and policy.war files from the FTP link and copy to all the 3 NSX-T Manager nodes using root and do the following steps.

(i) Make a copy of the existing war files:

# cp /opt/vmware/proton-tomcat/webapps/nsxapi.war /root/nsxapi.war.firestar.bak
# cp /opt/vmware/policy-tomcat/webapps/policy.war /root/policy.war.firestar.bak

(ii) Stop the manager and policy services:

# systemctl stop proton
# systemctl stop nsx-policy-manager

(iii) For example, if the new nsxapi.war and policy.war files are under /tmp directory, use these commands to replace the existing war files:

# cp /tmp/nsxapi.war /opt/vmware/proton-tomcat/webapps/
# cp /tmp/policy.war /opt/vmware/policy-tomcat/webapps/
# chown uproton:uproton /opt/vmware/proton-tomcat/webapps/nsxapi.war
# chown uproton:uproton /opt/vmware/policy-tomcat/webapps/policy.war
# chmod 644 /opt/vmware/policy-tomcat/webapps/policy.war
# chmod 644 /opt/vmware/proton-tomcat/webapps/nsxapi.war

(iv) Restart the manager and policy services:

# systemctl start proton
# systemctl start nsx-policy-manager

(b) After the management cluster status is stable, obtain the pem_encoded field from the cluster certificate API:

(i) Use the following API GET:

GET https://{{NSX_MANAGER_VIP/IP}}/api/v1/trust-management/certificates?type=cluster_api_certificate

(ii) From the resulting JSON, copy the value for field pem_encoded, excluding the double quotes.

(d) Add the new certificate to the client truststore and kafka broker truststore on the NSX Intelligence appliance.

(i) Using SSH, log in to the NSX Intelligence appliance.

# ssh root@<nsx-intelligence-appliance-ip>
# export NEW_CERT_FILE=/root/new_cert.pem
# export HTTP_CERT_PWD_FILE=/config/http/.http_cert_pw
# export HTTP_CERT_PW=$(cat $HTTP_CERT_PWD_FILE)
# export CLIENT_TRUSTSTORE_FILE="/home/secureall/secureall/.store/.client_truststore"
# export KAFKA_TRUSTSTORE_FILE="/home/secureall/secureall/.store/.kafka_broker_truststore"

(ii) Paste in the new pem_encoded field from the JSON:

# echo -e "contents copied from step b(ii)" > $NEW_CERT_FILE

(iii) In case the cluster certificate is a CA signed certificate, there will be multiple "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" in $NEW_CERT_FILE. Split them into separate files. Skip this step if the cluster certificate is a self-signed certificate.

# csplit -f 'new_cert_' -b '%01d.pem' -z -s new_cert.pem -- '/-----BEGIN CERTIFICATE-----/' '{*}'
Typically there will be three or more new files generated. Export all file paths accordingly:

# export NEW_CERT_FILE_0=/root/new_cert_0.pem
# export NEW_CERT_FILE_1=/root/new_cert_1.pem
# export NEW_CERT_FILE_2=/root/new_cert_2.pem

(iv) If the cluster certificate is a CA signed certificate, insert all the certificates into client truststore and kafka broker truststore. The alias should be prefixed with nsx-manager-certificate.

# keytool -import -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt

If this command fails with error: "keytool error: java.lang.Exception: Certificate not imported, alias <nsx-manager-certificate mp-cluster 0> already exists", delete the existing certificate using the below command and run the above command again.

# keytool -delete -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
# keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 1" -file $NEW_CERT_FILE_1 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
# keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 2" -file $NEW_CERT_FILE_2 -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
# keytool -import -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt

If this command fails with this error: "keytool error: java.lang.Exception: Certificate not imported, alias <nsx-manager-certificate mp-cluster 0> already exists", delete the existing the certificate using below command and run the above command again.

# keytool -delete -alias "nsx-manager-certificate mp-cluster 0" -file $NEW_CERT_FILE_0 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt
# keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 1" -file $NEW_CERT_FILE_1 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt
# keytool -import -trustcacerts -alias "nsx-manager-certificate mp-cluster 2" -file $NEW_CERT_FILE_2 -keystore $KAFKA_TRUSTSTORE_FILE -storepass "$HTTP_CERT_PW" -noprompt

(v) Verify the certificates were successfully added:

# keytool -list -v -keystore $CLIENT_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt
# keytool -list -v -keystore $KAFKA_TRUSTSTORE_FILE -storepass $HTTP_CERT_PW -noprompt

(e) Restart proxy, kafka, nsx-config and processing services in the NSX Intelligence appliance.

# systemctl restart proxy
# systemctl restart kafka
# systemctl restart nsx-config
# systemctl restart processing

You should now be able to refresh the "Plan & Troubleshoot" page and view the flow information as before in the NSX-T Manager User Interface (UI).

Note: If the above workaround does not resolve the issue, you may have to delete the NSX Intelligence appliance using direct REST API.

Obtain the deployment ID (denoted by vm_id field in the API response) for the NSX Intelligence appliance:

GET https://<NSXMGR_IP>/api/v1/intelligence/nodes/deployments

Delete the NSX Intelligence appliance using the following API:

POST https://<NSXMGR_IP>/api/v1/intelligence/nodes/deployments/<deployment_id_from_prev_step>?action=delete

Deploy a new NSX Intelligence appliance and follow the steps in the KB above to resolve the issue.

For NSX-T Data Center 2.5.0 with NSX Intelligence 1.1.0.

Delete the failed deployment of NSX Intelligence.

Follow the steps above up-to Step 2 (a).

Redeploy NSX Intelligence 1.1.0.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
vCloud Director Appliance のインストールのベスト プラクティス (76559)

KB
•
vCloud Director 设备安装最佳实践 (76559)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

Related Versions:

VMware NSX-T Data Center 2.5.x

VMware NSX-T Data Center

Take Our Survey

Support Assistant