Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Replace expired internal certificate in vRealize Operations 6.3-8.10 (71018)

Last Updated: 3/19/2024Categories: TroubleshootingTotal Views: 49468Language:
Chinese (Simplified)KoreanJapaneseEnglish

40 like this article

Subscribe to get an email
newsletter with new updates

40 people found this helpful Please provide article feedback:

Symptoms

The vRealize Operations internal certificate has expired.

This could be manifested by being unable to log into the Admin UI.

The cluster is Offline and you are unable to bring it Online, you see a message similar to:

"Data Retriever is not initialized yet. Please wait.".

The vRealize Operations internal certificate will expire soon.

Note: This article is not applicable or required for VMware Aria Operations 8.12 or later.

Cause

The internal certificate in vRealize Operations is generated upon initial deployment.
Currently, upgrading to later versions of vRealize Operations does not upgrade the internal certificate.

Note: It is not possible or supported to replace the internal certificate with a custom certificate.

Impact / Risks

The following instructions are for vRealize Operations 6.3 - 8.10.x.
There is no recovery option for vRealize Operations Manager 6.2.x and earlier.

Upgrading to vRealize Operations 8.6 also upgrades the internal certificate during the upgrade process, except for setups with Cloud Proxies connected.
The VMware vRealize Operations Certificate Renewal PAK file must be applied on vRealize Operations 8.6 only if the internal certificate has expired and if a Cloud Proxy node was connected to vRealize Operations before upgrade to 8.6.

If you are on vRealize Operations 8.5 or lower with Cloud Proxies connected, you must upgrade to vRealize Operations 8.6 before upgrading to vRealize Operations 8.10 or later.
If you are on vRealize Operations 8.5 or lower, and you do not have Cloud Proxies connected, you can upgrade directly to vRealize Operations 8.10 or later, within the bounds of your Upgrade Path.

After upgrading the vRealize Operations certificate by PAK file, any Cloud Proxies certificates need to be upgraded manually (vRealize Operations 8.4 or later).
This must be done by extracting the root certificate from vRealize Operations by using any browser and upload to Cloud Proxy by following Add CA certs while deploying a cloud proxy in vRealize Operations 8.4 or later.

Before following the resolution below, it is vital to snapshot the vRealize Operations nodes by following How to take a Snapshot of vRealize Operations.
Note: If the certificate has already expired, and the Admin UI is not accessible, steps 1, 2, 10, and 11 can be skipped on the above mentioned KB.

Note: Upgrading vRealize Operations is not a replacement for this article.  The steps below still must be followed.

Resolution

Identify if Certificate Renewal is Required

First, validate is the certificate renewal is required. If the certificate is not yet expired, the certificate can be checked from a Web Browser.  See the steps below for the most common web browsers:

Mozilla Firefox

Google Chrome

Microsoft Edge

Alternatively, if the certificate is expired or the UI is inaccessible, the certificate must be checked from the Primary node's command line.

Command Line

Note: Starting in vRealize Operations 8.0, a pop up is displayed in the UI, warning when certificate expiration will occur.

Mozilla Firefox

Open https://Primary_Node_IP_or_FQDN:6061.

Notes:

Replace Primary_Node_IP_or_FQDN with the actual IP or FQDN of the vRealize Operations Primary node.

The page displays a Warning: Potential Security Risk Ahead or Secure Connection Failed message; this is expected.

The Gemfire service must be running for a certificate to be presented.

No web page is expected to load, this is normal behavior; continue with the steps.

Click on Advanced and then on View Certificate.

Check the certificate end date under Period of Validity.

Google Chrome

Open https://Primary_Node_IP_or_FQDN:6061.

Notes:

Replace Primary_Node_IP_or_FQDN with the actual IP or FQDN of the vRealize Operations Primary node.

The page displays a Your connection is not private or This site can't provide a secure connection message; this is expected.

The Gemfire service must be running for a certificate to be presented.

No web page is expected to load, this is normal behavior; continue with the steps.

Click on Not secure in address bar then click on Certificate (Invalid).

Check the certificate end date under Valid From.

Microsoft Edge

Open https://Primary_Node_IP_or_FQDN:6061.

Notes:

Replace Primary_Node_IP_or_FQDN with the actual IP or FQDN of the vRealize Operations Primary node.

The page displays a This site is not secure message; this is expected.

The Gemfire service must be running for a certificate to be presented.

No web page is expected to load, this is normal behavior; continue with the steps.

Click on Certificate error in address bar then click on View certificate.

Check the certificate end date under Valid To.

Command Line

If the certificate is expired or the UI is inaccessible, the certificate must be checked from the Primary node's command line.

Log into the Primary node as root via SSH or Console.

Run the following command:

/bin/grep -E --color=always -B1 'java.security.cert.CertPathValidatorException: validity check failed|java.security.cert.CertificateExpiredException' $ALIVE_BASE/user/log/*.log | /usr/bin/tail -20

Note:

If step 2 returns nothing, certificate renewal is not yet required.

If step 2 returned output containing validity check failed, certificate renewal is required immediately.

Certificate Renewal

To renew the certificate, install the applicable pak file to generate a new internal certificate.
Depending on if the certificate has expired or not, choose the following applicable steps to install the PAK file.

Internal Certificate Not Expired

Internal Certificate Expired

Internal Certificate Not Expired

If the vRealize Operations internal certificate has not yet expired, install the vRealize Operations Certificate Renewal PAK file while the vRealize Operations cluster is in an Offline state.

Note: Ensure all of the following steps are completed on all nodes in the vRealize Operations cluster unless noted otherwise.

Snapshot the vRealize Operations nodes by following How to take a Snapshot of vRealize Operations.

Download the Certificate Renewal PAK file for your version of vRealize Operations from the VMware Patch Portal.

Select vROps Certificate Renewal as the Product.�

Select the vRealize Operations product version.

Notes:

For versions 6.3 to 8.1.1, select version 8.0.0.  This pak is compatible with vRealize Operations versions 6.3 to 8.1.1.

For versions 8.4.x to 8.10.x, select version 8.4.0.  This pak is compatible with vRealize Operations versions 6.4 to 8.10.x.

Log into all nodes in the vRealize Operations cluster as root via SSH or Console.

Log into the vRealize Operations Admin UI as the local admin user.

Click Take Offline under Cluster Status.

Note: Wait for Cluster Status to show as Offline.

Click Software Update in the left panel.

Click Install a Software Update in the main panel.

Follow the steps in the wizard to locate and install your PAK file.

Install the certificate renewal PAK file.

Wait for the software update to complete. When it does, the Administrator interface logs you out.

Note: If the cluster does not report the installation as Completed after a long time, compete the 4 steps listed just after step 14.

Log into the vRealize Operations Admin UI as the local admin user.

Clear the browser caches and if the browser page does not refresh automatically, refresh the page.

Click Bring Online under Cluster Status.

Note: The cluster status changes to Going Online. When the cluster status changes to Online, the upgrade is complete.

Run the following commands on all nodes in the vRealize Operations cluster:

chown admin:admin -R /storage/vcops/user/conf/ssl/ /storage/vcops/user/conf/ssl_bak/ /storage/db/casa/webapp/hsqldb/

chown -h root:root /storage/vcops/user/conf/ssl/web_cert.pem /storage/vcops/user/conf/ssl/web_chain.pem /storage/vcops/user/conf/ssl/web_key.pem

chmod guo+r -R /storage/vcops/user/conf/ssl/

chmod 444 /storage/vcops/user/conf/ssl/cacert.pem /storage/vcops/user/conf/ssl/slice_*_cert.pem

chmod 400 /storage/vcops/user/conf/ssl/cakey.pem /storage/vcops/user/conf/ssl/slice_*_cert.pfx /storage/vcops/user/conf/ssl/slice_*_key.pem

chmod 640 /storage/vcops/user/conf/ssl/tcserver.keystore

Note: For version 8.4 and later, also run the following commands on the Primary node and Primary Replica node (if present) and all data nodes:

chown postgres:root /storage/vcops/user/conf/ssl/postgres_vcopsrepl_*

chmod 600 /storage/vcops/user/conf/ssl/postgres_vcops_key.pk8 /storage/vcops/user/conf/ssl/postgres_vcopsrepl_key.pem

chmod 640 /storage/vcops/user/conf/ssl/postgres_vcops_cert.pem /storage/vcops/user/conf/ssl/postgres_vcopsrepl_cert.pem

If the admin UI after a long time does not report that installation of the Certificate Renewal PAK file as completed, complete the following steps.

Log into the Primary node as root via SSH or Console.

Run the following command to update the PAK installation status:

sed -i -e 's/\"initialization_state\"\:\"INITIALIZING\"/\"initialization_state\"\:\"NONE\"/g' /data/db/casa/webapp/hsqldb/casa.db.script

Repeat steps 1-2 on the Primary Replica node (if present).

Run the following command on the Primary and Primary Replica (if present) nodes to restart the CaSA service:

service vmware-casa restart

Internal Certificate Expired

If the vRealize Operations internal certificate has already expired, the vRealize Operations Certificate Renewal PAK file will need to be installed manually.  Complete the following steps on the vRealize Operations cluster while the cluster is in an Offline state.  If you are unable to take the cluster offline through the Admin UI, contact VMware Support.

Note: Ensure all of the following steps are completed on all nodes in the vRealize Operations cluster unless noted otherwise.

Snapshot the vRealize Operations nodes by following How to take a Snapshot of vRealize Operations.

Download the Certificate Renewal PAK file for your version of vRealize Operations from the VMware Patch Portal.

Select vROps Certificate Renewal as the Product.�

Select the vRealize Operations product version.

Notes:

For versions 6.3 to 8.1.1, select version 8.0.0.  This pak is compatible with vRealize Operations versions 6.3 to 8.1.1.

For versions 8.4.x to 8.10.x, select version 8.4.0.  This pak is compatible with vRealize Operations versions 8.4 to 8.10.x.

Copy the vRealize Operations Certificate Renewal PAK file to the /tmp/ directory on all nodes in the vRealize Operations cluster using an SCP utility.

Log into all nodes in the vRealize Operations cluster as root via SSH or Console.

Run the following command on all nodes in the vRealize Operations cluster to make the necessary directories:

mkdir -p /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted

Unzip the vRealize Operations Certificate Renewal PAK file by running the following command on all nodes in the vRealize Operations cluster:

unzip /tmp/vRealize_Operations_Manager_Enterprise_Certificate_Renewal-build.pak -d /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted

Note: Replace build with the build number of the downloaded vRealize Operations Certificate Renewal PAK file.
Example: unzip /tmp/vRealize_Operations_Manager_Enterprise_Certificate_Renewal-8.0.0.15217416.pak -d /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted

Stop all services by running the following commands:

service vmware-vcops-watchdog stop
service vmware-vcops stop

Note: For versions 8.3 and later, ensure that all services have been stopped by running the vrops-status command.
If there is a running service please kill it manually.

Example: (vpostgres) is running (3557)
Run this command to terminate the process: kill -9 3557

Check if the is_admin property is set only for the Primary node in casa.db.srcipt.

On all nodes (including Remote Collectors and Witness) run the following command to verify the status of the is_admin property:

sed -nre "/clusterMembership/ s/^[^']+'([^']+)','([^']+)'.*/\2/p" /storage/db/casa/webapp/hsqldb/casa.db.script | python -m json.tool

In the output ""is_admin_node": true" should only be set when the "slice_name": "MASTER".  If true is set for other nodes, complete the following on all nodes (including Remote Collectors and Witness):

Run service vmware-casa stop

Edit /storage/db/casa/webapp/hsqldb/casa.db.script and ensure "is_admin_node" is set to true for the Primary node, and false for all other nodes.

Run service vmware-casa start

The following command needs to be run in a particular order.  Follow each sub-step carefully.

Command: $VMWARE_PYTHON_BIN /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted/updateCoordinator.py EXPIRED

First, run the command on all Remote Collector nodes (if present) in the cluster, and wait for the task to complete.  Continue to step 8.2.

Next, run the command on all Data nodes, the Witness node (if present), and the Primary Replica node (if present) in the cluster; do not wait for each node to complete, just start the command on all nodes.  Once Waiting for certificate generation to complete appears on the last node, wait roughly 60 seconds, and continue to step 8.3.

Finally, run the command on the Primary node.

The expected behavior is for the command to finish, then shortly afterwards the pending tasks on the Data nodes and Primary Replica node (if present) will complete.

Note: To ensure that the command completes successfully check for the existence of the /var/vmware/_cert_generation_completed file on the Primary node.

Change newly generated certificates permissions on all nodes in the vRealize Operations cluster by running the following commands:

chown admin:admin -R /storage/vcops/user/conf/ssl/ /storage/vcops/user/conf/ssl_bak/ /storage/db/casa/webapp/hsqldb/

chown -h root:root /storage/vcops/user/conf/ssl/web_cert.pem /storage/vcops/user/conf/ssl/web_chain.pem /storage/vcops/user/conf/ssl/web_key.pem

chmod guo+r -R /storage/vcops/user/conf/ssl/

chmod 444 /storage/vcops/user/conf/ssl/cacert.pem /storage/vcops/user/conf/ssl/slice_*_cert.pem

chmod 400 /storage/vcops/user/conf/ssl/cakey.pem /storage/vcops/user/conf/ssl/slice_*_cert.pfx /storage/vcops/user/conf/ssl/slice_*_key.pem

chmod 640 /storage/vcops/user/conf/ssl/tcserver.keystore

Note: For version 8.4 and later, also run the following commands on the Primary node and Primary Replica node (if present) and all data nodes:

chown postgres:root /storage/vcops/user/conf/ssl/postgres_vcopsrepl_*

chmod 600 /storage/vcops/user/conf/ssl/postgres_vcops_key.pk8 /storage/vcops/user/conf/ssl/postgres_vcopsrepl_key.pem

chmod 640 /storage/vcops/user/conf/ssl/postgres_vcops_cert.pem /storage/vcops/user/conf/ssl/postgres_vcopsrepl_cert.pem

Log into the vRealize Operations Admin UI as the local admin user.

Click Bring Offline under Cluster Status.

If the cluster fails to go offline, click Force Offline under Cluster Status.

Note: Wait for the Cluster Status to show as Online.

Click Bring Online under Cluster Status.

Note: Wait for the Cluster Status to show as Online.

Related Information

Note: After the certificate renewal, vRealize Operations retains the previous key in it's truststore once the new certificate is generated.  vRealize Operations will use both the old and new certificate for validation.
Currently there is no revocation mechanism.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)KoreanJapaneseEnglish

Attachments

post_upgrade

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vRealize Operations Manager

VMware vRealize Operations

VMware vCenter Operations Manager

VMware vCenter Operations

Related Versions:

VMware vRealize Operations Manager 7.5.x

VMware vRealize Operations Manager 7.0.x

VMware vRealize Operations Manager 6.7.x

VMware vRealize Operations Manager 6.6.x

VMware vRealize Operations Manager 6.5.x

VMware vRealize Operations Manager 6.4.x

VMware vRealize Operations Manager 6.3.x

VMware vRealize Operations 8.x

VMware vRealize Operations 8.6.x

VMware vRealize Operations 8.5.x

VMware vRealize Operations 8.4.x

VMware vRealize Operations 8.3.x

VMware vRealize Operations 8.2.x

VMware vRealize Operations 8.10.x

VMware vRealize Operations 8.1.x

VMware vRealize Operations 8.0.x

view all

Take Our Survey

Support Assistant