Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Troubleshooting VMware Identity Manager postgres cluster deployed through vRealize Suite Lifecycle Manager (75080)

Last Updated: 1/4/2024Categories: TroubleshootingTotal Views: 39198Language:
Chinese (Simplified)Portuguese (Brazil)JapaneseSpanishEnglish

17 like this article

Subscribe to get an email
newsletter with new updates

17 people found this helpful Please provide article feedback:

Symptoms

VMware Identity Manager cluster health status is shown as CRITICAL in vRealize Suite Lifecycle Manager Health Notification.

Purpose

Remediation of health issues associated with VMware Identity Manager clusters.

Resolution

From VMware vRealize Suite Lifecycle Manager 8.1 to 8.4, cluster health can be remediated by clicking on the Power ON button.

From VMware vRealize Suite Lifecycle Manager 8.5+, cluster health can be remediated by clicking on the REMIDIATE CLUSTER button.

Workaround

Prerequisites

Ensure vRSLCM's inventory is up-to-date. Sometimes there will be a drift/changes in the environment outside vRSLCM and for vRSLCM to know the current state of the system, its inventory needs to be updated. To do so, its advisable to perform an inventory sync of vIDM (global environment) in vRSLCM by clicking on the Trigger Inventory Sync button.

For the notification to be precise, vRSLCM should be able to reach all the vIDM nodes and also be able to log-in to all nodes to perform health checks.

Make sure all the vIDM nodes in the cluster are powered ON and running. If any of the nodes are powered OFF, then the health notification in vRSLCM would show a description message similar to

Host(s) <IPs> not reachable from vRSLCM.

Ensure vRSLCM is able to ssh into each vIDM node, if not a notification with a description message similar to

Unable to get postgres status from any of the nodes in the cluster.

and

Unable to get pgpool status

if postgres is unavailable from any of the nodes in the cluster.

Ensure vRSLCM has the right root passwords else the following note will be displayed

root passwords are expired

In any of these cases, ensure that vRSLCM can reach vIDM nodes and all the nodes credentials are updated in vRSLCM's inventory.

Note: vRSLCM does a ping to all the vIDM nodes to check if they are available or not.

Once inventory is updated, cURL for the current cluster health status

curl -H "Authorization: Basic token" -k https://vRSLCMFQDN/lcm/authzn/api/vidmcluserhealth

Syntax Help:
vRSLCMFQDN: The hostname / IP of vRSLCM managing the vIDM cluster.
token: Run the following command to get the Base64 encoded value of username:password. Here username is admin@local, and password is admin@local user's password.

echo -n 'admin@local:password' | base64

Note: On VCF mode replace admin@local with vcfadmin@local and its respective password.

Note: The API will trigger a request to re-calculate the cluster health which will generate a notification on the current overall status in vRSLCM.
Note: The automatic health check triggered by vRSLCM against a vIDM cluster is on a 1 hour default.

Remediate cluster health

Note: If the file /usr/local/etc/pgpool.pwd is present on any vIDM node, then while executing all the below steps replace the password with the one present in the file.
Note: If an install, upgrade or scale out request is IN PROGRESS or FAILED state in vRSLCM, Do not follow the below steps.

Ensure the pgpool service is running on all the nodes in the cluster

/etc/init.d/pgService status

Expected Response

Getting pgpool service status
pgpool service is running

Note: This is critical since the pgpool service is responsible for orchestrating the postgres fail-over. Without pgpool service running on a given node, quorom will not be maintained and the node will not be understood to be part of the cluster.

If the response is not as expected and pgpool service is not running, you will receive an error similar to

Pgpool service not running on the node(s) <IPs> in the cluster.

Start the service if it is not running

/etc/init.d/pgService start

To find the pgpool MASTER, run the below command on any of the vIDM nodes

su root -c "echo -e 'password'|/usr/local/bin/pcp_watchdog_info -p 9898 -h localhost -U pgpool"

Note: vIDM versions 3.3.2 and below will use the explicit string text of password in echo -e 'password'. For vIDM versions 3.3.3 and above, if /usr/local/etc/pgpool.pwd exists, use this value in "echo -e 'password'.

Command parameters help

-h : The host against which the command would be run, here its 'localhost'.
-p : Port on which pgpool accepts connections, which is 9898
-U : The Pgpool health check and replication delay check user, which is pgpool

Expected Response

3 YES <Host1>:9999 Linux <Host1> <Host1>

<Host1>:9999 Linux <Host1> <Host1> 9999 9000 4 MASTER
<Host2>:9999 Linux <Host2> <Host2> 9999 9000 7 STANDBY
<Host3>:9999 Linux <Host3> <Host3> 9999 9000 7 STANDBY

In the response, there needs to be a MASTER node present else vRSLCM will pop-up a notification with description message similar to

Could not find a pgpool master node, as all nodes are in standby state.

To check postgres status on all the vIDM nodes

Run the command on the pgpool MASTER node found above in Step #2, this will generate a list of all configured nodes with corresponding postgres status.

su root -c "echo -e 'password'|/opt/vmware/vpostgres/current/bin/psql -h localhost -p 9999 -U pgpool postgres -c \"show pool_nodes\""

Command parameters help

-h : The host against which the command would be run, here it would be 'localhost'
-p : The port on which Pgpool accepts connections, here its 9999
-U : The Pgpool user, which is pgpool
-c : The command to run, which is 'show pool_nodes'

Expected response

node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------
0 | Host1 | 5432 | up | 0.333333 | primary | 0 | false | 0 | 2019-10-14 06:05:42
1 | Host2 | 5432 | up | 0.333333 | standby | 0 | false | 0 | 2019-10-14 06:05:42
2 | Host3 | 5432 | up | 0.333333 | standby | 0 | true | 0 | 2019-10-14 06:05:42
(3 rows)

Role column - Informs postgres master node as known to pgpool service
Status column - Informs postgres status as known to pgpool service

Note: In an ideal scenario, there must be at least one postgres master node, if so, proceed to the next step.
Note: If none of the nodes are marked primary in the role column, vRSLCM will provide a notification with a message similar to Could not find a postgres master node, as all nodes are in standby state.

Warning:  This is a special case. Proceed go to the special cases section. Once complete, proceed to Step #4.

Ensure the delegate IP is properly assigned. This will be the free IP that user provided while clustering vIDM via vRSLCM for the delegate IP field.

vRLCM's postgres clustering solution assigns delegate IP to the postgres master node, so at any point in time the node which is marked master should hold the delegate IP. Check the role's column from the above show pool_nodes command response run on Step #3, for the postgres master node and ensure delegateIP is assigned on the same node.

Run the following command on the postgres master node, to make sure delegate IP is still assigned

ifconfig eth0:0 | grep 'inet addr:' | cut -d: -f2

Expected response

delegateIP Bcast

delegateIP: Make sure this is the same IP provided while clustering vIDM via vRSLCM for the delegate IP field. If so, skip this and continue to the step #5.

If the above command does not return any output, it would mean delegateIP is not assigned on the master postgres node. In such event, the health notification in vRSLCM would show a description message similar to

DelegateIP IP is not assigned to any nodes in the cluster.

Make sure the delegateIP is not held by any other non-master nodes by running above ifconfig command on other nodes. If any of the non-master nodes still have the delegateIP run the following command first to detach it

ifconfig eth0:0 down

Run the below command on the master node to re-assign the delegateIP

ifconfig eth0:0 inet delegateIP netmask Netmask

delegateIP - This is a keyword and need not be substituted with the actual IP
Netmask - Netmask currently configured for this node

Once the above command is successful, ensure the delegateIP is assigned to eth0:0 by running

ifconfig -a

and make sure eth0:0 is holding the expected delegateIP.

Note: Assigning the delegateIP explicitly needs a horizon-service restart on all the nodes. Run the following command to restart horizon service on all vIDM nodes

service horizon-workspace restart

From Step # 3's show pool_nodes response

If any of the vIDM nodes in the cluster are marked as down in the 'status' column, vRSLCM would pop-up a notification with description message similar to 'Postgres status of node(s) <IPs> marked as down in the cluster.'

This generally happens when the postgres on the node was down (or) the node went for a reboot (or) the node went out of network (or) even during a network glitch.

If the replication_delay column has a value greater then 0, a node is not able to sync its data from the current postgres master. In such case, vRSLCM would pop-up a notification with a description message similar to

Node(s) <IPs> have a replication delay with respect to the postgres master node.

In both the above cases, the nodes would not be syncing its data from the master node and will not be able to participate in further master election in case of failover. Run the below command to recover/catch-up with master on all such nodes. This command will allow the standby nodes to be in-sync with the postgres master node and will mark the node as up in the cluster.

Note: If replication delay has been corrected, the concerned node's postgres has to be brought down before performing the following command.

To stop postgres for VMware Identity Manager 3.3.2

service vpostgres stop

To stop postgres for VMware Identity Manager 3.3.3 or later

/etc/init.d/vpostgres stop

/usr/local/bin/pcp_recovery_node -h delegateIP -p 9898 -U pgpool -n node_id

Command parameter help

-h : The host against which the command would be run, its the delegateIP referring to the master node.
-p : Port on which PCP process accepts connections, which is 9898
-U : The Pgpool user, which is pgpool
-n : Node id which needs to be recovered. <node_id> will be the node that is being corrected. This can obtained from 'node_id' column from the show pool_nodes command.
delegateIP : This is keyword. Need not be changed with IP. Use as is.
pgpool : This is pgpool user. Need not be changed. Use as is.

The above command would prompt for a password. Enter Password as password.
Expected response

pcp_recovery_node -- Command Successful

Ensure to run the show pool_nodes command from Step # 3 to ensure all nodes are having the status column as up and the replication_delay column should have a value of 0.

Note: If the above command execution is not successful follow step 6.

Check if the file /etc/hosts in all the vIDM cluster nodes has the same master host entry,

If master host entry is found missing on any of the nodes or if it is inconsistent across the nodes, follow the below steps

Stop pgpool service on all the nodes (replica and then the master)

/etc/init.d/pgService stop

Check whether pgpool service is stopped, run the command

/etc/init.d/pgService status

Correct the master host entry in the file /etc/hosts.

Note: Master host can be found by executing the command in Step #2. Ensure master host entry should be the same in all the vIDM nodes in the cluster.

Start pgpool service on all the nodes (master and then the replica), run the command

/etc/init.d/pgService start

Check the status of the pgpool service, run the command

/etc/init.d/pgService status

Repeat Step # 5 to bring up the nodes in the cluster.

Once all the above checks and corrections are done, re-trigger the health check API mentioned in Step #3 of the Prerequisites section of this article to ensure the cluster health notification turns GREEN displaying a message similar to

vIDM postgres cluster health status is ok

Special cases

When all the nodes are marked standby in the role column from show pool_nodes command response in Step #3, it means there is no postgres master elected and it is a CRITICAL state. vIDM will not function. Choose one of the nodes as postgres master and the other nodes as replicas.

Remediation depends on the value present in status column. Choose any one node which is marked as up in status column. This will be the new postgres master being elected.

To promote the chosen node as postgres master, run the below command on the that node

/usr/local/etc/failover.sh <Old master node_id> <Old master node_hostname> 5432 /db/data <choosen node_id> <choosen node_hostname> <Old master node_id> <Old master node_hostname> 5432 /db/data

Command help

node_id and hostname can be obtained from the 'show pool_nodes' response in step #3
<choosen node_id> - The node_id of the choosen node, which is marked as 'up' in the 'status' column.
<choosen node_hostname> - The hostname of the current choosen node
<Old master node_id> - This will be the node_id of the previous postgres master node. If not known, this can be any one of the unavailable ones.
<Old master node_hostname> - This will be the hostname of the previous postgres master node. If not known, this can be any on of the unavailable ones.

Once the command completes, stop pgpool service on all the nodes. Now sequentially start pgpool service on the new postgres master chosen above, followed by other nodes

/etc/init.d/pgService stop
/etc/init.d/pgService start

Bring up the other nodes and follow the instructions present in Steps #4,5,6 to sync them against the new postgres master node.

Running remediation commands using script

Note: Remediation scripts work only if VMware Identity Manager is installed using vRealize Suite Lifecycle Manager 8.2 and above.

Download the utility script KB_75080_vidm_postgres_cluster_remediation.sh from the attachments to /root partition in vIDM- KB_75080_vidm_postgres_cluster_remediation.sh

Change file permission of the script by executing the below command
Command: chmod 777 KB_75080_vidm_postgres_cluster_remediation.sh

Execute the below command to find the command options which are executable using the utility script
Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --help

To see the pgpool status of vIDM cluster nodes or to find the pgpool master execute the below command
Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --show_pgpool_status

To see the postgres status of vIDM cluster nodes execute the below command
Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --show_postgres_status

To recover/ bring up the vIDM nodes in the cluster execute the below command
Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --pcp_recovery <node_id>
Command parameters help :
<node_id> will be the node that is being corrected. This can obtained from 'node_id' column from the show pool_nodes command.

Related Information

Note: These are involved steps, if you encounter issues with the procedure, please open a support ticket referencing this kb number and the area in which you are stuck and this can be evaluated.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)Portuguese (Brazil)JapaneseSpanishEnglish

Attachments

KB_75080_vidm_postgres_cluster_remediation

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Workspace ONE Access

VMware Identity Manager

VMware Aria Suite Lifecycle

Related Versions:

VMware vRealize Suite Lifecycle Manager 8.x

Take Our Survey

Support Assistant