Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Troubleshooting Elasticsearch related issues within vRealize Automation 7.x (71297)

Last Updated: 9/23/2021Categories: TroubleshootingTotal Views: 8855Language:
English

2 like this article

Subscribe to get an email
newsletter with new updates

2 people found this helpful Please provide article feedback:

Symptoms

vRealize Automation 7.3 through 7.6 contain a number of unassigned shards when manually executing the following health check command:

curl http://localhost:9200/_cluster/health?pretty=true

Note:  Anything other than a Green / OK status can cause unpredictable application behavior.

vRealize Automation 7.6 Virtual Appliance Management Interface Summary health page fails on Elasticsearch health check

Purpose

VMware implements internal health checks against the ElasticSearch service to maintain vRealize Automation 7.x application reliability as embedded VMware Identity Manager instances heavily leverage ElasticSearch in its normal application operations.

Common troubleshooting steps to restore the health of an ElasticSearch, single or multi-node, embedded cluster instance(s) within the vRealize Automation 7.x appliance(s) are contained within this article.

Cause

Datacenter network and storage outages can persist UNASSIGNED shards in a cluster overtime during ElasticSearch shard assignment tasks on cluster recovery.

Impact / Risks

The shard is the unit at which Elasticsearch distributes data around the cluster. The speed at which Elasticsearch can move shards around when rebalancing data, e.g. following a failure, will depend on the size and number of shards as well as network and disk performance.

Removing CLUSTER_RECOVERED and other stale and old UNASSIGNED shards has limited to no impact on a running cluster once removed.  If shards persist in UNASSIGNED for an extended period of time, unexpected application behavior may occur, to include a failure of the health status check for ElasticSearch.

Resolution

Restoring Green Status to ElasticSearch health checks in a vRealize Automation 7.x Single node or Multi-Node Cluster

SSH into the master vRealize Automation appliance.

Determine current health status:

curl http://localhost:9200/_cluster/health?pretty=true
Example:
{"cluster_name" : "horizon",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 10,
"active_shards" : 10,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0}

Note:  In the above command output, Elasticsearch cluster status can be: Red, Yellow, Green.
The health status will flag as Red, if there are a number of UNASSIGNED shards within the cluster.
Note:  Elasticsearch logs are located at /opt/vmware/elasticsearch/logs/horizon.log

Determine node name(s) registered within the cluster:

curl -s -XGET http://localhost:9200/_cat/nodes
Example: cava-n-84-170.eng.vmware.com 127.0.0.1 6   d * Red Skull II

If the command output from step 2 details more than zero UNASSIGNED shards, curl for further details on ALL shards piped to only UNASSIGNED:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED

Example:
% Total  % Received % Xferd  Average Speed  Time  Time  Time  Current  Dload  Upload  Total  Spent  Left  Speed
100   980  100   980    0     0  54444      0 --:--:-- --:--:-- --:--:-- 54444
searchentities 2 r UNASSIGNED CLUSTER_RECOVERED
searchentities 0 r UNASSIGNED CLUSTER_RECOVERED
searchentities 3 r UNASSIGNED CLUSTER_RECOVERED
searchentities 1 r UNASSIGNED CLUSTER_RECOVERED
searchentities 4 r UNASSIGNED CLUSTER_RECOVERED
v3_2019-07-17  4 r UNASSIGNED INDEX_CREATED
v3_2019-07-17  0 r UNASSIGNED INDEX_CREATED
v3_2019-07-17  3 r UNASSIGNED INDEX_CREATED
v3_2019-07-17  1 r UNASSIGNED INDEX_CREATED
v3_2019-07-17  2 r UNASSIGNED INDEX_CREATED

Determine if the UNASSIGNED shards can be assigned to another replica member with the Cluster Reroute function and allocate command:

Note:  ElasticSearch Cluster Reroute API Technical Documentation

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{"commands":[{"allocate":{"index":"searchentities","shard":0,"node":"Red Skull II","allow_primary":"true"}}]}'

Note:  The following response may occur if a valid copy of this shard already exists on the master:

shard cannot be allocated on same node [qAoqsUEITxuNbLXA6NASiA] it already exists on

If shards are orphaned and cannot be rerouted, attempt to cancel the replica shard:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{"commands":[{"cancel":{"index":"searchentities","shard":0,"node":"Red Skull II","allow_primary":"true"}}]}'

Determine if the UNASSIGNED shards fail to cancel and still persist by re-running Step #4.

Continue to Step #9 only if there are UNASSIGNED shards after the previous Steps, #1-6.

If the shards persist, delete them:

Note:  The below command will DELETE all UNASSIGNED shards from the ElasticSearch cluster.  It is recommended to first reallocate or cancel them first.

curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"

Verify that all UNASSIGNED shards have been deleted by rerunning Step #4.

Example:
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

Validate status returns "green":

curl http://localhost:9200/_cluster/health?pretty=true

Note:  The output should be showing 0 value for unassigned shards
Example:
{
"cluster_name" : "horizon",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}

Related Information

Elasticsearch, a search and analytics engine, used for auditing, reports, and directory sync logs, is embedded within the VMware vRealize Automation / Identity Manager virtual appliance. To verify the health of Elasticsearch, you must use the curl tool. If curl is not installed on the windows machine, you can query from a Linux or Mac machine to curl http://<localhost>:9200/_cluster/health?pretty

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
NSX-T ESXi トランスポート ノードのアップグレードが失敗する (75131)

KB
•
重新引导后，使用 cloud-init 自定义的虚拟机设置为 DHCP (71264)

KB
•
一括エクスポート要求が以下のエラーとともに失敗する：「エクスポートの作成中にエラーが発生しました。値を null にすることはできません。パラメータ名: stringToEscape (An error occurred while creating export: Value cannot be null.Parameter name: stringToEscape) (71263)

KB
•
VeloCloud SD-WAN Edge マルチキャストの構成および監視 (71223)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Aria Automation

Related Versions:

VMware vRealize Automation 7.x

Take Our Survey

Support Assistant