Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Troubleshooting and addressing accumulation of tombstones in a Platform Services Controller(embedded or external) (52387)

Last Updated: 11/8/2023Categories: TroubleshootingTotal Views: 19503Language:
English

3 like this article

Subscribe to get an email
newsletter with new updates

3 people found this helpful Please provide article feedback:

Symptoms

Multiple issues can occur if a Platform Services Controller has more than 100,000 tombstone entries, below this threshold the symptoms in this article are likely unrelated.

To determine the number of tombstone entries on a Platform Services Controller Appliance(embedded or external), run this command:

/opt/likewise/bin/ldapsearch -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' -b "cn=Deleted Objects,dc=vsphere,dc=local" -s sub "" -e 1.2.840.113556.1.4.417 dn| perl -p00e 's/\r?\n //g' | grep '^dn' | wc -l

Notes:

If the default SSO domain name has been changed, the dc value will need to be updated to match your environment.

If all nodes in the environment are Windows, a PSC appliance will need to be deployed to complete these steps. The appliance does not have to be connected to the current SSO domain to run the command above against a remote node.

Tombstone removal is recommended if the above command returns more than 100,000 tombstone entries.  Above this threshold, you may experience one or more of these symptoms:

A Platform Services Controller or vCenter Server deployment takes several hours to deploy.

A Platform Services Controller deployment fails with the error:

Firstboot script execution error

Could not configure the Service Control Agent administration group

Repointing a vCenter Server to another Platform Services Controller with the cmsso-util script intermittently fails during move-services.

Intermittent CPU spikes (150-400%) for the vmdir process.

The Appliance process name is vmdird.

The Windows process name is VMWareDirectoryService.

Slow logins (~3-5 minutes) to the Web Client with a PSC or domain account.

vCenter Server Appliance Upgrade or Migrations fail due to space constraints.

This issue is due to extra space consumed by the tombstones within vmdir.

vCenter Server Appliance Upgrade Fails during Stage 2 export with the error

"/storage/db/vmware-vmdir/data.mdb', '[Errno 28] No space left on device"

/var/log/firstboot/vmafd-firstboot.py_XXXX_stderr.log contains the following error message:

Error: [('/storage/db/cis-export-folder/vmafd/data/vmdir/data.mdb', '/storage/db/vmware-vmdir/data.mdb', '[Errno 28] No space left on device')]

Resolution

This is a known issue affecting vCenter Server 6.x.

vCenter Server 6.0 Update 3 and 6.5 Patch 1 includes new functionality to remove tombstone entries to resolve this issue. Tombstone entries cannot be removed prior to these releases. These patches include a code level change that gives permissions to the root account to remove tombstones. These permissions cannot be added without the code changes on prior releases.

Note: Tombstone removal does not replicate between Platform Services Controllers in a SSO domain and must be performed on all Platform Services Controllers.

Caution: Depending on the number of tombstone entries, removal can be very time consuming.  In some cases removal can take over 24 hours.  The Platform Services Controller will still be functional during tombstone removal.

To manually remove tombstones in vCenter Server 6.0 Update 3 and later

Connect to the Platform Services Controller using SSH and root credentials.

Run this command to enable access the Bash shell:

shell.set --enabled true

Type shell and press Enter.

Run this command to create the all.txt file:

/opt/likewise/bin/ldapsearch -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' -b "cn=Deleted Objects,dc=vsphere,dc=local" -s sub "" -e 1.2.840.113556.1.4.417 dn | perl -p00e 's/\r?\n //g' | grep '^dn' > all.txt

Run this command to remove all tombstone entries:

RULER="========"; while read line;do echo "$RULER$line$RULER"; time -p echo -e "$line\nchangetype: delete\n" | /opt/likewise/bin/ldapmodify -c -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' > /dev/null 2>/dev/null; done < all.txt | tee result.txt

Note:

If the default SSO domain name has been changed, then the dc value will need to be changed to match your environment.

If all nodes in the environment are Windows, a PSC appliance will need to be deployed to complete these steps. The appliance does not have to be connected to the current SSO domain to run the command above against a remote node.

To verify the tombstone entries have been removed, run this command:

/opt/likewise/bin/ldapsearch -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' -b "cn=Deleted Objects,dc=vsphere,dc=local" -s sub "" -e 1.2.840.113556.1.4.417 dn| perl -p00e 's/\r?\n //g' | grep '^dn' | wc –l

Resolve vmdir white space issues during upgrade or migration by increasing the partition size

Removing tombstones with the steps above does not cleanup whitespace in the vmdir database.  This can cause vCenter Server Appliance upgrades and migrations to fail.  There are two options to proceed with an upgrade if /storage/db runs out of space due to tombstones.

Increase the partition space of /storage/db to 20 GB on the destination vCenter Server Appliance before stage 2 of the upgrade.  For more information, see Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.0 or Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5

Use the mdb_copy command to remove the white space in vmdir:

The mdb_copy command only reduces white space after tombstone removal. If tombstones can not be removed because the Platform Services Controller is not on vSphere Update 3 or 6.5 Patch 1 or later, the only option  is to increase the partition size of /storage/db in step 1.

Steps for Linux

Download the attached 52387_Appliance_mdb_copy.gz.

Note:for 7.0 there is a new file mdb_copy

Stop the vmdird service with this command:

service-control --stop vmdird

Take a backup of the original /storage/db/vmware-vmdir/data.mdb file.

Compact the data.mdb database with this command:

./mdb_copy -c /storage/db/vmware-vmdir/ destination directory

Replace original data.mdb file with compacted one.

Start the vmdird service with this command:

service-control --start vmdird

Note: To stop and start services on the vCenter Server Appliance, see Stopping, starting, or restarting VMware vCenter Server Appliance 6.x services

Steps for Windows

Download the attached 52387_Windows_mdb_copy.gz.

Stop the VMWareDirectoryService service. For more information, see How to stop, start, or restart vCenter Server 6.x services

Take a backup of original C:\ProgramData\VMware\vCenterServer\data\vmdird/data.mdb file

Run the tool compact db,

C:\>mdb_copy.exe -c C:\ProgramData\VMware\vCenterServer\data\vmdird destination directory

Replace original data.mdb file with compacted one.

From the Windows UI, determine the size of the compacted data.mdb  file and add 500 MB to that number and record for later.

Open run program and type regedit.

Navigate to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VMWareDirectoryService\Parameters

Right-click in an open space and select New > DWORD (32-bit) Value.

Name the DWORD MaximumDbSizeMb.

Right-click on the DWORD and select Modify...

In the Value data filed enter the number determined in step 6.

For example, if the compacted data.mdb size is 1500 MB, then enter 2000 for this step.

Start the VMWareDirectoryService service.

Related Information

Tombstone FAQ

What are tombstones?
The vmdir process does not delete entries that have been removed, they are marked for deletion. These entries marked for deletion are known as tombstones.

Where are tombstones located?
Tombstones are found in the vmdir database that is used by Single Sign-on.  Vmdir databases are found on Platform Services Controllers.  This includes external Platform Services Controllers and vCenter Server with an embedded Platform Services Controller.

What generates tombstones?
Tombstone entries in vmdir will normally be caused by repointing vCenter between PSC nodes or unregistering a vCenter or PSC with cmsso-util.

What causes a large accumulation of tombstones?
The VSAN health service had a bug that caused frequent registration requests with Single Sign-On.  Each registration request generates four tombstones in the vmdir database.  This bug was introduced in vCenter Server 6.0 U2 and was resolved in vCenter Server 6.0 U3.  For more information, see vCenter Server 6.0 Update 2 displays on non-vSAN enabled ESXi hosts displays the message: Retrieve a ticket to register the vSAN VASA Provider

Why didn't the vmdir database size (/storage/db/vmware-vmdir/data.mdb) decrease after removing tombstones?
White space is never removed from the vmdir database.  If shrinking the vmdir database size is required, refer to the steps in the resolution section.

Do tombstones automatically delete over time?
Yes.  Automatic tombstone cleanup was added in vCenter Server 6.0 P06 and vCenter Server 6.5 U1.  Tombstone cleanup happens once a day and will remove any tombstones older than 45 days.  This tombstone removal will not clean up white space in the vmdir database.

Is it possible to remove tombstones before vSphere 6.0 Update 3 and 6.5 Patch 1?
No.  There is no workaround that allows tombstone removal.  The only option for tombstone removal is to upgrade to vSphere 6.0 Update 3 or 6.5 patch 1.  A code fix on these releases alters root level permissions which allows tombstone removal.  Performing the tombstone removal steps on any instance below these releases will result in no tombstones being removed.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

mdb_copy

52387_Appliance_mdb_copy.gz

52387_Windows_mdb_copy.exe.gz

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
将 VMware Cloud Foundation 升级到版本 2.3 后，“VDI 工作流：配置 Horizon View Cloud Pod 架构”的任务挂起 (52397)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vCenter Server

Related Versions:

VMware vCenter Server 7.0.x

VMware vCenter Server 7.0.3

VMware vCenter Server 7.0.2

VMware vCenter Server 7.0.1

VMware vCenter Server 6.5.x

VMware vCenter Server 6.0.x

Take Our Survey

Support Assistant