Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

vCenter Server Appliance disk space is full (76563)

Last Updated: 2/22/2024Categories: TroubleshootingTotal Views: 369018Language:
Chinese (Simplified)JapaneseEnglish

149 like this article

Subscribe to get an email
newsletter with new updates

149 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Details

This article provides information to troubleshoot vCenter Server Appliance (VCSA) high virtual disk space.

Errors appear as messages in vCenter Server similar to:

vSphere UI Health Alarm", "Log disk exhaustion on vcenter name"

Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name"

By default, the error messages begin when the disk reaches 80% full

Please note that in version 6.7, the /storage/archive disk can be up to 100% full by design

Earlier 6.7 versions may still present an error in the vCenter GUI. These can be safely ignored.

By default, when the some disks reach 95% full, the main vCenter service, vmware-vpxd, is turned off automatically

This is to protect the database from corruption.

Thus the problem may cause vCenter Server to be inaccessible with 503 Service Unavailable errors.

There are many possible causes for specific disk partitions running out of space. Some causes include:

Partitions are not sized correctly

Large files in the partitions

Tables in the database consuming space

Services are failing to clean up files

Solution

Identify the full partition

Through the vCenter GUI:

Log In to the vCenter Server Appliance Management Interface (VAMI) 

Select Monitor > Disks

Note which disks are showing 78% or higher usage

Please keep in mind that in 6.7 and higher versions, the "archive" disk being 80% or more full can be safely ignored 

Through the vCenter command line:

Log into the vCenter Server Appliance as root via SSH or the vCenter virtual machine Console.

Note: If you're unable to log in to vCenter Appliance and suspect this is due to the /dev/sda3 "root partition" being full, refer to the following article steps 1 to 7 to gain Single user mode shell access:  How to reset the lost or forgotten root password in vCenter Server Appliance 6.5 and above.

Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

Find out which partitions are full enough to trigger warnings. 78% or more full by running the command:

df -h

Here's an alternative command which only shows you the partitions which are getting full:

df -h |awk '0+$5 >= 78 {print}'

Note: We're looking for "78" because In many cases if the partition is at 78-79%, it may be exceeding the 80% needed to trigger warnings, then recovering back to the lower percentage. Similarly if it is at 93%, it may be exceeding the 95% needed to trigger vmware-vpxd service to  shut down, then recovering back to the lower percentage.

Select the article links for partitions that are displaying as full from this list

Filesystem

Mounted on

Resolution

/dev/sda3

"/"

/dev/sda3 - root partition 100% full due to Audit.log files not being rotated in vCenter Server Appliance

Root Partition full due to dnsmasq.log files in vCenter Server Appliance or Platform Service Controller

/dev/mapper/log_vg-log

/storage/log

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full

/dev/mapper/core_vg-core

/storage/core

/storage/core partition full due to core.netlogond.xxxxx files in vCenter Server Appliance

/dev/mapper/db_vg-db

/storage/db

/storage/db full or nearly full in VCSA

/dev/mapper/seat_vg-seat

/storage/seat

/storage/seat disk 100% full on vCenter Server Appliance 6.x/7.x

/dev/mapper/archive_vg-archive

/storage/archive

Note: This partition being full is normal and expected in vCenter versions 6.7 and above
"File system /storage/archive is low on storage space" warning in vCenter Server Appliance

Further troubleshooting:

If the previous links did not help with the problem, Identify large files in the partition

High disk space usage in vCenter Server Appliance starts with identifying which partition is impacted with high disk space usage.

Connect to vCenter Server Appliance through SSH or through the vCenter VM console

Type shell to switch to the BASH Shell. (for more information about enable or disable SSH ad Bash shell access, refer Enable or Disable SSH and Bash Shell Access

Use df -h command to get the disk space usage stats of the vCenter Server Appliance

Note: If you are not able to access vcsa bash shell, refer Unable to log in to the vCenter Server Appliance shell using root account even after password reset

Once the partition with low disk space is identified, below steps can be followed to identify the large files created in the respective partitions, except for the partitions store database files.

For partitions and mount points of vCenter Server Appliance, refer:

List of VMDKs for a vCenter Server Appliance 6.0 - Size Mount point and Purpose

List of VMDKs for a vCenter Server Appliance 6.5 - Size Mount point and Purpose

List of VMDKs/Partitions for a vCenter Server Appliance 6.7 - Size Mount point and Purpose

List of VMDKs/Partitions for a vCenter Server Appliance 7.0 - Size Mount point and Purpose

List of VMDKs/Partitions for a vCenter Server 8.0 - Size Mount point and Purpose

Use the below command within the affected partition to identify top 20 files with high disk usage:

du -a <Path> | sort -n -r | head -n 20

For example:

du -a /storage/log/ | sort -n -r | head -n 20

Change directories to any listed here which are suspiciously large, and look for large files using the command:

ls -lha

We can also do a more powerful search using the more complex command:
find ./ -type f -size +100M |less

If large files are not the issue it may be that numerous smaller files are causing a problem. The following command will show you the subdirectories containing the most files:

find ./ -type d -exec sh -c 'echo -n "{}: " && find "{}" -type f | wc -l' \; | awk '$2 > 100' | sort -k2,2nr |less

Please note, this command can take a few minutes to run!

Before deciding to remove any files to clear space, ensure you have a reliable backup for your vCenter.

Note: The *.tgz files are archived log bundle files and can be removed.

Resizing the vCenter virtual disks

If all else fails, it can be that normal activity on the vCenter has become greater than the existing disk size supports. You may therefore decide to resize the virtual disk:

Notice that in the VAMI the identity of the exact virtual disk to enlarge is shown in Monitor > Disks

Otherwise, please refer to the "List of VMDKs/Partitions" articles linked above

Please keep in mind that you cannot change disk size on the vCenter VM if there is a snapshot of the VM

It is critical to ensure you have a good backup of vCenter before resizing any virtual disks on it

Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.0

Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.5 and above

Related Information

Monitor and prevent vCenter Appliance disks from filling

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vCenter Server Appliance

VMware vCenter Server

Related Versions:

VMware vCenter Server Appliance 6.7.x

VMware vCenter Server Appliance 6.5.x

VMware vCenter Server Appliance 6.0.x

VMware vCenter Server 8.0

VMware vCenter Server 7.0.x

Take Our Survey

Support Assistant