Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

VMware vRealize Automation appliance reports /storage/log 100% full (70809)

Last Updated: 2/27/2024Categories: TroubleshootingTotal Views: 16412Language:
English

5 like this article

Subscribe to get an email
newsletter with new updates

5 people found this helpful Please provide article feedback:

Symptoms

VMware vRealize Automation (vRA) appliance /storage/log file system shows as 100% used.

Purpose

Explains the cause, troubleshooting steps, notes and explanations, possible resolution, workarounds  and recommendations on planning to avoid similar issues from happening in the future.

Cause

Two main scenarios that can cause filling of the/storage/log/vmware files system in are:

Improper configuration of log file rotation: If not configured properly, the log rotation can cause large sized files and also numerous log files, which overtime, can consume all the available space.

Verify the disk space usage for vcac, vco and horizon directories with the following commands from a SSH session/console:

du -h  /storage/log/vmware/vco

du -h /storage/log/vmware/vcac

du -h /storage/log/vmware/horizon

Creation of large Java Memory dump (.hprof ) files:

vRealize Orchestrator can run into an OutOfMemory exception based upon configuration and system usage which will dump an hprof file to disk.  This can subsequently cause a full drive.

Impact / Risks

When the log file system is full, newer log messages will not be properly recorded within the associated log files, (catalina.out, vcac-config.log, controlcenter.log, localhost_access_log.txt, horizon.log... etc)  which are required to be written to by the underlying applications and services.

The log directories mentioned above are located under the Linux root directory--filling up this volume can cause serious issues with system stability and services.

Resolution

Fix the configuration of log rotation, see examples here:

Persistent Log Growth: ...

Configuring log rotation by file size in vRA and vRO
Note! Open a Global Support Services Service Request if there are any additional questions or concerns regarding this action.

Allocate more space for the vRealize Automation log file system and/or for Java Heap dump (.hprof) files. Further details can be found in this KB: Increasing the memory allocated to a vRealize Orchestrator instance

Note: If a vco.hprof file is created repeatedly within /storage/log/vmware/vco/app-server this most likely indicates that vRO is under heavy load.  The first step should be to increase the Java heap size or add more space to the associated volume/disk.

Workaround

Delete the older log files to ensure there is available space for new log activity

Delete  /storage/log/vmware/vco/app-server/vco.hprof heap dump files and write a script file to delete the vco.hprof files periodically.

Note! hprof files are heap dumps in binary format, which can be used for detailed analysis of memory related problems in the Java stack.  These are safe to delete and are not actively in use by a healthy running system.
Note! A cron job can be created to run on a schedule to check for large *hprof files on the system to be removed.

Related Information

Notes and explanation:
1. Log files size and rotation configuration:
Depending on the level of server log and the scripting log the system is configured with, and how often these logs are generated, the file can grow in size substantially.
The default log level of the server log and the scripting log is INFO. Changing the log level affects all new messages that the server enters in the logs and the number of active connections to the database. The logging verbosity decreases in descending order.

Recommendations: Only set the log level to DEBUG or ALL to debug a problem. Do not use these settings in a production environment because it can seriously impair performance.
To prevent the server log from becoming too large, you can set the maximum file size and count of the server logs by modifying the values in the Max file count and Max file size (MB) text boxes.  Here is a document on how to limit the size of this log file. (https://kb.vmware.com/s/article/2119584)

The above document suggests to set the maximum size of the log file (Ex 5M). Once it exceeds the set size it will rotate 20 times and after that it compresses the old file and creates a new one.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
ロールアップ スクリプトの実行に時間がかかる場合に vCenter Server データベースのサイズを削減する (1007453)

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
对于混合 vSAN 环境中的某些 I/O 工作负载，vSAN 层中可能会出现内存不足的情况，从而导致 I/O 失败，并且 vSAN 会将目标磁盘组标记为永久错误状态 (70827)

KB
•
SPS 服务在 core.jsvc.xxx 文件中崩溃，从而导致/storage/core 在 vCSA 6.7 U2 中填满 (71170)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Aria Automation

Related Versions:

VMware vRealize Automation 7.x

Take Our Survey

Support Assistant