Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

VMware Cloud Foundation VRM VM vrm-tcserver log file catalina.out growing to fill root filesystem (51940)

Last Updated: 10/16/2020Categories: InformationalTotal Views: 13225Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

Most VMware Cloud Foundation operations fail due to the / filesystem being almost full.

In the /home/vrack/vrm/logs/lcm.log file of the SDDC Manager VM, you see entries similar to:

Nov 20, 2017 5:07:33 PM : Upgrade scheduled,
Nov 20, 2017 5:15:12 PM : Upgrade status changed to INPROGRESS,
Nov 20, 2017 5:15:18 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e status changed to INPROGRESS,
Nov 20, 2017 5:15:18 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e recorded stage ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS,
Nov 20, 2017 5:15:20 PM : Successfully ran upgrade stage ESX_HOST_UPGRADE_STAGE_MAKE_VIM_CONNECTIONS,
Nov 20, 2017 5:15:20 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e recorded stage ESX_HOST_UPGRADE_STAGE_PRECHECK,
Nov 20, 2017 5:15:27 PM : Successfully ran upgrade stage ESX_HOST_UPGRADE_STAGE_PRECHECK,
Nov 20, 2017 5:15:27 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e recorded stage ESX_HOST_UPGRADE_STAGE_DISABLE_DRS_RULES,
Nov 20, 2017 5:15:27 PM : Successfully ran upgrade stage ESX_HOST_UPGRADE_STAGE_DISABLE_DRS_RULES,
Nov 20, 2017 5:15:27 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e recorded stage ESX_HOST_UPGRADE_STAGE_TAKE_BACKUP,
Nov 20, 2017 5:15:27 PM : Upgrade element resourceType: ESX_HOST resourceId: 1e03e170-e056-4401-9180-e2c4d78d897e status changed to COMPLETED_WITH_FAILURE,
Nov 20, 2017 5:15:28 PM : Upgrade element resourceType: ESX_HOST resourceId: 89225694-7261-45a1-9dd8-aec90a268d09 status changed to SKIPPED,
Nov 20, 2017 5:15:28 PM : Upgrade element resourceType: ESX_HOST resourceId: 98dcbd68-e9f0-46a8-b592-044036d662d0 status changed to SKIPPED,
Nov 20, 2017 5:15:28 PM : Upgrade element resourceType: ESX_HOST resourceId: d32311da-32fc-4c01-b663-f9f76e1487a5 status changed to SKIPPED,
Nov 20, 2017 5:15:28 PM : Upgrade status changed to COMPLETED_WITH_FAILURE,

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

In the sos.log file of the SDDC Manager VM, you see these entries during an sos log bundle creation::

rack-1-vrm-1:/home/vrack/bin # /opt/vmware/evosddc-support/sos

Welcome to Supportability and Serviceability(SoS) utility!

Logs : /var/tmp/sos-2017-11-21-13-12-11-30296

Log file : /var/tmp/sos-2017-11-21-13-12-11-30296/sos.log

Progress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeProgress : 43%, Completed tasks : [AUDIT, ISVM-CASSANDRA, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter ServeTimeout on command: "/bin/bash -c "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email-masked] 'cd /;tar -ohcvf - opt/vrack/hms/config opt/vrack/hms/logs opt/vrack/upgrade var/log --exclude='temp*.zip'' > /var/tmp/sos-2017-11-21-13-12-11-30296/rack-1/hms.tar ; gzip -f /var/tmp/sos-2017-11-21-13-12-11-30296/rack-1/hms.tar""

Progress : 50%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, ESX, SWITCH, HEALTH-CHECK, vCenter Progress : 60%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, ESX, SWITCH, LCM, HEALTH-CHECK, vCeProgress : 66%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, ESX, NSX, SWITCH, LCM, HEALTH-CHECKProgress : 66%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, ESX, NSX, SWITCH, LCM, HEALTH-CHECKProgress : 66%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, ESX, NSX, SWITCH, LCM, HEALTH-CHECKProgress : 73%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, LCM, HEALTH-Progress : 80%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, HMS-ESX-NODEProgress : 80%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, HMS-ESX-NODEProgress : 80%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, HMS-ESX-NODEProgress : 80%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, HMS-ESX-NODE[Errno 28] No space left on device

Progress : 80%, Completed tasks : [AUDIT, ISVM-CASSANDRA, HMS, ISVM-ZOOKEEPER, PSC, ESX, NSX, SWITCH, HMS-ESX-NODETraceback (most recent call last):r]
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 885, in emit
self.flush()
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 845, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file base.py, line 218
Traceback (most recent call last):
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 885, in emit
self.flush()
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 845, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file base.py, line 220
Traceback (most recent call last):
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 885, in emit
self.flush()
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 845, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file base.py, line 132
Traceback (most recent call last):
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 885, in emit
self.flush()
File "/opt/vmware/lib/python2.7/logging/__init__.py", line 845, in flush
IOError: [Errno 28] No space left on device

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This occurs because the /home/vrack/vrm/logs/catalina.out file grows until the / (or root) filesystem is filled, or it is manually (re)moved.

Resolution

This is a known issue affecting VMware Cloud Foundation. Currently, there is no resolution.

Workaround

To workaround this issue on the VRM (2.1) or SDDC Manager Controller (2.2) VM, relocate the /home/vrack/vrm/logs/catalina.out file to another system.

To automatically rotate the catalina.out (vrm-tcserver service) log file, add it to the logrotate process configuration:

:SSH to the VRM VM (192.168.100.108) as the root user.

SSH to the VM as the root user:

for VMware Cloud Foundation 2.1, use the VRM VM (192.168.100.108).

​For VMware Cloud Foundation 2.2, use the SDDC Manager Controller VM (192.168.100.40).

Create the /etc/logrotate.d/tomcat file using the following command:

echo -e "/home/vrack/vrm/logs/catalina.out {\n\tcopytruncate\n\tdaily\n\trotate 7\n\tcompress\n\tmissingok\n\tsize 500M\n}" > /etc/logrotate.d/tomcat

Verify the file was created properly with:  cat /etc/logrotate.d/tomcat  command. The output looks similar to:

​/home/vrack/vrm/logs/catalina.out {
copytruncate
daily
rotate 7
compress
missingok
size 500M
}

Set the proper permissions on /etc/logrotate.d/tomcat file using the following command:

chmod 644 /etc/logrotate.d/tomcat

Either wait until 0:05 UTC for the /etc/cron.daily/logrotate cron job to run automatically, or manually run logrotate using the following command:

/usr/sbin/logrotate /etc/logrotate.conf

About the logrotate configuration for tomcat (created in step 2 and shown in step 3):

/home/vrack/vrm/logs/catalina.out - the (log) file to be rotated

daily - rotates the catalina.out daily

rotate – keeps at most 7 log files

compress – compresses the rotated files

size – rotates if the size of catalina.out is greater than 500 MB

Related Information

This is a known issue with a variety of VMware Appliances using Tomcat for web services (using various log file locations), including the VCSA 6.0 as described in /storage/log directory is full in vCenter Server Appliance 6.0.

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
创建保护组时报错：PreCreateCheckCallback 期间出现 PBM 错误："连接被拒绝" (53707)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Cloud Foundation

Related Versions:

Take Our Survey

Support Assistant