Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Smarts IP: domain crash - server was running out of file descriptors (fd) and which lead to the crash (520564)

Last Updated: 12/17/2018Categories: TroubleshootingTotal Views: 4321Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

- APM domain crash - server was running out of file descriptors (fd) and which lead to the crash
- Smart APM has crashed: Restarting it resolves for now

Cause

Below is the thread which was excepting http connections and crashed with the fata Exception. This server crashed while trying to allocate a file handler to an HTTP request.
The reason for this crash is the FDs (file descriptors) where getting exhausted.

-----
Thread 857 (Thread 140022979094272 (LWP 11994)):
#0  0x0000003ac960e7dd in read () from /lib64/libpthread.so.0
#1  0x00007f5a03712cb5 in _lineread (fd=4, buf=0x7f59a3edb870 "", bufsz=<value optimized out>) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:716
#2  0x00007f5a03712e6b in sm_POSIXstacktrace (fd=-1, stdinfo=<value optimized out>, line=466, file=0x7f5a03898608 "/work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c") at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:548
#3  0x00007f5a037138c8 in sm_LINUXstacktrace (fd=-1, stdinfo=1) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/misc/posix/POSIX_stackdump.c:466
#4  0x00007f5a032b382d in CI_Exception_::maybeTrace (this=0x7f59a3edda90, ctl=2) at /work/redcurrent/DMT-9.4.1.0/13/smarts/clsapi/ci_exception.c:111
#5  0x00007f5a032b4c5c in CI_ExFatal::throwThis (this=0x4) at /work/redcurrent/DMT-9.4.1.0/13/smarts/clsapi/ci_exception.c:762
#6  0x00007f59fbf14f41 in HttpServer::accept_work (this=0x7f59be090cb0, flow=0x7f59a4e5c420) at /work/redcurrent/DMT-9.4.1.0/13/smarts/http/sm_httpd.c:949
#7  0x00007f59fbf16d69 in CI_ClosureM1<HttpServer, CI_FlowTCP_U*>::apply (this=0xffffffffffffffff) at /work/redcurrent/DMT-9.4.1.0/13/smarts/install/linux_rhAS50-x86-64/optimize/include/clsapi/ci_api.h:1843
#8  0x00007f5a032abb37 in CI_Thread::_thread_run (arg=0x7f59b30f5210) at /work/redcurrent/DMT-9.4.1.0/13/smarts/clsapi/ci_thread.c:548
#9  0x00007f5a032a696a in _thread_run_wrapper (arg=0x4) at /work/redcurrent/DMT-9.4.1.0/13/smarts/clsapi/ci_thread.c:66
#10 0x00007f5a0370db9f in thread_wrapper (data=0x7f59b78a6fb0) at /work/redcurrent/DMT-9.4.1.0/13/smarts/platform/thread/posix/sthread.c:672
#11 0x0000003ac9607a51 in start_thread () from /lib64/libpthread.so.0
#12 0x0000003ac92e893d in clone () from /lib64/libc.so.6
-----

The server was running out of file descriptors and which lead to the crash. The maximum allowable file descriptors are 1024 and from below, it could be seen that the server was using the max utilization.
When the InCharge server exhausted the maximum fd allocated for a running process by the OS, further incoming request was rejected and server crashed gracefully.

-----
CI_FlowBufferedHead_U [Flow in negotiations Acceptor Buffered Head] HEAD|BUFFERED @0x00007f59a4ed2bc0
Read buffer, 0 bytes available of 2145
?3?0000000000000000 0000000000000000 <^|0000000000000000 0000000000000000
Write buffer, 0 bytes written of 2048
?3?[^0000000000000000 0000000000000000 0000000000000000 0000000000000000
->CI_FlowTCP_U [Flow in negotiations Accepted physical flow] IN_FLOW|PHYSICAL @0x00007f591686b0c0
*:v4:9701 KS N/A, KR N/A
Open fd=1026, conn April 13, 2018 1:18:20 AM GMT+02:00, disc N/A,
172.20.41.13:9701 -> 172.20.41.17:41687, tmo 0 00:00:15 N/S 1/0
-----

Also you can see here below that the server cannot accept too many incoming client connections.
------
[April 13, 2018 11:02:00 PM GMT+02:00 +933ms] t@2750277376 HTTP Accept
CI-F-ETHREAD-Thread HTTP Accept(#2750277376) threw exception
HTTP-EACCEPT-Too many accept errors
<SYS>-EMFILE-Too many open files; in file "/work/redcurrent/DMT-9.4.1.0/13/smarts/http/sm_httpd.c" at line 947
------

Below is an excerpt of the clients programs which is eating up the all the open file descriptors and cause the server to crash.

---------
Line 1972968: Thread 3119183616 (3119183616)  SM_ProtocolEngine-854930 1568 OpsMgr2007 ASL (restartable)
Line 1972970: Thread 3123386112 (3123386112)  SM_ProtocolEngine-854988 1568 OpsMgr2007 ASL (restartable)
Line 1972974: Thread 3091867392 (3091867392)  SM_ProtocolEngine-855227 1568 OpsMgr2007 ASL (restartable)
Line 1972980: Thread 3037636352 (3037636352)  SM_ProtocolEngine-855931 1556 OpsMgr2007 ASL (restartable)
Line 1972986: Thread 3014522624 (3014522624)  SM_ProtocolEngine-856280 1556 OpsMgr2007 ASL (restartable)
Line 1972990: Thread 3012421376 (3012421376)  SM_ProtocolEngine-856576 1556 OpsMgr2007 ASL (restartable)
Line 1972994: Thread 2985105152 (2985105152)  SM_ProtocolEngine-856761 1556 OpsMgr2007 ASL (restartable)
Line 1972996: Thread 2949601024 (2949601024)  SM_ProtocolEngine-857055 1556 OpsMgr2007 ASL (restartable)
----------

Resolution

Please check your client connections and have a logic to close these connections gracefully.
Also, increasing the FD limit won t help here since that depends on the limit provided by OS. The recommendation will be to keep it at 1024 only.

There is no fix for this issue as it s universally for any running process, the Operating system allocates a max number of file handles to 1024 and any application exhausting this limit would eventually crash and this behavior is expected and if the issue has to be stopped, you need to terminate each and every client connection gracefully.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

Related Versions:

VMware Smart Assurance - SMARTS

Take Our Survey

Support Assistant