Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

vRNI 3.8 may cause NSX controller nodes to run out of memory leading to VM connectivity loss (57311)

Last Updated: 8/25/2018Categories: TroubleshootingTotal Views: 3935Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

Connectivity loss is experienced in the environment for one or more VMs in Logical Switches or for VMs utilizing the NSX Distributed Logical Router.

One or more NSX controller nodes become disconnected from NSX Manager.

You are unable to log-in to one or more NSX controller nodes using SSH.

You see ‘Out of memory’ errors displayed on the console of one or more NSX controller nodes.

CPU usage of NSX controller nodes appears high.

NSX controller nodes may be reporting ‘Disk Alerts’ in the NSX Installation section of the vSphere Client.

In the vRNI UI, you may see the following NSX alerts:

Event code 1930: 2 Controllers not able to establish connection with one controller.

No NSX Controller majority.

Less than three active NSX Controllers.

The /var/log/kern.log contains oom-killer messages similar to the following on impacted controller nodes:

2018-08-07T21:40:03.380316+00:00 nsx-controller kernel: [2795201.231270] xxxxxxxxxxxxx invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
2018-08-07T21:40:03.380339+00:00 nsx-controller kernel: [2795201.231274] xxxxxxxxx cpuset=/ mems_allowed=0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

vRNI 3.8 polls the NSX controllers for information using the NSX Manager ‘Central CLI’ and via SSH connections. Under certain conditions, SSH sessions are not closed by vRNI causing memory depletion on the controllers. When this happens, impacted NSX controllers become disconnected and a cluster majority may no longer be maintained. This can lead to dataplane impact of VMs on Logical Switches as well as to VMs utilizing the Distributed Logical Router.

Notes:

vRNI 3.7 and older releases are not impacted by this issue.

vRNI deployments that are not configured to poll NSX controllers are not impacted by this issue.

Resolution

This is a known issue affecting VMware vRealize Network insight 3.8 with all versions of NSX for vSphere.

This issue is resolved in VMware vRealize Network Insight 3.8.0 Patch P4-201808101430. For more information on the patch and how to apply it, please see VMware KB 57683.

Please subscribe to this article by clicking on "Subscribe to Article" in the actions box to get notified on the latest updates.

Workaround

To work around this issue, reboot any NSX controller nodes that have experienced memory exhaustion. After reboot, they should re-join the cluster majority automatically and return to normal function. After rebooting, verify the following:

All NSX controllers are connected and green in the Installation section of the NSX UI.

All NSX controllers have joined the cluster by running the show control-cluster status command from the CLI of each controller node.

If VM connectivity problems persist, a force-sync of routing and VXLAN services should be done.

Until the patch can be applied, VMware recommends disabling controller node polling in vRNI 3.8. To do this, use the following process:

From the vRNI settings page, click 'Accounts and Data sources'.

Click 'Edit NSX Manager(s)'.

Un-check the 'Enable NSX Controller Data Collection' checkbox.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vRealize Network Insight

VMware NSX for vSphere

Related Versions:

VMware vRealize Network Insight 3.x

Take Our Survey

Support Assistant