Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Excessively large flow table on NSX Edge leading to connection drops (78180)

Last Updated: 3/25/2021Categories: TroubleshootingTotal Views: 12571Language:
English

5 like this article

Subscribe to get an email
newsletter with new updates

5 people found this helpful Please provide article feedback:

Symptoms

Listing the concurrent connections on the NSX ESG shows an excessively large number: show flowstats

Total Flow Capacity: 1000000
Current Flow Entries: 543120
[...]

In the above example, there are 543,120 known flows on the NSX ESG, the capacity being at 1,000,000 flows. The figure should be compared to the expected traffic scale in the relevant environment.

Connections through the NSX ESG experience drops.

NSX Edge High Availability is enabled: show service highavailability

The NSX ESG is in version 6.4.0 or above.

A large amount of flows with packet counters at zero is recorded: show flowtable

123: tcp 6 2553 ESTABLISHED src=10.0.0.1 dst=10.0.0.2 sport=10001 dport=10002 pkts=0 bytes=0 src=10.0.0.2 dst=10.0.0.1 sport=10002 dport=10001 pkts=0 bytes=0 [ASSURED] mark=0 rid=0 use=1

In the above example, the TCP flow opened between 10.0.0.1:10001 and 10.0.0.2:10002 is known as ESTABLISHED, will time out in 2,553 seconds, but no traffic has been recorded.

A large amount of flows is marked with a TCP timeout value greater than the configured (maximum) TCP timeout.

To check the configured TCP timeout: show flowtimeouts

nf_conntrack_tcp_timeout_syn_sent = 30
nf_conntrack_tcp_timeout_syn_recv = 30
nf_conntrack_tcp_timeout_established = 21600
nf_conntrack_tcp_timeout_fin_wait = 20
nf_conntrack_tcp_timeout_close_wait = 60
nf_conntrack_tcp_timeout_last_ack = 30
nf_conntrack_tcp_timeout_time_wait = 30
nf_conntrack_tcp_timeout_close = 10
[...]

In the above example, the TCP timeout for Established flows is configured at 21,600 seconds.
To list the known flows: show flowtable

234: tcp 6 4253932 ESTABLISHED src=10.0.0.1 dst=10.0.0.3 sport=10011 dport=10013 pkts=0 bytes=0 src=10.0.0.3 dst=10.0.0.1 sport=10013 dport=10011 pkts=0 bytes=0 [ASSURED] mark=0 rid=0 use=1

In the above example, the TCP flow has a timeout of 4,253,932 seconds, so far greater than the configured TCP timeout.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

As part of the NSX Edge High Availability featureset, the synchronisation of the flow table from the active appliance to the standby appliance avoids the breaking of flows in case of HA failover. Indeed, when the formerly standby appliance becomes the newly active appliance, the flows are already known, so the stateful firewall is able to match the traffic to the known flows.
The flow table synchronisation leverages conntrackd. The intended behavior is for the flow table to be pushed from the active appliance to the standby appliance. The issue is introduced by an unintended bidirectional synchronisation, overwritting flow status on the active appliance from the standby appliance.

In associtation to this synchronisation issue, flows may get assigned a TCP timeout value that is greater than the configured value (TCP timeout is a decrementing value). This contributes to the growth of the flow table since flows may not time out in timely manner.

Empty flows, seen in the table with packet counter at 0, are expected in the following situations:

A Load Balancer service monitor configured on the NSX ESG may simply establish the flow to confirm the availability of the backend server, resulting in an intentendly empty flow.

When a flow is synchronised from the active appliance to the standby appliance, its traffic counter on the standby appliance is at 0 since the standby appliance does not participate in the datapath. If the flow is no longer used and a HA failover occurs, the empty flow may be seen as established on the newly active appliance, until it reaches its timeout.

Impact / Risks

This condition may result in fast growing flow table on the NSX ESG and eventually connection drops, essentially affecting workload performance in the environment.

Resolution

This issue is resolved in VMware NSX Data Center for vSphere 6.4.8, available at VMware Downloads .

Workaround

Disabling either of the below features, or both, removes the conntrackd synchronisation:

NSX Edge High Availability: This removes the need for a standby appliance, but reduces the level of redundancy of the NSX ESG.

NSX Edge Firewall: This removes the need to synchronise the flow table, but it prevents any traffic filtering to be done on the NSX ESG.

If neither of these options are possible, and to work around this issue, contact VMware Support and note this Article ID in the problem description. For more information, see How to Submit a Support Request.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

KB
•
CVE-2019-11477 および CVE-2019-11478 の Unified Access Gateway（および Access Point）に関する回避策 (70899)

KB
•
无法重新连接虚拟机的网络适配器：设备 "0" 的配置无效 (77134)

KB
•
Edge 使用中继逻辑交换机时，Edge 与 ESXi 之间的 BFD 隧道已关闭 (70745)

Results 1-5 of 5

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware NSX Data Center for vSphere

Related Versions:

VMware NSX Data Center for vSphere 6.4.x

Take Our Survey

Support Assistant