Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

After NSX-v upgrade from 6.3.4 to 6.4.4 VMs running on vSAN clusters start reporting performance issues (70655)

Last Updated: 9/20/2019Categories: TroubleshootingTotal Views: 4155Language:
English

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Symptoms

Multiple production VMs report Network and Storage performance issues

High Write Latency observed in vSAN Cluster Performance Graphs

High ping latency observed between ESXi hosts in the problem clusters  for the VTEP's

Intermittent latency from VMK interfaces to immediate Layer-3 Physical gateway

BGP flaps reported intermittently between NSX edge and uplink switch

Cause

Root cause for this issue is the introduction of rule stats collection in 6.4.2 which is enabled by default where NSX DFW UI provides FW “Rule hit Stats” information.

All the events from NSX side which would result in firewall publish and eventually lead to stat collection and network latency

The rule stats feature gets triggered every 5 minutes. During this stage there is a PCPU lockup on datapath that causes the latency.
This feature can be disabled at Management plane (MP) but cannot be disabled at the dataplane till 6.4.4.

PCPU lockups gets triggered during rule publish event also and this will cause similar latency.

Graphical representation of DFW Rule stats Feature on NSX manager:

Log messages relevant to the issue

Below logs can be checked on impacted esxi hosts

From vsfwd logs we see a firewall publish event which acts as trigger for rule stats collection along with Rule stat hits which eventually triggers latency

File:-vsfwd.log, Location:- /var/run/log

2019-05-02T15:42:00Z vsfwd: [INFO] Applied RuleSet 1556795924921 for all vnics
2019-05-02T15:42:00Z vsfwd: [INFO] Compressed config data from 3130764 to 406478 bytes2019-05-02T15:42:00Z vsfwd: [INFO] Successfully saved config to file /etc/vmware/vsfwd/vsipfw_ruleset.dat

2019-05-02T15:42:03Z vsfwd: [INFO] Filter nic-81423981-eth0-vmware-sfw.2 has 16662 rule hit counts of gennum 1556795924921
2019-05-02T15:42:03Z vsfwd: [INFO] created rule hit count of filter nic-81423981-eth0-vmware-sfw.2
2019-05-02T15:42:06Z vsfwd: [INFO] Filter nic-81421968-eth0-vmware-sfw.2 has 16662 rule hit counts of gennum 1556795924921
2019-05-02T15:42:06Z vsfwd: [INFO] created rule hit count of filter nic-81421968-eth0-vmware-sfw.2
2019-05-02T15:42:09Z vsfwd: [INFO] Filter nic-81422860-eth0-vmware-sfw.2 has 16662 rule hit counts of gennum 1556795924921
2019-05-02T15:42:09Z vsfwd: [INFO] created rule hit count of filter nic-81422860-eth0-vmware-sfw.2

Note: The rule stats get initiated whenever there is a publish of rule happens at data plane.

From the hostd logs  we would see below triggers for PCPU locks:

File:–hostd.log ,  Location:- /var/run/log

2019-05-05T03:00:11.762Z cpu6:65575)WARNING: Heartbeat: 794: PCPU 5 didn't have a heartbeat for 7 seconds; *may* be locked up.
2019-05-05T03:00:32.763Z cpu7:856683)WARNING: Heartbeat: 794: PCPU 5 didn't have a heartbeat for 7 seconds; *may* be locked up.
2019-05-05T03:04:23.769Z cpu0:885462)WARNING: Heartbeat: 794: PCPU 7 didn't have a heartbeat for 8 seconds; *may* be locked up.

From vmkwarning logs  we observe TX hang

File:–vmkwarning.log, Location:- /var/run/log

$ grep -i "Tx hang" vmkwarning.*
vmkwarning.log:2019-04-28T10:29:48.929Z cpu46:65757)WARNING: i40en: i40en_SetResetFlags:10801: Tx hang detected, device resetting
vmkwarning.log:2019-04-28T10:30:00.136Z cpu74:65757)WARNING: i40en: i40en_SetResetFlags:10801: Tx hang detected, device resetting
vmkwarning.log:2019-04-29T04:58:51.730Z cpu56:65757)WARNING: i40en: i40en_SetResetFlags:10801: Tx hang detected, device resetting

Resolution

Rule stats collection has to be disabled end to end ( NSX manager and ESXI hosts) to avoid any PCPU lockup and subsequent latencies.

• NSX 6.4.4 does not allow to disable the feature locally at the ESXi host level, we can only disable stats collection at NSX Manager level.

• The fix to disable locally and globally is only made available in 6.4.5, the best way forward to avoid this issue is to plan an upgrade to 6.4.5.

• The impact on latency during rule stats collection is scheduled for a fix in the next release of NSX.  .

Workaround

Below workaround is recommended for NSX Manager and Hosts running 6.4.4

Steps to disable Rule Stats collection every 5 minutes from NSX Manager.

1. Take configuration backup of NSX Manager  using steps mentioned in the following link: Back Up NSX Manager Data

2.Get the config of "FirewallStats Collection" parameter at NSX manager using below API:

API:- GET https://{{MGR_IP}}//api/4.0/firewall/config/globalconfiguration

<globalConfiguration>
<layer3RuleOptimize>false</layer3RuleOptimize>
<layer2RuleOptimize>true</layer2RuleOptimize>
<tcpStrictOption>false</tcpStrictOption>
<ruleStatsDisabled>false</ruleStatsDisabled>
</globalConfiguration>

3.Disable the "Firewall Stats Collection" by changing parameter "ruleStatsDisabled" to true using below API request.

PUT https://{{MGR_IP}}//api/4.0/firewall/config/globalconfiguration

<ruleStatsDisabled>true</ruleStatsDisabled>

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
コンプライアンス スコアが、vRealize Operations Manager 7.5 の分析ノードによって異なる (70665)

KB
•
如何调整 vRealize Automation postgres 数据库 ebs_event 表的清除时间间隔 (70759)

KB
•
Enterprise PKS 与 vRealize Log Insight 集成后，发现 CPU 利用率很高 (70647)

KB
•
vCenter Server Appliance 6.5 的 VMDK 列表 - 大小、挂载点和用途 (70624)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

Related Versions:

Take Our Survey

Support Assistant