Loading

×Sorry to interrupt
CSS Error

Refresh

Skip to main content

Products and Accounts

Support

Knowledge

Communities

Success

Learning

My Subscriptions

Loading

Knowledge Base

ESX/ESXi hosts might experience read or write performance issues with certain iSCSI storage arrays (1002598)

Last Updated: 12/2/2021Categories: TroubleshootingTotal Views: 337820Language:
Chinese (Simplified)JapaneseEnglish

31 like this article

Subscribe to get an email
newsletter with new updates

31 people found this helpful Please provide article feedback:

Details

Some iSCSI storage arrays from different array vendors do not behave appropriately during periods of network congestion. This problem is related to the TCP/IP implementation of these arrays and can severely impact the read performance of storage attached to the ESXi/ESX software through the iSCSI initiator. The problem has also been reported in native (non-virtualized) environments involving the Microsoft iSCSI initiator.

Background Concepts

To understand this problem, you should be familiar with several TCP concepts:

Delayed ACK

Slow start

Congestion avoidance

Delayed ACK

A central precept of the TCP network protocol is that data sent through TCP be acknowledged by the recipient. According to RFC 813, "Very simply, when data arrives at the recipient, the protocol requires that it send back an acknowledgement of this data. The protocol specifies that the bytes of data are sequentially numbered, so that the recipient can acknowledge data by naming the highest numbered byte of data it has received, which also acknowledges the previous bytes.". The TCP packet that carries the acknowledgement is known as an ACK.

A host receiving a stream of TCP data segments can increase efficiency in both the network and the hosts by sending less than one ACK acknowledgment segment per data segment received. This is known as a delayed ACK. The common practice is to send an ACK for every other full-sized data segment and not to delay the ACK for a segment by more than a specified threshold. This threshold varies between 100ms and 500ms. ESXi/ESX uses delayed ACK because of its benefits, as do most other servers.

Slow start and Congestion avoidance

Congestion can occur when there is a mismatch of data processing capabilities between two elements of the network leading from the source to the destination. Congestion manifests itself as a delay, timeout, or packet loss. To avoid and recover from congestion, TCP uses two algorithms—the congestion avoidance algorithm and the slow start algorithm. Although the underlying mechanisms of these two algorithms differ, the underlying concept is that when congestion occurs, the TCP sender must slow down its transmission rate and then increase the rate as retransmitted data segments are acknowledged.

When congestion occurs, the typical recovery sequence for TCP/IP networks that use delayed ACK and slow start is:

The sender detects congestion because it did not receive an ACK within the retransmission timeout period.

The sender retransmits the first data segment and waits for the ACK before sequencing the remaining segments for retransmission.

The receiver receives the retransmitted data segment and starts the delayed ACK timer.

The receiver transmits the ACK when the delayed ACK timer times out. During this waiting period, there are no other transmissions between the sender and receiver.

Having sent the ACK, the sender then retransmits the next two data segments back to back.

The receiver, upon receiving the second data segments, promptly transmits an ACK.

The sender, upon receiving the ACK, retransmits the next four data segments back to back.

This sequence continues until the congestion period passes and the network returns to a normal traffic rate.

In this recovery sequence, the longest lag time comes from the initial delayed ACK timer in step 3.

Problem Description

The affected iSCSI arrays in question take a slightly different approach to handling congestion. Instead of implementing either the slow start algorithm or congestion avoidance algorithm, or both, these arrays take the very conservative approach of retransmitting only one lost data segment at a time and waiting for the host's ACK before retransmitting the next one. This process continues until all lost data segments have been recovered.

Coupled with the delayed ACK implemented on the ESXi/ESX host, this approach slows read performance to a halt in a congested network. Consequently, frequent timeouts are reported in the kernel log on hosts that use this type of array. Most notably, the VMFS heartbeat experiences a large volume of timeouts because VMFS uses a short timeout value. This configuration also experiences excessively large maximum read-response times (on the order of several tens of seconds) reported by the guest. This problem is exacerbated when reading data in large block sizes. In this case, the higher bandwidth contributes to network congestion, and each I/O is comprised of many more data segments, requiring longer recovery times.

In the vmkernel.log file, you may see entries similar to:

-09-17T15:07:19Z iscsid: discovery_sendtargets::Running discovery on IFACE iscsi_vmk@vmk2(iscsi_vmk) (drec.transport=iscsi_vmk)
-09-17T15:07:19Z iscsid: cannot make connection to 10.0.68.2:3260 (111)
-09-17T15:07:19Z iscsid: connection to discovery address 10.0.68.2 failed
-09-17T15:07:19Z iscsid: connection login retries (reopen_max) 5 exceeded
-09-17T15:07:19Z iscsid: Login Target Skipped: iqn.2003-10.com.lefthandnetworks:hvtiscsi:1808:vmwds01 if=iscsi_vmk@vmk1 addr=10.0.68.2:3260 (TPGT:1 ISID:0x2) (Already Running)
-09-17T15:07:19Z iscsid: Login Target Skipped: iqn.2003-10.com.lefthandnetworks:hvtiscsi:1808:vmwds01 if=iscsi_vmk@vmk2 addr=10.0.68.2:3260 (TPGT:1 ISID:0x3) (Already Running)

Solution

Until the iSCSI array vendors address this issue on their end, consider designing your IP storage network with enough capacity to account for the peak usage and lower the risk of congestion. If you experience the problem described in this article and are unable to alter your network configuration or find ways to guarantee a congestion-free environment, you can experiment with the following workaround.

This workaround involves disabling delayed ACK on your ESX/ESXi host through a configuration option.

Note: Please contact your storage vendor to confirm if disabling Delayed ACK is a workaround that they recommend for their storage array.

Differences in dealing with delayed ACK between ESX/ESXi 3.5 and 4.x/5.x/6.0

The way you disable delayed ACK varies slightly between ESX/ESXi 3.5 and 4.x/5.x/6.0.The main difference between ESX/ESXi 3.5 and 4.x/5.x/6.0 is in how the delayed ACK setting works and how it is set:

In ESX/ESXi 3.5, the delayed ACK setting is global to the ESX/ESXi system.

By setting this option, you completely disable delayed ACK regardless of whether TCP is attempting congestion recovery and operating normally. This option affects the entire TCP/IP stack and, therefore, all TCP/IP applications are impacted if you implement this change. As a consequence, setting this option might degrade the performance of iSCSI and other applications—for example, NFS and vMotion—as well as contribute to the congestion. Nonetheless, the option allows a speedy recovery from congestion and a reasonable performance during other transactions with the affected iSCSI arrays.

In ESX/ESXi 4.x/5.x/6.0, the recommended method for configuring the delayed ACK setting is on a per discovery target. However, you can configure the delayed ACK setting on a per individual target. Furthermore, though not recommended, you can configure the delayed ACK setting globally for ALL iSCSI targets.

Configuring Delayed ACK in ESXi.

To implement this workaround in ESXi, use the vSphere Client to disable delayed ACK.

Disabling Delayed ACK in ESXi 6.5, ESXi 6.7

Log in to the Web Client and select the host.

Right click the host and click Maintenance Mode.

Wait for the enter maintenance mode task to complete.

Navigate to the Configure tab.

Click Storage Adapters.

Click the iSCSI vmhba that you want to modify.

Modify the delayed ACK setting on a discovery address :

Click Targets tab on Adapter Details.

Click Dynamic Discovery.

Click Advanced Settings.

In the Delayed ACK Advanced Parameter options, uncheck inherit from parent and DelayedACK.

Click OK

Reboot the host.

Disabling Delayed ACK in ESX/ESXi 4.x, ESXi 5.x and ESXi 6.0

Log in to the vSphere Client and select the host.

Right click the host and click Maintenance Mode.

Wait for the enter maintenance mode task to complete.

Navigate to the Configuration tab.

Click Storage Adapters.

Click the iSCSI vmhba that you want to modify.

Click Properties.

Modify the delayed ACK setting, using the option that best matches your site's needs:

Modify the delayed ACK setting on a discovery address (recommended):

On a discovery address, click the Dynamic Discovery tab.

Click the Server Address tab.

Click Settings > Advanced.

In the Delayed ACK Advanced Parameter options, uncheck inherit from parent and DelayedACK.

Click OK.

Modify the delayed ACK setting on a specific set of targets:

On a discovery address, click the Static Discovery tab.

Select all targets that reside on the array you are working with.

Click Remove to delete the selected entries.

Click the Dynamic Discovery tab.

Remove all entries that reside on the array you are working with.

Enter the discovery address but do not rescan the adapter yet.

Select the discovery address to modify and click Settings > Advanced.

In the Delayed ACK Advanced Parameter options, uncheck inherit from parent and DelayedACK.

Click OK.

Repeat steps f-i for each discovery address that needs to be modified.

Modify the delayed ACK setting globally:

Select the General tab.

Click Advanced.

In the Delayed ACK Advanced Parameter options, uncheck inherit from parent and DelayedACK.

Click OK.

Reboot the host.

Re-enabling Delayed ACK in ESX/ESXi 4.x, ESXi 5.x and ESXi 6.0

Log in to the vSphere Client and select the host.

Navigate to the Advanced Settings page, as described in the preceding task Disabling Delayed ACK in ESX/ESXi 4.x, ESXi 5.x and ESXi 6.0.

Click Inherit From parent > DelayedAck.

Reboot the host.

Checking the Current Setting of Delayed ACK in ESX/ESXi 4.x, ESXi 5.x and ESXi 6.0

Log in to the vSphere Client and select the host.

Navigate to the Advanced Settings page, as described in the preceding task Disabling Delayed ACK in ESX/ESXi 4.x, ESXi 5.x and ESXi 6.0.

Observe the setting for DelayedAck.

If the DelayedAck setting is checked, this option is enabled. If you perform this check after you change the delayed ACK setting, but before you reboot the host, the result shows the new setting rather than the setting currently in effect.

Notes:

To disable delayed_ack, run this command from the command line:

vmkiscsi-tool -W -a delayed_ack=0 -j vmhbaXX

To enable delayed_ack, run this command:

vmkiscsi-tool -W -a delayed_ack=1 -j vmhbaXX

To check this parameter, run this command:

vmkiscsi-tool -W vmhbaXX

Configuring Delayed ACK in ESX/ESXi 3.5

To implement this workaround in ESX/ESXi 3.5, use the VI Client to alter the Net.TcpipDelayedAck advanced parameter setting. By default, this option is set to 1, which enables delayed ACK. To disable delayed ACK, perform the following steps.

Disabling Delayed ACK in ESX/ESXi 3.5

Log in to the VI Client and select the host.

Click the Configuration tab and click Advanced Settings.

Click Net and scroll through the advanced parameter list until you locate the Net.TcpipDelayedAck parameter.

Set the value for the parameter to 0.

Click OK.

Reboot the host.

Re-enabling Delayed ACK in ESX/ESXi 3.5

Log in to the VI Client and select the host.

Click the Configuration tab and choose Advanced Settings.

Choose Net and scroll through the advanced parameter list until you locate the Net.TcpipDelayedAck parameter.

Set the value for the parameter to 1.

Click OK.

Reboot the host.

Checking the Current Setting of Delayed ACK In ESX/ESXi 3.5

Log in to the VI Client and select the host.

Click the Configuration tab and choose Advanced Settings.

Choose Net and scroll through the advanced parameter list until you locate the Net.TcpipDelayedAck parameter.

Check the setting. A setting of 1 means that the delayed ACK is enabled and a setting of 0 means that it is disabled.

Click OK.

Note: A reboot is required for delayed ACK setting changes to take effect. If the host has not been rebooted after making a change, the result of this check shows the new setting rather than the setting currently in effect.

Additional Information

For translated versions of this article, see:

日本語: 特定のストレージ アレイを使用する ESX/ESXi ホストで読み取りまたは書き込みパフォーマンスの問題が生じる (2080851)

简体中文: ESX/ESXi 主机的某些存储阵列可能存在读取或写入性能问题 (2086182)

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

VMware ESXi 4.1.x Installable

VMware ESXi 4.1.x Embedded

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESXi 3.5.x Installable

VMware ESXi 3.5.x Embedded

VMware ESX Server 3.5.x

VMware ESX 4.1.x

VMware ESX 4.0.x

view all

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Cookie Preference Center

General Information on cookies

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Please confirm your settings by refreshing the page.

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Please confirm your settings by refreshing the page.

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Support Assistant