Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Controlling LUN queue depth throttling in VMware ESXi (1008113)

Last Updated: 3/21/2024Categories: How toTotal Views: 171807Language:
English

10 like this article

Subscribe to get an email
newsletter with new updates

10 people found this helpful Please provide article feedback:

Symptoms

If the ESX host detects a queue full condition, it may fail the SCSI commands

Queue Full may show up as QFULL or Task Set Full state

If QFULL conditions exist, the ESX VMkernel log may contain entries similar to:

In ESX/ESXi 4.0 and later:

H:0x0 D:0x28 P:0x0 Valid sense data: 0x## 0x## 0x##

H:0x0 D:0x08 P:0x0 Valid sense data: 0x## 0x## 0x##

The hexadecimal 28 in the error is the SCSI status code for the queue full state. The value 0x08 in the above error is the SCSI Status code indicating a device busy state.

In ESX/ESXi 4.0 and later, you also see a device busy error

Resolution

VMware ESX 3.5 Update 4 introduces an adaptive queue depth algorithm that adjusts the LUN queue depth in the VMkernel I/O stack. This algorithm is activated when the storage array indicates I/O congestion by returning a BUSY or QUEUE FULL status. These status codes may indicate congestion at the LUN level or at the port (or ports) on the array. When congestion is detected, VMkernel throttles the LUN queue depth. The VMkernel attempts to gradually restore the queue depth when congestion conditions subside.

This algorithm can be activated by changing the values of the QFullSampleSize and QFullThreshold parameters. When the number of QUEUE FULL or BUSY conditions reaches the QFullSampleSize value, the LUN queue depth reduces to half of the original value. When the number of good status conditions received reaches the QFullThreshold value, the LUN queue depth increases one at a time.

Note: Careful consideration is needed if multiple hosts access the same LUN or array ports. For the adaptive queue depth algorithm to be effective, all hosts accessing the LUN/port must have some form of adaptive queue depth algorithm. If some hosts run the adaptive queue depth algorithm while other hosts do not, the hosts that are not running the algorithm may consume the resources/slots on the array that are freed up by the adaptive hosts. This causes the hosts running the algorithm to exhibit lower disk I/O throughput. This may also increase the I/O congestion that initially triggered the adaptive algorithm.

If hosts running operating systems other than ESX/ESXi are connected to array ports that are being accessed by ESX/ESXi hosts, and the ESX/ESXi hosts are configured to use the adaptive algorithm, either make sure the operating systems use an adaptive queue depth algorithm or isolate those hosts on different ports on the storage array.

VMware ESX versions 3.5-5.0

In ESX/ESXi versions 3.5-5.0, QFullSampleSize and QFullThreshold are system-wide configuration parameters. Even if the system has different vendor storage arrays connected, all the arrays experience the throttling effect if one LUN/port from any array returns the QUEUE FULL or BUSY errors.

To enable the algorithm:

Use the vSphere Client or vSphere Web Client to navigate to the Configuration tab of the VMware ESX host you want to modify.

Click Advanced Settings under the Software section.

Click Disk in the left side pane.

Set QFullSampleSize to a value greater than zero. The usable range is 0 to 64.

For 3PAR, NetApp and IBM XIV storage arrays, set the QFullSampleSize value to 32.

For other storage arrays, contact your storage vendor.

Set QFullThreshold to a value lesser than or equal to QFullSampleSize. The usable range is 1 to 16.

For 3PAR storage arrays, set the QFullThreshold value to 4.

ForNetApp and IBM XIV storage arrays, set the QFullThreshold value to 8.

For other storage arrays, contact your storage vendor.

The settings take effect immediately. You do not need to reboot the ESX/ESXi host.

VMware ESXi 5.1

In ESXi releases earlier than ESXi 5.1, QFullSampleSize and QFullThreshold are set globally, that is, on all devices seen by the ESXi host. In VMware ESXi 5.1, these parameter are not set globally because different vendors have different optimal values for their arrays. You set QFullSampleSize and QFullThreshold on a per-device level.

Run the following ESXCLI command:.
esxcli storage core device set --device device_uid --queue-full-threshold
Q --queue-full-sample-size S

For example:

esxcli storage core device set --device device_uid --queue-full-sample-size 32 --queue-full-threshold 4

Settings are persistent across reboots.
You can retrieve the values for a device by using the corresponding list command.

esxcli storage core device list

The command supports an optional --device parameter.

esxcli storage core device list --device device

The recommended values are the same as in earlier releases.
QFullSampleSize:

For 3PAR, NetApp and IBM XIV storage arrays, set the QFullSampleSize value to 32.

For other storage arrays, contact your storage vendor.

QFullThreshold:

For 3PAR storage arrays, set the QFullThreshold value to 4.

ForNetApp and IBM XIV storage arrays, set the QFullThreshold value to 8.

For other storage arrays, please contact your storage vendor.

The settings take effect immediately. You do not need to reboot the ESX/ESXi host.

VMware ESXi 5.1 Patch 1 and later versions

In ESXi 5.1, QFullSampleSize and QFullThreshold are not set globally because different vendors have different optimal values for their arrays. You set QFullSampleSize and QFullThreshold on a per-device level. With ESXi 5.1 Patch 1 and later versions, the earlier option to set QFullSampleSize and QFullThreshold globally on all devices seen by the ESXi host, is available again. If you apply both the global option and also set one of the parameters for a specific device, the setting for the specific device always takes precedence over the global setting.

Note: Devices managed by the global parameter will return incorrect LUN queue depth values when queried using the command esxcli storage core device list. Instead, these devices will return these a value of zero:

Queue Full Sample Size: 0

Queue Full Threshold: 0

Do not be alarmed, the correct global LUN queue depth values are being applied to these devices, this can be confirmed using esxtop, for details see: Checking the queue depth of the storage adapter and the storage device (1027901).

Related Information

Checking the queue depth of the storage adapter and the storage device
Changing the queue depth for QLogic, Emulex, and Brocade HBAs
Controlando a limitação da densidade da fila do LUN no VMware ESX/ESXi
控制 VMware ESX/ESXi 中的 LUN 队列深度限制
VMware ESX/ESXi での LUN のキューの深さの調整のコントロール

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Result 1 of 1

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 8.0

VMware vSphere ESXi 7.0

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant