Loading

×Sorry to interrupt
CSS Error

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

Skip to main content

My Subscriptions

Loading

Knowledge Base

Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205)

Last Updated: 3/2/2021Categories: How toTotal Views: 756633Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglishGerman

61 like this article

Subscribe to get an email
newsletter with new updates

61 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Details

This article provides information about esxtop and latency statistics that can be used when troubleshooting performance issues with SAN-connected storage (Fibre Channel or iSCSI).

Note: In ESXi 5.x and later, you may see messages indicating that performance has deteriorated. For more information, see Storage device performance deteriorated (2007236).

Solution

The interactive esxtop utility can be used to provide I/O metrics over various devices attached to a VMware ESX host.

Configuring monitoring using esxtop

To monitor storage performance per HBA:

Start esxtop by typing esxtop at the command line.

Press d to switch to disk view (HBA mode).

To view the entire Device name, press SHIFT + L and enter 36 in Change the name field size.

Press f to modify the fields that are displayed.

Press b, c, d, e, h, and j to toggle the fields and press Enter.

Press s and then 2 to alter the update time to every 2 seconds and press Enter.

See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

Note: These options are available only in VMware ESX 3.5 and later.

To monitor storage performance on a per-LUN basis:

Start esxtop by typing esxtop from the command line.

Press u to switch to disk view (LUN mode).

Press f to modify the fields that are displayed.

Press b, c, f, and h to toggle the fields and press Enter.

Press s and then 2 to alter the update time to every 2 seconds and press Enter.

See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

To increase the width of the device field in esxtop to show the complete naa id:

Start esxtop by typing esxtop at the command line.

Press u to switch to the disk device display.

Press L to change the name field size.

Note: Ensure to use uppercase L.

Enter the value 36 to display the complete naa identifier.

To monitor storage performance on a per-virtual machine basis:

Start esxtop by typing esxtop at the command line.

Type v to switch to disk view (virtual machine mode).

Press f to modify the fields that are displayed.

Press b, d, e, h, and j to toggle the fields and press Enter.

Press s and then 2 to alter the update time to every 2 seconds and press Enter.

See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

Note: If the number of lines displayed exceeds the window size you can press 2 to select and navigate between the lines you wish to remove. Press 4 on a highlighted line to remove it.

Analyzing esxtop columns

Refer to this table for relevant columns and descriptions of these values:

Column

Description

CMDS/s

This is the total amount of commands per second and includes IOPS (Input/Output Operations Per Second) and other SCSI commands such as SCSI reservations, locks, vendor string requests, unit attention commands etc. being sent to or coming from the device or virtual machine being monitored.

In most cases, CMDS/s = IOPS unless there are a lot of metadata operations (such as SCSI reservations)

DAVG/cmd

This is the average response time in milliseconds per command being sent to the device.

KAVG/cmd

This is the amount of time the command spends in the VMkernel.

GAVG/cmd

This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG

These columns are for both reads and writes, whereas xAVG/rd is for reads and xAVG/wr is for writes. The combined value of these columns is the best way to monitor performance, but high read or write response time it may indicate that the read or write cache is disabled on the array. All arrays perform differently, however, DAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed more than 10 milliseconds (ms) for sustained periods of time.

Note: VMware ESX 3.0.x does not include direct functionality to monitor individual LUNs or virtual machines using esxtop. Inactive LUNs lower the average for DAVG/cmd, KAVG/cmd, and GAVG/cmd. These values are also visible from the vCenter Server performance charts. For more information, see the Performance Charts section in the Basic System Administration Guide.

If you experience high latency times, investigate current performance metrics and running configuration for the switches and the SAN targets. Check for errors or logging that may suggest a delay in operations being sent to, received, and acknowledged. This includes the array's ability to process I/O from a spindle count aspect, or the array's ability to handle the load presented to it.

If the response time increases to over 5000 ms (or 5 seconds), VMware ESX will time out the command and abort the operation. These events are logged; abort messages and other SCSI errors can be reviewed in these logs:

ESX 3.5 and 4.x – /var/log/vmkernel

ESXi 3.5 and 4.x – /var/log/messages

ESXi 5.x and later - /var/log/vmkernel.log

The type of storage logging you may see in these files depends on the configuration of the server. You can find the value of these options by navigating to Host > Configuration > Advanced Settings > SCSI > SCSI.Log* or SCSI.Print*.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Troubleshooting LUN connectivity issues on ESXi hosts
You can also collect performance snapshots using vm-support. For more information, see:

Collecting performance snapshots using vm-support (1967)

Creating an alarm or email notification that triggers when Disk Latency or DAVG is high

For translated versions of this article, see:

Español: Usando esxtop para identificar problemas de rendimiento en almacenamiento (2003665)

Português: Usando esxtop para identificar problemas de desempenho de armazenamento do ESX/ESXi (diversas versões) (2017691)

日本語: ストレージのパフォーマンスに関する問題を識別するために、esxtopを使用 (2014768)

简体中文: 使用 esxtop 识别存储性能问题 (2074626)

Deutsch: Verwenden von esxtop zum Identifizieren von Problemen bei der Speicherleistung für ESX/ESXi (mehrere Versionen) (2145096)

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)Portuguese (European)JapaneseSpanishEnglishGerman

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vSphere ESXi

VMware ESX Server

Related Versions:

VMware vSphere ESXi 7.0.0

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

VMware vSphere ESXi 5.5

VMware vSphere ESXi 5.1

VMware vSphere ESXi 5.0

VMware ESXi 4.1.x Installable

VMware ESXi 4.1.x Embedded

VMware ESXi 4.0.x Installable

VMware ESXi 4.0.x Embedded

VMware ESXi 3.5.x Installable

VMware ESXi 3.5.x Embedded

VMware ESX Server 3.5.x

VMware ESX Server 3.0.x

VMware ESX 4.1.x

VMware ESX 4.0.x

view all

Take Our Survey

Was this article helpful?

YES

NO

Copyright © 2005-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

Terms of Use

Your California Privacy Rights

Privacy

Accessibility

Cookie Settings

Support Assistant