Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

PSOD and VMFS corruption can occur when using HPE Rebranded Qlogic QLE269x series HBAs on HPE/Hitachi Servers (76353)

Last Updated: 2/27/2024Categories: TroubleshootingTotal Views: 30114Language:
Chinese (Simplified)JapaneseEnglish

13 like this article

Subscribe to get an email
newsletter with new updates

13 people found this helpful Please provide article feedback:

Learn how VMware Skyline Advisor and Skyline Health Diagnostics work together to provide proactive intelligence and self-service log analysis.

Use Skyline Advisor to avoid issues before they occur across VCF, vSphere, vSAN, NSX, vROps & Horizon. Skyline Findings are based on top KBs, VMSAs/CVEs and design best practices.

Use Skyline Health Diagnostics to root cause vSphere & vSAN logs when an issue occurs. Skyline Health Diagnostic Findings are based on top KBs and VMSA/CVEs. You can use the Skyline Health Diagnostics tool in online or offline mode.

If you require additional help from technical support, please use Skyline Advisor to initiate a Log Assist to reduce time to upload logs. TSEs can also initiate logs to reduce effort.

Symptoms

When using HPE rebranded Qlogic QLE269x HBAs,  ESXi host fails with  PSOD as well as potentially VMFS corruption. This applies to the HPE rebranded HBAs SN1100Q & SN1600Q

The most common indicator of an issue with the QLE269x chipset is the following message in /var/log/vmkernel.log, usually repeated numerous times leading up to the PSOD:
WARNING: qlnativefc: vmhba2(12:0.0): Invalid ISP SCSI completion handle(281) req=1
WARNING: qlnativefc: vmhba2(12:0.0): Invalid ISP SCSI completion handle(282) req=1
WARNING: qlnativefc: vmhba2(12:0.0): Invalid ISP SCSI completion handle(284) req=1

Below are additional qlnativefc driver messages that have been observed around the time of the PSOD:
qlnativefc: vmhba3(12:0.1): Inside qlnativefcAbortIsp
qlnativefc: vmhba3(12:0.1): Performing ISP error recovery - ha= 0x4308e6398b50.
qlnativefc: vmhba2(12:0.0): Inconsistent NVRAM detected: checksum=0x8c54d9ed id=^@ version=0x0.
qlnativefc: vmhba2(12:0.0): Falling back to functioning (yet invalid -- WWPN) defaults.
qlnativefc: vmhba2(12:0.0): 83XX: F/W Error Reported: Check if reset required.
qlnativefc: vmhba2(12:0.0): Heartbeat Failure encountered, chip reset required.

When VMFS corruption occurs as a result, the following messages may be observed immediately before the PSOD:
WARNING: Fil3: 7920: Found invalid object on 5c80ba44-31b46bc0-d894-xxxxxxxxxxxx <FD c49 r5> expected <FD c21 r107>

Other messages related to the datastores  could be observed after reboot depending on whether the corruption is preventing the VMFS volume from mounting.

The PSOD backtrace has varied across environments however these three examples would be the most common observed:
Example 1

2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b3d0:[0x41801e8edaf1]PanicvPanicInt@vmkernel#nover+0x545 stack: 0x41801e8edaf1, 0x0, 0x4391ae49b478, 0x0, 0x1f48f6cc0
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b470:[0x41801e8edb7d]Panic_NoSave@vmkernel#nover+0x4d stack: 0x4391ae49b4d0, 0x4391ae49b490, 0x2000418000000000, 0x41801ec3e2f1, 0x1280
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b4d0:[0x41801e9466a5]DLM_malloc@vmkernel#nover+0x1469 stack: 0x1, 0x18, 0x8, 0x41801eb22010, 0x18
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b520:[0x41801e943a8c]Heap_AlignWithTimeoutAndRA@vmkernel#nover+0xc4 stack: 0x800000000, 0x0, 0x18, 0x4306f4695030, 0x0
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b5a0:[0x41801eb21f57]SCSIVsiDeviceWorldListIsDup@vmkernel#nover+0x4f stack: 0x0, 0x41801eb22010, 0x46, 0x4306f48f6cc0, 0x4391ae49b5d0
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b5c0:[0x41801eb22010]SCSIVsiDeviceWorldList@vmkernel#nover+0x80 stack: 0x4391ae49b5d0, 0x4391ae49b5d0, 0x0, 0x0, 0x5ca
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b600:[0x41801e80229f]VSI_GetListInfo@vmkernel#nover+0x253 stack: 0x4391ae49b6b0, 0x4391ae49b740, 0x417fde843242, 0xeb19a8, 0x8d
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49b680:[0x41801ef1ec03]UWVMKSyscallUnpackVSI_GetList@(user)#<None>+0x24f stack: 0x4391ae49bf30, 0xd99d7b78a037a0d4, 0x8de47592fb, 0x2c000eb5ca0, 0x300000001
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49bef0:[0x41801ef124c0]User_UWVMKSyscallHandler@(user)#<None>+0xa4 stack: 0x0, 0x0, 0x0, 0x41801e910025, 0x0
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49bf20:[0x41801e910025]User_UWVMKSyscallHandler@vmkernel#nover+0x1d stack: 0x0, 0x13b, 0x0, 0x4ad, 0x5ca
2019-11-07T22:22:13.500Z cpu6:67017)0x4391ae49bf30:[0x41801e93d067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x4ad, 0x5ca, 0x0, 0x2896448
2019-11-07T22:22:13.524Z cpu6:67017)ESC[45mESC[33;1mVMware ESXi 6.5.0 [Releasebuild-13004031 x86_64]ESC[0m
PANIC bora/vmkernel/main/dlmalloc.c:4736 - Corruption in dlmalloc

Example 2
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79bd30:[0x418022545702]DLM_malloc@vmkernel#nover+0x4c6 stack: 0x4306748bd580, 0xff, 0x8, 0x418022734afd, 0xff
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79bd80:[0x418022543a8c]Heap_AlignWithTimeoutAndRA@vmkernel#nover+0xc4 stack: 0x888902a80, 0x418000000000, 0xff, 0x430674618030, 0x430674810ed0
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79be00:[0x4180227326de]SCSI_VaaiCacheUpdate@vmkernel#nover+0x372 stack: 0x43914a79be60, 0x417fe277f010, 0x0, 0x418022734eb7, 0xf000f0300000000
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79bf30:[0x418022734afd]SCSIDeviceReclaimByFilters@vmkernel#nover+0x21 stack: 0x4306746a9d00, 0x4180224ca3ed, 0x430198ebb050, 0x25, 0x430198ebb050
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79bf50:[0x4180224ca3ed]helpFunc@vmkernel#nover+0x3c5 stack: 0x430198ebb050, 0x0, 0x0, 0x0, 0x0
2019-10-07T09:32:34.245Z cpu17:65871)0x43914a79bfe0:[0x4180226cb675]CpuSched_StartWorld@vmkernel#nover+0x99 stack: 0x0, 0x0, 0x0, 0x0, 0x0
2019-10-07T09:32:34.269Z cpu17:65871)ESC[45mESC[33;1mVMware ESXi 6.5.0 [Releasebuild-11925212 x86_64]ESC[0m
#GP Exception 13 in world 65871:SCSI periodi @ 0x418022545702

Example 3
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9ba30:[0x418032553fdf]DLM_free@vmkernel#nover+0x323 stack: 0x430c7f9e3790, 0x418032551501, 0x430c7fa800a0, 0x57ab5420a5c18, 0x1ff
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9ba50:[0x418032551500]Heap_Free@vmkernel#nover+0x115 stack: 0x1ff, 0x430c7f9e3790, 0x1, 0x0, 0x92a000
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9baa0:[0x41803341b24c]FSAts_Lock@esx#nover+0xf1 stack: 0x0, 0x417fd26044c0, 0x430c00000000, 0x200000000001, 0x4
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bb20:[0x41803340e92f]FS3_DiskBufferLock@esx#nover+0xa0 stack: 0x0, 0x92a000, 0x0, 0x430c7f94c380, 0x430c7f94c380
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bb80:[0x41803340eb47]FS3_DiskLockWithPrefetch@esx#nover+0x150 stack: 0x0, 0x430c7f9cc5e0, 0x4180334319e0, 0x418032515653, 0x1848920
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bc60:[0x4180334adb4e]Res3LockClusterVMFS6@esx#nover+0x9f stack: 0x0, 0x0, 0x430c7e1dbff0, 0x430c7f94a370, 0x0
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bcd0:[0x4180334bbc4b]Res6OnDiskLockRC@esx#nover+0x1d0 stack: 0x430c7f94a3a8, 0x100000000000000, 0x7d000000005, 0x5, 0x430c7fa63f60
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bd50:[0x4180334c9ecb]UnmapLockCommon@esx#nover+0x4d4 stack: 0x430c7f7d0b50, 0x5, 0x2ff87396e329, 0xffff00000000, 0x0
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9be30:[0x4180334cc165]UnmapAsyncClusterProcessing@esx#nover+0x2ce stack: 0x4519c00f0000, 0x41803251c74a, 0x0, 0x837bea000001cc01, 0x0
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bf30:[0x4180324eb1f2]HelperQueueFunc@vmkernel#nover+0x30f stack: 0x430c7e1e2e28, 0x430c7e1e2e18, 0x430c7e1e2e50, 0x451a18ca3000, 0x451a0f89bf60
2019-09-14T03:36:03.898Z cpu2:2456914)0x451a18c9bfe0:[0x41803270e322]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0, 0x0, 0x0, 0x0, 0x0
2019-09-14T03:36:03.922Z cpu2:2456914)ESC[45mESC[33;1mVMware ESXi 6.7.0 [Releasebuild-13981272 x86_64]ESC[0m
#GP Exception 13 in world 2456914:Unmap Helper @ 0x418032553fdf

“Issue is observed on both ESX 6.5 and ESX 6.7”  This is a card firmware issue, not a driver version issue
We have observed this issue mostly on HPE DL360, DL380, & DL580 or Hitachi equivalent rebranded HPE servers (HA8000V).

Note:The preceding log excerpts are only example.Date,time and environmental variables may vary depending on your environment

Purpose

To address the PSOD issue and VMFS corruption.

Cause

Firmware incorrectly replays stale (or previously completed) I/O requests, including DMA transfers to potentially freed memory locations causing memory heap corruption and  getting propagated to disk  and  some writes getting lost.

Due to this there are  number of SCSI read, write, ATS failures and retries logged in the vmkernel leading up to the crash.

Impact / Risks

If the datastores are affected, it  is required to fixed using  vSphere On-disk Metadata Analyzer(VOMA).

Resolution

HPE has identified the root cause for this issue and has created a fix for their rebranded cards - Refer to HPE Advisory

For HPE branded adapters listed in (HPE Advisory)  on ESXi6.7 refer to HPE firmware release notes

For HPE branded adapters listed in (HPE Advisory) on ESXi6.5 refer to HPE firmware release notes

For the VMFS Metadata consistency to  be fixed the volume should be mounted to atleast one ESXi 6.7 U2 host as VOMA allows you to check and fix issues with VMFS volumes metadata, LVM metadata, and partition table inconsistencies- For more info refer to ESXi 6.7 U2 release notes

Refer to VOMA guide for more information

Workaround

There is no workaround for this issue.

Related Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

KB
•
SNMP サービスがクラッシュし、snmpd-zdump ファイルが生成される (76311)

KB
•
克隆虚拟机时出现“指定的参数不正确: vm.ctkEnabled”错误 (76280)

KB
•
vRealize Operations 8.0.x 调整大小准则 (75162)

KB
•
vSAN 健全性サービス - ファイル サービス - インフラストラクチャの健全性 (76677)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

Related Versions:

VMware vSphere ESXi 6.7

VMware vSphere ESXi 6.5

VMware vSphere ESXi 6.0

Take Our Survey

Support Assistant