Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

"Cannot retrieve service content from vCenter Server localhost:80", EAM firstboot fails when migrating from Windows vCenter Server 6.0 to vCenter Server Appliance 6.7 (60356)

Last Updated: 5/4/2021Categories: TroubleshootingTotal Views: 24418Language:
English

5 like this article

Subscribe to get an email
newsletter with new updates

5 people found this helpful Please provide article feedback:

Symptoms

EAM firstboot fails when migrating from Windows vCenter Server 6.0 to vCenter Server Appliance 6.7

In eam_firstboot.py_10578_stderr.log, you see entries similar to:

INFO:root:Found 1 matching service. ID is e5e50715-1930-4df7-a43b-32bd744ebaef
2018-10-31T14:25:46.795Z Error during EAM extension registration in VC.
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)
2018-10-31T14:25:46.806Z Traceback (most recent call last):
File "/usr/lib/vmware-eam/firstboot/eaminstall/extension/vCenter.py", line 217, in _retrieveServiceContent
self.siC = si.RetrieveContent()
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
.
.
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:719)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
.
File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
self._sslobj.do_handshake()
cis.baseCISException.BaseInstallException: {
"problemId": null,
"componentKey": "eam",
"resolution": {
"translatable": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request.",
"id": "install.eam.fileSR.resolution",
"localized": "Search for these symptoms in the VMware knowledge base for any known issues and possible workarounds. If none can be found, collect a support bundle and open a support request."
},
"detail": [
{
"translatable": "Cannot retrieve service content from vCenter Server %(0)s:%(1)s.",
"args": [
"localhost",
80
],
"id": "install.eam.error.extension.service_content",
"localized": "Cannot retrieve service content from vCenter Server localhost:80."
},
{
"translatable": "Unable to register ESX Agent Manager as an extension in vCenter Server.",
"args": [],
"id": "install.eam.error.extension",
"localized": "Unable to register ESX Agent Manager as an extension in vCenter Server."
}
]
}

2018-10-31T14:25:46.806Z  VMware ESX Agent Manager firstboot failed.

Or

2019-03-18T12:08:43.352Z Error during EAM extension registration in VC.[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
2019-03-18T12:08:43.361Z Traceback (most recent call last):File "/usr/lib/vmware-eam/firstboot/eam_firstboot.py", line 355, in main fb.registerExtension()

In eam_firstboot.py_10578_stdout.log, you see entries similar to:

While Retrieving machine ssl cert:

2018-10-31T14:25:46.613Z ESX Agent Manager service installed
2018-10-31 14:25:46.613495 Retrieving Vpxd certificate.
2018-10-31T14:25:46.613Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/tmp03e4hcol']
2018-10-31T14:25:46.634Z Done running command
2018-10-31 14:25:46.635274 Retrieving Vpxd private key.
2018-10-31T14:25:46.635Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getkey', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/tmpcci2pk6e']
2018-10-31T14:25:46.660Z Done running command

While retrieving signing certificate of the Machine SSL:

2018-10-31 14:25:46.680541 Retrieving CA certificate.
2018-10-31T14:25:46.680Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'list', '--store', 'TRUSTED_ROOTS']
2018-10-31T14:25:46.725Z Done running command
2018-10-31T14:25:46.726Z Running command: ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'TRUSTED_ROOTS', '--alias', '31393071fac886b194715db3f716f16ad298967f', '--output', '/var/tmp/vmware/tmple3kevq7']
2018-10-31T14:25:46.747Z Done running command
2018-10-31 14:25:46.747608 Registering EAM service as a VC extension.

As per the below code firstboot scripts is only looking (or expecting) the first entry in trusted root store to be signing certificate of the Machine SSL and if it is not migrate fails with eam firstboot.

/usr/lib/vmware-eam/firstboot/eaminstall/vecs.py:

self._vecsCAEntry.get_cert(caAliases[0], output_file)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

This issue is resolved in vCenter Server Appliance 6.7 U3g, available at VMware Downloads .

Workaround

Workaround 1:
To work around the issue, un-publish and re-publish certs from trusted roots store so that the machine SSL signing cert is first entry in trusted store.

Create a snapshot of vCenter Server.

Identify Alias of Machine SSL signing cert:

List the certificates stored in trusted_roots and look for keyword "Alias"

“C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry list --store TRUSTED_ROOTS --text | findstr Alias
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias

Identify the machine SSL signing cert alias in the trusted roots store [Alias : __MACHINE_CERT]

Take backup of all certs that are available in trusted roots store.

Use the below commands to backup certs in trusted root store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias> --output /storage/core/<alias.crt>
"C:\Program Files\VMware\vCenter Server\vmafdd\"vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate alias>--output c:\certificates\<alias>.crt

Repeat the above step for all the certs available in trusted store.

Unpublish the certificate.

Use dir-cli to unpublish for each cert file:

C:\Program Files\VMware\vCenter Server\vmafdd\dir-cli.exe trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <cert#.crt> --login administrator@<SSO Domain> --password <passwd>

Re-publish certs and make sure signing certificate of the Machine SSL is the first certificate published to trusted roots store.

Use dir-cli to re-publish for each cert file:

C:\Program Files\VMware\vCenter Server\vmafdd> dir-cli.exe trustedcert publish --cert <C:\Trusted_certs\To be published\cert#.crt> --login administrator@<SSO Domain> --password <passwd>
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert </path_to_cer_file/cert#.crt> --login administrator@<SSO Domain> --password <passwd>

Force VECS to refresh after republishing certificate to see changes using below command,

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Additional reference KB Article: Firstboot Failed during Install/Deployment, Upgrade or Migration in vCenter Server and vCenter Server Appliance 6.7 (55746)

For example:

Before changes:
vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias

Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693

After Changes:

Published signing certificate of the Machine SSL to be first entry in trusted root store, also published intermediate and root cert.

vecs-cli.exe entry list --store TRUSTED_ROOTS --text | findstr Alias
Alias : 67ed5600354da88909ccda23ad133648313702f2 --> Machine ssl signing cert
Alias : 75a28c785a3b6cb4a74827a67d6a698609ec9926 --> Intermediate cert
Alias : 47e332b21bafbefa11e8ee2c334b9ac440922ff1 --> Root cert
Alias : 31393071fac886b194715db3f716f16ad298967f
Alias : ecfba4d095d7aa0678acaae2e205ae84baa7620f
Alias : eeccd9edd4f069c8e8baedb37ba0457062fe4b05
Alias : c8a0707d9240aefc88702949172292578aaf9a56
Alias : b0e38fe338fd287e8f978cf2d060460a20572688
Alias : 1b01800b62f827005480f06baa34497c60193e65
Alias : 356de6d9badb26bcd18cfb6a091713b8e3d0f5ce
Alias : 24c4e596e5db7dd7db01799a77b649cd405b0363
Alias : 3058753e77ed23b3070e9fafb882cc74ef392bd2
Alias : c7fc47982ce7f2e3b46b6aeb188dd4a9aff65693

Retry the upgrade/migrate process.

Workaroud 2:

To reset the certificates to VMware default certs, you can follow the below steps to reset the certificates to VMware Default certs and make sure if the VMCA root certificate is in first position in the TRUSTED_ROOTS store.

Launch the vSphere 6.x Certificate Manager.

For vCenter Server 6.x Appliance:
/usr/lib/vmware-vmca/bin/certificate-manager

For Windows vCenter Server 6.x:
C:\Program Files\VMware\vCenter Server\vmcad\certificate-manager

Select Option 4 (Regenerate a new VMCA Root Certificate and replace all certificates)

Once the certificates are replaced verify if the VMCA Root certificate is in first position in TRUSTED_ROOTS store.

Use the below command to list the certificates in TRUSTED_ROOTS store

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

Retry the upgrade/migrate process.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
确定 ESXi 主机关闭电源或重新启动的原因 (1019238)

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware vCenter Server Appliance

VMware vCenter Server

Related Versions:

VMware vCenter Server Appliance 6.7.x

VMware vCenter Server Appliance 6.5.x

VMware vCenter Server Appliance 6.0.x

VMware vCenter Server 6.7.x

VMware vCenter Server 6.5.x

VMware vCenter Server 6.0.x

Take Our Survey

Support Assistant