Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Unified Access Gateway (UAG) high CPU utilization : HTTP 503 error (78419)

Last Updated: 4/7/2023Categories: Best PracticesTotal Views: 17332Language:
Chinese (Simplified)JapaneseEnglish

11 like this article

Subscribe to get an email
newsletter with new updates

11 people found this helpful Please provide article feedback:

Details

Unified Access Gateway virtual appliances becomes intermittently unresponsive with high CPU pegged near 100%.

When Unified Access Gateway is overloaded and using excessive CPU (>90% across all vCPUs), it will inform a load balancer to back off (with an HTTP 503 error) so it can spread the load to other less busy UAG.

When this level is exceeded, Unified Access Gateway responds to HTTP requests with a 503 error to indicate it is unable to handle the request due to temporary overloading. The support for this configuration allows a load balancer to allocate new sessions to alternative appliances. The default value is 100% so will never be exceeded, but a lower value can be configured in system settings using the Admin UI.

The definition of an HTTP 503 response is: The server cannot handle the request for a temporary overloading or maintenance of the server.

From the command line of a UAG , either from the console or via SSH (disabled by default:Permitting Root Login with SSH)

The top tool monitors system resources, workloads, and performance. It can unmask problems caused by processes or applications overconsuming CPUs, time, or RAM.

The UAG keeps an archive of Top output (/opt/vmware/gateway/logs/top_archive) which is included in a diagnostic bundle Collecting Logs from the Unified Access Gateway Appliance

To view a textual display of resource consumption, run the top command from the appliance terminal

top

Sample Command Output:

Tasks: 128 total,  3 running, 125 sleeping,  0 stopped,  0 zombie
%Cpu0 : 62.5/31.2  94[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu1 : 68.8/25.0  94[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PID USER     PR NI   VIRT   RES %CPU %MEM    TIME+ S COMMAND
4864 gateway 25 5 931.4m 80.8m 81.2 0.5 22962:07 R /opt/vmware/gateway/bin/node /opt/vmware/gateway/lib/bsg/lib/absg.js 4874 gateway 25 5 928.6m 73.8m 81.2 0.5 23108:07 R /opt/vmware/gateway/bin/node /opt/vmware/gateway/lib/bsg/lib/absg.js 4879 gateway 25 5 930.3m 81.1m 75.0 0.5 23542:07 R /opt/vmware/gateway/bin/node /opt/vmware/gateway/lib/bsg/lib/absg.js 4865 gateway 25 5 923.0m 72.8m 68.8 0.5 22864:54 R /opt/vmware/gateway/bin/node /opt/vmware/gateway/lib/bsg/lib/absg.js

In the above example, we see that BSG is a contributor to the load. We can then examine the logs for this process. Collecting Logs from the Unified Access Gateway Appliance contains a table listing the location and description of each logfile.

/opt/vmware/gateway/logs/bsg.log

[absg-worker] - Missing Host header from client ::ffff:1P.IP.1P.IP
absg-worker] - Missing Host header from client ::ffff:1P.IP.1P.IP
absg-worker] - Missing Host header from client ::ffff:1P.IP.1P.IP

The frequency of these entries might indicate a bad health check.

Solution

Potential reasons for a CPU spike:

Blocking UDP 8443 at the firewall. This forces all this traffic to TCP only which uses significantly more CPU.

Switching Blast TCP port from the default of 8443 to 443 (by putting :443 at the end of the blastExternalUrl) which then involves double-hop TCP 443 > 8443.

UAG logging set to Debug/Trace level.

Healthcheck / polling intervals are set too frequent

Best Practices to avoid CPU congestion:

Use UDP

Allow UDP 8443 on the firewall. Blast UDP  has more efficient encryption/ decryption mechanism incurring less overhead.

If UDP 8443 is blocked (or UDP 22443 is blocked from UAG to the desktop) , the client can't use this more efficient UDP protocol and has to back down to TCP only. Blast TCP is a TLS WebSockets connection which involves more decrypt/encrypt by the Blast BSG component.

Use 8443 for  your blastExternalUrl.

If 443 is used for blastExternalUrl, Blast connections arrive on UAG on port 443, there is a double hop within UAG to forward it on TCP 8443 to BSG. Double hop is less efficient than single hop.

If UDP is not blocked, then most clients will use UDP so in this case the effect of 2 is much less significant because TCP use will be much less anyway so the double-hop less significant. So, not blocking UDP is the most significant way to improve efficiency.

Ensuring Reasonable Health Monitoring polling Intervals.

Monitoring health of Horizon Connection Server using Load Balancer, timeout, Load Balancer persistence settings in Horizon 7 and 8 (56636) outlines guidelines . Please also reference vendor guidance.

Interval: 30 seconds
Timeout: 91 seconds
Monitor String: GET /favicon.ico

Note: UAG might stop responding when the total session count go beyond 20k per UAG. Rebooting UAG to clear out the session count helps alleviate the issue temporary. Correcting health check polling interval on load balancer should be the permanent way to address it.

Additional Considerations:

If modifying above settings do not help with CPU utilization, we also need to take into consideration below factors :

Number of connections landing on each UAG : Session count may vary depending upon the connection type (TCP/UDP).

Number of CPU's on the UAG appliance : Increase the CPU from default 2 to 4.

Note: Based on the use case, you can also scale up by adding more UAG appliances to the environment.

Recommendation would be to always use the most recent version of UAG :Unified Access Gateway(UAG):Lifecycle support policy for VMware Unified Access Gateway (2147313)
If you encounter an issue similar to the above, you must test  with UAG 3.10 or newer.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
Chinese (Simplified)JapaneseEnglish

Attachments

Additional Resources

No results for undefined

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware Unified Access Gateway

VMware Horizon

Related Versions:

VMware Horizon 7.x

VMware Horizon 7

Take Our Survey

Support Assistant