Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

Kubernetes service External IP changes without manual update in Enterprise PKS (71275)

Last Updated: 10/26/2020Categories: InformationalTotal Views: 6347Language:
English

Subscribe to get an email
newsletter with new updates

Please provide article feedback:

Symptoms

Kubernetes service External IP (LB VIP) changes automatically without user recreation or update of the service.

In the /var/vcap/sys/log/ncp/ncp.stdout.log on the K8S Master node, you see the entries to service LB getting recreated:
2019-08-09T21:01:06.273Z 42053046-217c-458c-9e7c-ac57ca49aedf NSX 11715 - [nsx@5876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.k8s.service_lb_controller There is an error happened or service type changed to service Loadbalancer nginx project nginx, LB will be deleted
2019-08-09T21:01:07.958Z 42053046-217c-458c-9e7c-ac57ca49aedf NSX 11715 - [nsx@5876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.k8s.service_lb_controller External ip used for service nginx in namepsace nginx is 10.40.14.42 from ip_pool e9219c53-4454-4b86-af06-9aaefa2d78ae
2019-08-09T21:01:08.715Z 42053046-217c-458c-9e7c-ac57ca49aedf NSX 11715 - [nsx@5876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.k8s.service_lb_controller Successfully updated Loadbalancer resources for service nginx under project nginx

In the ncp.stdout.log file, you also see that NCP API request rate is exceeding the limit (default 100) on NSX-T Manager.
2019-08-09T20:52:55.958Z 42053046-217c-458c-9e7c-ac57ca49aedf NSX 11563 - [nsx@5876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] vmware_nsxlib.v3.client The HTTP request returned error code 429, whereas 200 response codes were expected. Response body {u'module_name': u'common-service', u'error_message': u'Exceeded request rate of 100 per second', u'error_code': u'98'}

Cause

In scaled PKS environments, the heartbeat API calls sent from NCP process to NSX Manager may not get a response in time due to high CPU utilization on NSX Manager.  This triggers the NCP processes in the Kubernetes clusters to restart causing NCP Master re-election. When NCP processes in all Kubernetes clusters go for re-election at the same time, the number of API calls would exceed the default API rate limits configured on NSX Manager.  During this process some of the API calls fail to get a response which causes NCP process to fail and restart again leading to re-election loop.

So whenever this happens, during that time if NCP request for the service LB detail, Manager won't be able to provide that information and so eventually NCP would assume that the LB is not present and it would start creating it.

Resolution

This is a known issue impacting VMware Enterprise PKS 1.3.x and earlier.

This behavior is improved in VMware Enterprise PKS 1.4.1 + VMware NSX-T 2.4.1 by updating the default values for API rate limit and concurrency limit. Additionally, you can tune the NCP parameters for increased PKS Scaling by following the KB Tuning NCP parameters for increased PKS Scaling in VMware Enterprise PKS environment (68085)

In NSX-T 2.4.1, the default API rate limit values are:

nsxmgr-01a> get service http
Service name:                     http
Service state:                    running
Session timeout:              1800 seconds
Connection timeout:           30 seconds
Client API rate limit:        100 requests/sec
Client API concurrency limit: 40 connections
Global API concurrency limit: 199 connections
Redirect host:                (not configured)

Workaround

To work around this issue in VMware Enterprise PKS 1.3.x, increase the API rate limits on NSX-T Manager and update the NCP parameters on each K8S node in PKS cluster:

To update the rate limit values on NSX-T Manager:

Connect to NSX-T manager as admin user.

Set API rate to unlimited and concurrency limit to 199 by running the below commands on NSX-T Manager:
set service http client-api-rate-limit 0
set service http client-api-concurrency-limit 199

To update the NCP parameters in on each master node:

Connect to the first master node using Bosh:
bosh -d <service-instance> ssh master/0
Navigate to sudo by running : sudo -i

Stop the ncp service by running the command: monit stop ncp

Add the below parameters under [ha] section in /var/vcap/jobs/ncp/config/ncp.ini file:
master_timeout = 18
heartbeat_period = 6
update_timeout = 10

Start the ncp service by running the command: monit start ncp

Repeat the same steps on other master nodes.

Repeat the above steps for other PKS clusters as well.

Note: These NCP HA changes are ephemeral and reset to default values if you recreate the Master node from ops manager/bosh cck. In that case you may need to manually update the NCP HA changes again.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
VeloCloud SD-WAN Edge 多播配置和监控 (71223)

KB
•
请求批量导出失败并显示错误：“创建导出时出错: 值不能为 null。参数名称: stringToEscape (An error occurred while creating export: Value cannot be null. Parameter name: stringToEscape)” (71263)

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
在启用了 UAC 的 Windows 上无法安装 vRealize Operations Manager 7.5 Telegraf 代理 (70780)

Results 1-4 of 4

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware PKS

Related Versions:

VMware PKS 1.x

Take Our Survey

Support Assistant