Loading

×Sorry to interrupt
This page has an error. You might just need to refresh it.
[NoErrorObjectAvailable] Uncaught ReferenceError: addCTracking is not defined
Failing descriptor: {markup://c:KM_ArticleThemeLayout}

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

PKS cluster upgrade stuck at draining the node (67275)

Last Updated: 9/1/2023Categories: TroubleshootingTotal Views: 2975Language:
English

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Symptoms

↵

PKS cluster upgrade will get stuck at draining the node ​and it fails.

You see messages similar to the following:

 I, [2019-02-23T13:43:35.524172 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Applying VM state
I, [2019-02-23T13:45:14.166716 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Running pre-start for worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:45:27.234148 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Starting instance worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:45:27.707511 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Waiting for 10.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:45:37.708100 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 10.0 seconds
I, [2019-02-23T13:45:37.723070 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:45:52.723546 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:45:52.740554 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:46:07.740966 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:46:07.760749 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:46:22.761471 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:46:22.778678 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))]  INFO -- DirectorJobRunner: Running post-start for worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:46:49.969120 #15670] [instance_update(worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15))]  INFO -- DirectorJobRunner: Updating instance worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15), changes: "stemcell, packages, configuration, job"
I, [2019-02-23T13:46:49.995086 #15670] [instance_update(worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15))]  INFO -- DirectorJobRunner: Running drain for worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15) 

Resolution

You need to manually clear the pods on specific nodes after identifying any apps are deployed using Daemon sets or Replica sets.

bosh task <task id> --debug | grep INFO > to get the current status, if its stuck with drain, proceed with below.

kubectl get nodes -o wide (Will give you the details about node ID)
kubectl get pod --all-namespaces -o wide | grep <Node ID>

SSH to the node which is stuck in drain state and stop Kubelet and docker service

kubectl delete pods 'pod name' -n namespace --grace-period=0 --force (Most of the time, after deleting failed pods the upgrade will automatically proceed

If after deleting the failed pods the issue persists, then manually drain the node using following command:

kubectl drain <Node ID> --force --ignore-daemonsets

Note: Before you delete anything, ensure you have documented the deployment is used to deploy the pods.

If it is daemonset then you know need to worry because daemonset will maintain the pod copies as per the number of nodes

If it is Replica sets then make sure at least one set of app is available always until your devops  team confirms you can shutdown for time being

kubectl get pods --all-namespaces -o wide (To know how many copies are running)
kubectl get ds --all-namespaces (To know what type of deployment it is)
kubectl get deployment nginx -o yaml > first.yaml (To read the deployment file)

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
End Of General Availability of the Free vSphere Hypervisor (ESXi 7.x and 8.x) (2107518)

KB
•
Download and Install VMware Fusion (2014097)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware PKS

Related Versions:

VMware PKS 1.x

Take Our Survey

Support Assistant