---

In this article we will discuss about, how to drain node in Kubernetes cluster. For maintenance activity such as patching etc.  requires downtime. In order to carry out such activities without impacting applications running within cluster, we need to follow some step by step command. We will discuss those commands in this article for impact less maintenance.

---

How to drain node in Kubernetes cluster?

Pre-requisite:

We have got Kubernetes cluster with one master node and two worker node as below:

$ kubectl get node
NAME          STATUS   ROLES    AGE    VERSION
k8s-control   Ready    master   167m   v1.18.5
k8s-worker1   Ready    <none>   166m   v1.18.5
k8s-worker2   Ready    <none>   166m   v1.18.5
$ 

Now lets run one pod and one deployment with following YAML definition as below for the testing purpose:

Pod Definition:

$ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
 name: my-pod
spec:
 containers:
 - name: nginx
   image: nginx
   ports:
   - containerPort: 80

Deployment definition:

$ cat deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-deployment
spec:
 replicas: 2
 selector:
    matchLabels:
     app: my-deployment
 template:
   metadata:
    labels:
      app: my-deployment
   spec:
     containers:
     - name: nginx
       image: nginx:1.14.2
       ports:
       - containerPort: 80

Now lets run both the definition file for creating the objects:

$ kubectl create -f pod.yaml
pod/my-pod created
$ kubectl create -f deploy.yaml
deployment.apps/my-deployment created
$

Cross verify the nodes on which Pod and the deployment Pods are running using below command:

$ kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
my-deployment-655f58b4cb-2sn4r   1/1     Running   0          25s     192.168.194.70   k8s-worker1   <none>           <none>
my-deployment-655f58b4cb-t4s7t   1/1     Running   0          25s     192.168.126.6    k8s-worker2   <none>           <none>
my-pod                           1/1     Running   0          5m13s   192.168.126.5    k8s-worker2   <none>           <none>
$

You can also see from the above output that, single Pod is running on the worker node “k8s-worker2” and the deployment pods are running on the both the nodes of Kubernetes cluster.

Draining the node?

Now lets drain the node “k8s-worker2” since its running Pod and the deployment Pods also.

$ kubectl drain k8s-worker2 --ignore-daemonsets --force
node/k8s-worker2 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-svhvv, kube-system/kube-proxy-5nk2w; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: default/my-pod
evicting pod default/my-deployment-655f58b4cb-hkdfh
evicting pod default/my-pod
pod/my-pod evicted
pod/my-deployment-655f58b4cb-hkdfh evicted
node/k8s-worker2 evicted
$

Note:

Make sure you use the option “–ignore-daemonsets” and “–force” in case you have standalone Pod and deployment running within your Kubernetes cluster other wise system wont allow you to drain the node and it throws following error:

$ kubectl drain k8s-worker2
node/k8s-worker2 cordoned
error: unable to drain node "k8s-worker2", aborting command...

There are pending nodes to be drained:
 k8s-worker2
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-svhvv, kube-system/kube-proxy-5nk2w
cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): default/my-pod
$

Once you drain the node successfully check the status of the node using below command:

$ kubectl get node
NAME          STATUS                     ROLES    AGE    VERSION
k8s-control   Ready                      master   172m   v1.18.5
k8s-worker1   Ready                      <none>   171m   v1.18.5
k8s-worker2   Ready,SchedulingDisabled   <none>   171m   v1.18.5
$

In the above output you can see that the node “k8s-worker2” has been marked as disabled for Scheduling the resource on it.  Now lets check the status of our Pod and Deployment Pods after draining the node.

$ kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP               NODE          NOMINATED NODE   READINESS GATES
my-deployment-655f58b4cb-2sn4r   1/1     Running   0          116s   192.168.194.70   k8s-worker1   <none>           <none>
my-deployment-655f58b4cb-gtjt4   1/1     Running   0          20s    192.168.194.71   k8s-worker1   <none>           <none>
$

From the above output its cleared that, the standalone Pod “my-pod” being deleted due to Drain activity, however Pod that is being run using deployment got killed and rescheduled on the other node which is “k8s-worker1”.

Joining the maintenance node back again to the Kubernetes cluster: 

Now lets assume that, our maintenance activity on the node “k8s-worker2” is completed and we want to join the node to our Kubernetes cluster. Then we need to uncordon the node as below:

$ kubectl uncordon k8s-worker2
node/k8s-worker2 uncordoned
$

Now if you check the status of the node it will show Ready status and that to without “SchedulingDisabled” status as below:

$ kubectl get node
NAME          STATUS   ROLES    AGE    VERSION
k8s-control   Ready    master   174m   v1.18.5
k8s-worker1   Ready    <none>   174m   v1.18.5
k8s-worker2   Ready    <none>   174m   v1.18.5
$

And also if you check the status of the deployment Pods its remains unchanged.

$ kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
my-deployment-655f58b4cb-2sn4r   1/1     Running   0          3m37s   192.168.194.70   k8s-worker1   <none>           <none>
my-deployment-655f58b4cb-gtjt4   1/1     Running   0          2m1s    192.168.194.71   k8s-worker1   <none>           <none>
$

This states that, even if you join the maintenance node again back to Kubernetes cluster the earlier running Pods keeps running on the same node.

So this is all about question “How to drain node in Kubernetes?” for the maintenance activity.

In case you want to go through other Kubernetes articles follow this link.