Cluster upgrades

Exam tasks: 1.4 (manage the lifecycle of Kubernetes clusters)

The decision: in what order do you upgrade which packages on which nodes, so the cluster moves up one minor version without breaking the version skew policy or dropping workloads?

Order of operationsCopy Anchor Link

• Control plane first, then workers. The kubelet must never be newer than the API server.
• One minor at a time. Going from 1.33 to 1.35 is two upgrades: 1.33 to 1.34, then 1.34 to 1.35. Patch versions within a minor can be skipped.
• kubeadm upgrade upgrades the static Pods (API server, controller manager, scheduler, etcd), CoreDNS and kube-proxy. It never upgrades the kubelet package or your CNI plugin.

Version skew you must respectCopy Anchor Link

First control plane nodeCopy Anchor Link

ssh cp-a
# 1. switch the repo to the target minor (repeat on every node before its turn)
sudo sed -i 's#/v1.34/#/v1.35/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -3                  # find the exact package version, e.g. 1.35.2-1.1

# 2. kubeadm first
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm='1.35.2-1.1'
sudo apt-mark hold kubeadm
kubeadm version -o short

# 3. plan and apply
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.2

# 4. kubelet and kubectl, with the node drained
k drain cp-a --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.35.2-1.1' kubectl='1.35.2-1.1'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
k uncordon cp-a

• upgrade plan checks that the cluster is healthy and lists the versions you can move to. Read it; it also tells you if a component config needs manual attention.
• upgrade apply renews the certificates kubeadm manages on that node unless you pass --certificate-renewal=false.
• Additional control plane nodes run sudo kubeadm upgrade node instead of apply, then the same kubelet steps. Do them one at a time; addons such as CoreDNS are upgraded after the last control plane finishes.

Worker nodesCopy Anchor Link

ssh node-b
sudo sed -i 's#/v1.34/#/v1.35/#' /etc/apt/sources.list.d/kubernetes.list && sudo apt-get update
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='1.35.2-1.1' && sudo apt-mark hold kubeadm
sudo kubeadm upgrade node                  # refreshes the local kubelet configuration
exit

k drain node-b --ignore-daemonsets --delete-emptydir-data   # from wherever kubectl works

ssh node-b
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.35.2-1.1' kubectl='1.35.2-1.1'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
exit

k uncordon node-b
k get nodes                                 # node-b now shows v1.35.2

Drain, cordon and uncordonCopy Anchor Link

• A drain that hangs is usually blocked by a PodDisruptionBudget that can't be satisfied, for example a single-replica Deployment with minAvailable: 1. Scale it up or wait, don't delete the PDB unless the task says so.
• DaemonSet Pods are never evicted, which is why --ignore-daemonsets is required.

CertificatesCopy Anchor Link

kubeadm-issued certificates are valid for one year. Upgrading at least once a year renews them automatically.

sudo kubeadm certs check-expiration
sudo kubeadm certs renew all            # or renew apiserver, renew admin.conf, ...
# static Pods don't reload certificates on their own: restart them
sudo mkdir -p /root/mf-hold
sudo mv /etc/kubernetes/manifests/*.yaml /root/mf-hold/ && sleep 20
sudo mv /root/mf-hold/*.yaml /etc/kubernetes/manifests/

renew all also rewrites admin.conf; copy it to ~/.kube/config again if you use that.

Moving the manifests out makes the kubelet stop those static Pods; moving them back starts them with the new certificates. Use an empty folder outside /etc/kubernetes/manifests, and on an HA cluster do one node at a time.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context cka-up

The cluster has control plane up-cp and worker up-w1, both on v1.34. Upgrade the control plane node only (control plane components, kubelet and kubectl) to the latest v1.35 patch available in the v1.35 package repo. Leave up-w1 untouched. Make sure up-cp is schedulable again at the end.

ssh up-cp
sudo sed -i 's#/v1.34/#/v1.35/#' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
apt-cache madison kubeadm | head -1          # e.g. 1.35.3-1.1
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm='1.35.3-1.1'
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.3 -y

k drain up-cp --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.35.3-1.1' kubectl='1.35.3-1.1'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
k uncordon up-cp
exit

k get nodes                     # up-cp v1.35.3 Ready (not SchedulingDisabled), up-w1 still v1.34.x
k version                       # server v1.35.3
k get pods -n kube-system       # all Running

Further readingCopy Anchor Link

• Upgrading kubeadm clusters
• Version skew policy
• Safely drain a node
• Certificate management with kubeadm