Asterrr's Handbook

Deployments, rolling updates and rollbacks

How a Deployment rolls out a new revision through ReplicaSets, tuning maxSurge and maxUnavailable, Recreate, kubectl rollout history, undo, pause and restart, and spotting a stalled rollout.

Exam tasks: 2.1 (Deployments, rolling updates and rollbacks)

The decision: how do you change a running app's version so that capacity and downtime stay within the limits the task sets, and how do you get back to a known-good revision when it goes wrong?

How a rollout works

  • A revision is created only when the Pod template changes. Scaling, or editing strategy, doesn't create one.
  • Each revision is a ReplicaSet named <deployment>-<pod-template-hash>. Old ReplicaSets stay at 0 replicas so rollout undo can scale them back up. revisionHistoryLimit (default 10) caps how many are kept.
  • The selector is immutable in apps/v1. To change it, delete and recreate the Deployment.

Strategy parameters

FieldDefaultEffect
strategy.typeRollingUpdateRecreate kills every old Pod before starting new ones: brief outage, never two versions at once
rollingUpdate.maxSurge25% (rounded up)How many Pods above replicas may exist during the rollout
rollingUpdate.maxUnavailable25% (rounded down)How many Pods below replicas may be not Ready during the rollout
minReadySeconds0A new Pod counts as available only after it has been Ready this long
progressDeadlineSeconds600After this many seconds without progress the Deployment reports ProgressDeadlineExceeded
revisionHistoryLimit10Old ReplicaSets kept for rollback
spec:
  replicas: 6
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # up to 8 Pods in total
      maxUnavailable: 0  # never fewer than 6 available
  • maxSurge and maxUnavailable can't both be 0.
  • maxUnavailable: 0 with maxSurge: 1 is the "never lose capacity" setting, at the cost of a slower rollout and one Pod's worth of spare node capacity.
  • Use Recreate when two versions must never run together, for example an app that migrates a database schema on start or holds a ReadWriteOnce volume that only one node can mount.

Exam signal

"Zero downtime" or "at no point fewer than N Pods" means maxUnavailable: 0. "No more than N extra Pods" sets maxSurge. "The old and new versions must never serve traffic at the same time" means Recreate.

Commands you'll use

k create deploy ledger --image=registry.local/ledger:2.3 --replicas=4 -n pay
k set image deploy/ledger ledger=registry.local/ledger:2.4 -n pay   # container=image
k annotate deploy/ledger kubernetes.io/change-cause="ledger 2.4: new fee rules" -n pay
k rollout status deploy/ledger -n pay --timeout=120s
k rollout history deploy/ledger -n pay
k rollout history deploy/ledger -n pay --revision=3   # Pod template of revision 3
k rollout undo deploy/ledger -n pay                   # back to the previous revision
k rollout undo deploy/ledger -n pay --to-revision=2
k rollout pause deploy/ledger -n pay                  # batch several edits into one revision
k rollout resume deploy/ledger -n pay
k rollout restart deploy/ledger -n pay                # new Pods, same spec
  • In set image, the part before = is the container name, not the Deployment name. Check it with k get deploy ledger -o jsonpath='{.spec.template.spec.containers[*].name}'.
  • rollout restart works by stamping a kubectl.kubernetes.io/restartedAt annotation on the Pod template, so it does create a new revision. Use it to pick up changed ConfigMaps consumed as env vars.
  • rollout undo copies the old template forward as a new revision number. The revision you rolled back to disappears from the history list under its old number.
  • While paused, template edits pile up without rolling. resume rolls them out as one revision.

Legacy: use the kubernetes.io/change-cause annotation instead

The --record flag, which stored the command line in change-cause, is deprecated. Set the annotation yourself with kubectl annotate after each change so rollout history shows something useful.

Waiting for an automatic rollback

When a rollout passes progressDeadlineSeconds, Kubernetes only marks the Deployment's Progressing condition as False. It doesn't roll back on its own. The broken Pods sit in ImagePullBackOff or CrashLoopBackOff until you run rollout undo or fix the template.

Reading a rollout's state

k get deploy ledger -n pay            # READY, UP-TO-DATE, AVAILABLE
k get rs -n pay -l app=ledger         # one RS per revision, which one has the replicas
k describe deploy ledger -n pay       # Conditions and the scaling events per RS
ColumnMeans
UP-TO-DATEPods built from the current template
AVAILABLEPods Ready for at least minReadySeconds
Two ReplicaSets both with replicasA rollout is in progress, paused, or stuck
25% / 25%
Default maxSurge (rounded up) and maxUnavailable (rounded down).
10
Default revisionHistoryLimit: old ReplicaSets kept for undo.
600 s
Default progressDeadlineSeconds before ProgressDeadlineExceeded.
Template only
Only Pod template changes create a new revision.

Blue/green and canary with plain Deployments

  • Canary: run a second Deployment (ledger-canary, 1 replica) whose Pods share the label the Service selects on. Traffic splits roughly by Pod count.
  • Blue/green: run ledger-blue and ledger-green with a track label and switch the Service selector from one to the other with k patch svc. Rollback is switching it back.
  • For weighted splits that don't depend on Pod counts, use Gateway API HTTPRoute weights; see Gateway API.

Scenarios

Scenario
Deployment `checkout` in namespace `store` has 5 replicas. The task says the update to image `checkout:5.1` must never drop below 5 available Pods, and the cluster has room for only one extra Pod. Which strategy settings meet the requirement?
Scenario
After `kubectl set image deploy/quotes quotes=quotes:9.9`, `kubectl get rs` shows the new ReplicaSet with 1 Pod in ImagePullBackOff and the old ReplicaSet still at 3 of 4 Pods. Fifteen minutes later nothing has changed. What happened, and what is the fastest fix?
Scenario
You need to change the image, an environment variable and the memory limit of Deployment `relay`, and the task says the change must produce exactly one new revision. What should you do?

Drill

kubectl config use-context drill-w1. In namespace atlas:

  1. Create Deployment tiles with 4 replicas of nginx:1.27-alpine, container name tiles.
  2. Configure it so a rollout never has fewer than 4 available Pods and adds at most 1 Pod at a time.
  3. Update the image to nginx:1.28-alpine, recording the change cause upgrade to 1.28.
  4. Roll back to the first revision and confirm the image.

Further reading

On this page