Self-healing workloads

Exam tasks: 2.4 (understand the primitives used to create robust, self-healing application deployments)

The decision: which controller should own these Pods, how does Kubernetes know a container is broken or not ready yet, and how do you stop voluntary disruptions such as a node drain from taking the app down?

Pick the controllerCopy Anchor Link

Restart policy and backoffCopy Anchor Link

• restartPolicy is Pod-wide: Always (default, required for Deployment, StatefulSet, DaemonSet Pods), OnFailure or Never (the only values a Job allows).
• The kubelet restarts failed containers in place on the same node, with exponential backoff: 10 s, 20 s, 40 s and so on, capped at 5 minutes. That waiting state is CrashLoopBackOff. The delay resets after a container runs cleanly for 10 minutes.
• Rescheduling to another node is not the kubelet's job. If a node dies, the owning controller creates a replacement Pod, which the scheduler places elsewhere. A bare Pod with no controller is simply gone.

ProbesCopy Anchor Link

containers:
- name: quay
  image: registry.local/quay-api:7.0
  ports:
  - containerPort: 9090
  startupProbe:
    httpGet: { path: /healthz, port: 9090 }
    periodSeconds: 5
    failureThreshold: 24          # up to 120 s to boot
  livenessProbe:
    httpGet: { path: /healthz, port: 9090 }
    periodSeconds: 10
    failureThreshold: 3
  readinessProbe:
    tcpSocket: { port: 9090 }
    periodSeconds: 5

• Mechanisms: httpGet (2xx or 3xx passes), tcpSocket (port accepts), exec (exit code 0), grpc (gRPC health checking protocol).
• Defaults: periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1, initialDelaySeconds 0.
• Prefer a startupProbe over a long initialDelaySeconds for slow starters: it waits only as long as the app actually needs.

Native sidecar containersCopy Anchor Link

A sidecar is an init container with restartPolicy: Always. Stable since v1.33.

spec:
  initContainers:
  - name: log-forwarder
    image: fluent/fluent-bit:4.0
    restartPolicy: Always          # this line makes it a sidecar
    volumeMounts:
    - { name: logs, mountPath: /var/log/app }
  containers:
  - name: app
    image: registry.local/billing:2.2
    volumeMounts:
    - { name: logs, mountPath: /var/log/app }
  volumes:
  - name: logs
    emptyDir: {}

• It starts before the main containers (in init order) and keeps running alongside them; later init containers start once it has started (or its startupProbe passes).
• It's stopped after the main containers on shutdown, so it can flush logs or proxy final requests.
• It doesn't block a Job from completing, the problem with old-style sidecars in containers.
• It's restarted on failure regardless of the Pod's restartPolicy, and it can have probes.

Jobs and CronJobsCopy Anchor Link

k create job db-migrate --image=registry.local/migrator:1.4 -n ops -- /migrate --up
k create cronjob nightly-export --image=registry.local/export:3 --schedule="15 2 * * *" -n ops -- /export
k create job export-now --from=cronjob/nightly-export -n ops   # run a CronJob once, now

• podFailurePolicy can fail a Job immediately on a specific exit code or ignore disruptions (such as eviction during a drain) so they don't count against backoffLimit.

PodDisruptionBudgetsCopy Anchor Link

k create pdb quay-pdb -n api --selector=app=quay --min-available=2
# or: --max-unavailable=1
k get pdb -n api        # ALLOWED DISRUPTIONS

• A PDB limits voluntary disruptions that go through the Eviction API: kubectl drain, cluster autoscaler scale-down, node upgrades. It doesn't stop kubectl delete pod, node crashes or OOM kills.
• Set either minAvailable or maxUnavailable, as a number or percentage. maxUnavailable follows scaling better.
• unhealthyPodEvictionPolicy: AlwaysAllow lets a drain evict Pods that are already not Ready, so a crash-looping app can't block node maintenance forever.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context drill-w4. In namespace edge:

1. Create a DaemonSet node-probe running busybox:1.36 with sleep 86400, that also runs on control plane nodes.
2. Create a CronJob tidy that runs busybox:1.36 with echo tidy every 30 minutes, never runs two at once, and keeps only 1 successful Job. Trigger one run now as Job tidy-manual.

kubectl config use-context drill-w4
k create deploy node-probe --image=busybox:1.36 -n edge --dry-run=client -o yaml -- sleep 86400 > ds.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-probe
  namespace: edge
spec:
  selector:
    matchLabels:
      app: node-probe
  template:
    metadata:
      labels:
        app: node-probe
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      containers:
      - name: busybox
        image: busybox:1.36
        command: ["sleep", "86400"]

k apply -f ds.yaml
k get ds node-probe -n edge          # DESIRED equals the number of nodes
k get pods -n edge -o wide -l app=node-probe

k create cronjob tidy -n edge --image=busybox:1.36 --schedule="*/30 * * * *" \
  --dry-run=client -o yaml -- echo tidy > tidy.yaml
# add under spec: concurrencyPolicy: Forbid and successfulJobsHistoryLimit: 1
k apply -f tidy.yaml
k create job tidy-manual --from=cronjob/tidy -n edge
k get cronjob tidy -n edge -o jsonpath='{.spec.concurrencyPolicy} {.spec.successfulJobsHistoryLimit}{"\n"}'
k logs job/tidy-manual -n edge       # tidy

Further readingCopy Anchor Link

• Workload management
• Configure liveness, readiness and startup probes
• Sidecar containers
• Specifying a disruption budget for your application