Packaging and rollouts
Helm charts, values and releases vs Kustomize bases and overlays, Deployment rolling updates and rollback, blue/green and canary releases, and progressive delivery with Argo Rollouts and Flagger.
Exam tasks: 3.1 (application delivery: packaging, release strategies, progressive delivery)
The decision: how do you produce the right manifests for each environment, and how do you swap the old version for the new one without taking users down with a bad release?
Packaging: Helm vs Kustomize
| Helm | Kustomize | |
|---|---|---|
| What it is | Package manager for Kubernetes. CNCF graduated | Template-free customization tool. A Kubernetes SIG CLI project, built into kubectl |
| Unit | Chart: templates plus values.yaml and Chart.yaml | Base plus overlays, each with a kustomization.yaml |
| How it varies config | Go templates filled from values (--set, -f values-prod.yaml) | Patches, name prefixes, labels, image overrides, generated ConfigMaps and Secrets |
| Tracks installs | Yes: each install is a release with numbered revisions | No: it just outputs YAML |
| Rollback | helm rollback <release> <revision> | Re-apply an older commit |
| Distribution | Chart repositories and OCI registries. Find public charts on Artifact Hub | Git directories |
| Run it | helm install, helm upgrade --install, helm uninstall | kubectl apply -k <dir>, kubectl kustomize <dir> |
Helm terms
- Chart: the package. Release: one installed instance of a chart in a cluster, with its own name. You can install the same chart twice as two releases.
- Values: inputs to the templates. Defaults come from the chart's
values.yaml, and you override them per environment. - Revision: every
upgradeorrollbackadds one.helm history <release>lists them. - Release state is stored in the cluster, as Secrets in the release's namespace by default.
- Dependencies: a chart can pull in other charts (a database subchart, for example).
Legacy: use Helm 3 and later: client-only, release state in Secrets instead
Helm 2 used a server-side component called Tiller running in the cluster with broad permissions. Helm 3
removed Tiller entirely: the helm client talks to the API server with your own credentials and RBAC. If an
option mentions Tiller, it's describing a version that hasn't been supported for years.
Kustomize layout
deploy/
base/ # shared Deployment, Service, kustomization.yaml
overlays/
staging/ # 1 replica, image tag :rc
production/ # 6 replicas, image tag :3.2.0, extra resource limits# deploy/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
namePrefix: prod-
images:
- name: ghcr.io/tidewater/ledger
newTag: "3.2.0"
patches:
- path: replicas.yamlExam signal
"Package manager", "chart", "release", "values", "Artifact Hub" all mean Helm. "Overlays", "patches without
templates", "kubectl apply -k" mean Kustomize. Neither one is a GitOps agent: Argo CD and Flux render both.
What Kubernetes does on its own
A Deployment has exactly two built-in strategies:
| Strategy | Behaviour | Downtime |
|---|---|---|
| RollingUpdate (default) | Creates a new ReplicaSet and shifts Pods over gradually, limited by maxSurge and maxUnavailable (both default 25%) | None if readiness probes are right |
| Recreate | Deletes all old Pods, then creates the new ones | Yes, by design. Use it when two versions can't run at the same time |
- A rollout starts when the Pod template changes (image, env, labels). Changing
replicasscales but doesn't create a new revision. kubectl rollout status deployment/ledgerwaits for it,kubectl rollout undo deployment/ledgergoes back to the previous ReplicaSet, andkubectl rollout historylists revisions (10 kept by default,revisionHistoryLimit).- Readiness probes gate the rollout: a new Pod only receives Service traffic, and only counts as available, once it's ready. Without a probe, a broken version can look healthy.
Blue/green and canary
| Rolling update | Blue/green | Canary | |
|---|---|---|---|
| Versions live at once | Both, briefly, mixed | Both, but only one gets traffic | Both, by traffic percentage |
| Switch | Pod by Pod | All traffic at once | Gradual steps (5% → 25% → 100%) |
| Rollback | New rollout backwards | Instant: point traffic back | Send the canary's share back to stable |
| Extra cost | Little | Double capacity during the switch | A few extra Pods |
| Built into Kubernetes | Yes | No: done by switching a Service's selector, or by a tool | No: needs weighted routing (Gateway API, Ingress controller, service mesh) or a tool |
- Blue/green by hand: run
ledger-blueandledger-greenDeployments and change the Service's selector fromversion: bluetoversion: green. - A crude canary by hand: two Deployments behind one Service, with a 9:1 replica ratio. The split is only as fine as your replica counts, which is why real canaries use weighted routing.
- A/B testing routes by user attribute (header, cookie, region) rather than by percentage, to compare behaviour. Feature flags (OpenFeature is a CNCF project) hide new code paths without redeploying.
Canary is a Deployment strategy
strategy.type accepts only RollingUpdate or Recreate. There's no Canary or BlueGreen value in a
Deployment. Options that set one are wrong; those strategies need extra objects or a tool such as Argo Rollouts.
Progressive delivery
Progressive delivery is canary or blue/green with automation: shift a little traffic, measure (error rate, latency from Prometheus or another metrics source), and either promote or roll back automatically.
| Argo Rollouts | Flagger | |
|---|---|---|
| Part of | Argo project (graduated) | Flux project (graduated) |
| How you use it | Replace the Deployment with a Rollout object that has canary or blue/green steps | Keep your Deployment and add a Canary object. Flagger creates the primary and canary copies |
| Analysis | AnalysisTemplate queries metrics between steps | Metric checks with thresholds on each interval |
| Traffic | Service meshes, Ingress controllers, Gateway API | Service meshes, Ingress controllers, Gateway API |
Exam signal
"Automatically roll back the canary if the error rate exceeds 1%" is progressive delivery: Argo Rollouts or
Flagger. A plain Deployment can't read metrics, and kubectl rollout undo is a manual step.
Scenarios
Helm installs a chart as a named release, takes per-cluster values and records revisions you can upgrade or roll
back. Kustomize can produce per-cluster YAML but has no release or revision tracking. Plain kubectl apply
handles neither variation nor versions. Recreate is a rollout strategy, not a packaging tool.
Weighted traffic steps plus automatic rollback on metrics is progressive delivery, which Argo Rollouts (or
Flagger) provides. Deployments have no Canary strategy. maxSurge limits extra Pods during a rolling update
but doesn't control traffic percentages or read metrics. Recreate causes downtime and has no automation.
Blue/green runs both versions in full and moves all traffic at once, so switching back is immediate, at the cost of double capacity. A rolling update replaces old Pods, so rollback means rolling again. Recreate deletes the old version first. A/B testing splits users by attribute to compare behaviour, not to keep a standby.
Further reading
GitOps and CI/CD
Continuous integration vs continuous delivery vs continuous deployment, infrastructure as code, the four OpenGitOps principles, push vs pull deployment, and how Argo CD and Flux implement GitOps.
Debugging applications
Debugging a running application on Kubernetes with logs, events, kubectl exec, port-forward, kubectl debug and ephemeral containers, and reading why a Deployment rollout is stuck.