Workload autoscaling

Exam tasks: 2.3 (configure workload autoscaling)

The decision: should the workload get more Pods (horizontal), bigger Pods (vertical), or a one-off replica change, and what has to be in place for the autoscaler to see any metrics at all?

Which scalerCopy Anchor Link

Manual scalingCopy Anchor Link

k scale deploy/ingest --replicas=6 -n data
k scale sts/ledger-db --replicas=3 -n data             # StatefulSets scale one ordinal at a time
k scale deploy/ingest --current-replicas=6 --replicas=2 -n data   # only if it is currently 6

• If an HPA targets the workload, it overrides your manual change on its next sync. Edit the HPA's minReplicas/maxReplicas instead.
• kubectl apply of a manifest that sets replicas fights the HPA. Leave replicas out of manifests for autoscaled workloads.

HorizontalPodAutoscalerCopy Anchor Link

k autoscale deploy/ingest -n data --min=2 --max=12 --cpu=65%
k get hpa -n data          # TARGETS shows current/target, e.g. 41%/65%
k describe hpa ingest -n data

--cpu accepts a percentage (65%, utilization of requests) or a quantity (500m, average value), and --memory works the same way. Older kubectl releases only had --cpu-percent=65.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ingest
  namespace: data
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ingest
  minReplicas: 2
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization        # percent of the Pods' CPU *requests*
        averageUtilization: 65
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue       # absolute value per Pod
        averageValue: 400Mi
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 120   # default 300
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60               # remove at most 2 Pods per minute

How it decides:

• desired = ceil(currentReplicas × currentMetric / targetMetric). At 4 Pods averaging 130% against a 65% target, it asks for 8.
• With several metrics it computes each and takes the highest replica count.
• It ignores changes within a 10% tolerance of the target, and scale-down uses the highest recommendation from the last 5 minutes (the stabilization window), so it shrinks slowly by design.
• Other metric types: Pods and Object (custom metrics) and External need a metrics adapter such as Prometheus Adapter or KEDA. Only Resource works with metrics-server alone.

metrics-serverCopy Anchor Link

k apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
k -n kube-system rollout status deploy/metrics-server
k top nodes

In lab clusters whose kubelets use self-signed serving certificates, metrics-server can't scrape them. Adding --kubelet-insecure-tls to the metrics-server container args gets it working there; it isn't a production fix.

VerticalPodAutoscalerCopy Anchor Link

• An add-on from kubernetes/autoscaler: CRDs plus a recommender, an updater and an admission controller.
• It watches usage and sets requests (limits move proportionally) through updatePolicy.updateMode:

• Don't let a VPA and an HPA both act on CPU or memory for the same workload: VPA raises requests, which lowers utilization, which makes the HPA scale in. Pair VPA with an HPA on custom metrics, or use VPA in Off mode for sizing advice.

In-place Pod resizeCopy Anchor Link

Stable in v1.35: you can change CPU and memory of a running Pod's containers without recreating the Pod.

k patch pod ingest-0 -n data --subresource=resize -p \
  '{"spec":{"containers":[{"name":"ingest","resources":{"requests":{"cpu":"750m"},"limits":{"cpu":"1500m"}}}]}}'
k get pod ingest-0 -n data -o jsonpath='{.status.containerStatuses[0].resources}{"\n"}'

• The change goes through the resize subresource (kubectl edit pod ... --subresource=resize also works). An ordinary update of a Pod's resources is rejected.
• resizePolicy per resource chooses NotRequired (default, apply live) or RestartContainer (restart that container to apply, common for memory in JVM apps).
• The QoS class can't change. A resize that would turn a Guaranteed Pod into Burstable is refused.
• Pod conditions show progress: PodResizePending (reason Deferred or Infeasible when the node lacks room) and PodResizeInProgress.
• Resizing a Pod owned by a Deployment is temporary: the next rollout uses the template values. Change the template for anything permanent.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context drill-w3. Deployment render in namespace media runs 1 replica and has no resource settings. metrics-server is installed.

1. Give the render container a CPU request of 200m.
2. Create an HPA named render that keeps between 2 and 7 replicas at 70% average CPU utilization, and scales down by at most 1 Pod per 30 seconds.

kubectl config use-context drill-w3
k set resources deploy/render -n media -c render --requests=cpu=200m
k rollout status deploy/render -n media

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: render
  namespace: media
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: render
  minReplicas: 2
  maxReplicas: 7
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      policies:
      - type: Pods
        value: 1
        periodSeconds: 30

k apply -f render-hpa.yaml
k get hpa render -n media          # TARGETS shows a number like 3%/70% after a minute
k get deploy render -n media       # replicas raised to 2 by the HPA minimum

Further readingCopy Anchor Link

• Horizontal Pod Autoscaling
• HorizontalPodAutoscaler walkthrough
• Resize CPU and memory resources assigned to containers
• Vertical Pod Autoscaling