Pod placement

Exam tasks: 2.5 (configure Pod admission and scheduling: node affinity and related placement controls)

The decision: must this Pod run on (or avoid) certain nodes, should it be attracted by labels or should other Pods be repelled by taints, and what wins when the cluster is full?

How the scheduler places a PodCopy Anchor Link

Labels, nodeSelector and node affinityCopy Anchor Link

k label node worker-3 storage=nvme
k label node worker-3 storage-           # remove the label
k get nodes -L storage,topology.kubernetes.io/zone

Well-known labels you can rely on: kubernetes.io/hostname, kubernetes.io/os, kubernetes.io/arch, topology.kubernetes.io/zone, topology.kubernetes.io/region, node.kubernetes.io/instance-type.

spec:
  nodeSelector:
    kubernetes.io/os: linux
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:            # terms are ORed
        - matchExpressions:           # expressions inside one term are ANDed
          - key: storage
            operator: In
            values: ["nvme", "ssd"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50                    # 1 to 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["zone-b"]

• IgnoredDuringExecution: removing the label later doesn't evict running Pods.
• nodeSelector and node affinity together must both match.
• NotIn and DoesNotExist express "keep off these nodes" without touching the nodes.

Pod affinity and anti-affinityCopy Anchor Link

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: sentry-cache
      topologyKey: kubernetes.io/hostname    # at most one sentry-cache Pod per node
  podAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80
      podAffinityTerm:
        labelSelector:
          matchLabels:
            app: sentry-api
        topologyKey: topology.kubernetes.io/zone   # same zone as the API if possible

• topologyKey defines the "place": a node (hostname) or a zone. It must be a label on the nodes.
• Required anti-affinity per node with more replicas than nodes leaves the extras Pending. Use the preferred form, or topology spread, when replicas can outnumber nodes.

Taints and tolerationsCopy Anchor Link

k taint nodes worker-4 dedicated=analytics:NoSchedule
k taint nodes worker-4 dedicated=analytics:NoSchedule-     # remove (note the trailing -)
k describe node worker-4 | grep -i taints

tolerations:
- key: dedicated
  operator: Equal          # Exists matches any value; omit value with Exists
  value: analytics
  effect: NoSchedule
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 60    # leave an unreachable node after 60 s instead of 300 s

• kubeadm taints control plane nodes with node-role.kubernetes.io/control-plane:NoSchedule. DaemonSets that must run there need that toleration.
• The node controller adds node.kubernetes.io/not-ready and node.kubernetes.io/unreachable NoExecute taints to failing nodes. Every Pod gets default tolerations for them of 300 seconds, which is why Pods take about 5 minutes to move off a dead node.

Topology spread constraintsCopy Anchor Link

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule   # or ScheduleAnyway (soft)
  labelSelector:
    matchLabels:
      app: sentry-api
  matchLabelKeys: ["pod-template-hash"]   # count only Pods of the same rollout revision

• maxSkew is the largest allowed difference in matching Pods between the fullest and emptiest domain.
• Unlike required anti-affinity, it scales past one Pod per domain: 9 replicas in 3 zones go 3/3/3.
• minDomains (with DoNotSchedule) treats missing domains as having zero Pods, forcing spread across at least that many.

PriorityClass and preemptionCopy Anchor Link

k create priorityclass payments-critical --value=100000 --description="customer-facing payments"
k create priorityclass batch-low --value=100 --preemption-policy=Never

spec:
  priorityClassName: payments-critical

• Higher value schedules first. When no node fits, the scheduler may evict lower-priority Pods to make room (preemption). PDBs are respected on a best-effort basis only.
• preemptionPolicy: Never keeps the queue priority but never evicts others.
• globalDefault: true on one class sets the priority for Pods without a class; otherwise they get 0.
• Built-in system-cluster-critical and system-node-critical (about 2 billion) are reserved for cluster components.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context drill-w6. In namespace harbor:

1. Label node worker-2 with tier=edge and taint it tier=edge:NoSchedule.
2. Create Deployment beacon (image nginx:1.27-alpine, 2 replicas) whose Pods run only on tier=edge nodes.
3. Create PriorityClass beacon-high with value 50000 and use it for beacon.

kubectl config use-context drill-w6
k label node worker-2 tier=edge
k taint node worker-2 tier=edge:NoSchedule
k create priorityclass beacon-high --value=50000
k create deploy beacon -n harbor --image=nginx:1.27-alpine --replicas=2 --dry-run=client -o yaml > beacon.yaml

      priorityClassName: beacon-high
      tolerations:
      - key: tier
        operator: Equal
        value: edge
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: tier
                operator: In
                values: ["edge"]

k apply -f beacon.yaml
k get pods -n harbor -l app=beacon -o wide                  # NODE is worker-2 for both
k get pods -n harbor -l app=beacon -o jsonpath='{.items[*].spec.priority}{"\n"}'   # 50000 50000

Further readingCopy Anchor Link

• Assigning Pods to nodes
• Taints and tolerations
• Pod topology spread constraints
• Pod priority and preemption