Resources, QoS and quotas

Exam tasks: 2.5 (configure Pod admission and scheduling: limits)

The decision: how much CPU and memory should each container reserve and be capped at, and which namespace guardrail (LimitRange or ResourceQuota) enforces that for everyone?

Requests, limits and where each one actsCopy Anchor Link

resources:
  requests:
    cpu: 250m          # 0.25 core
    memory: 192Mi      # Mi = 2^20 bytes; M = 10^6
  limits:
    cpu: "1"
    memory: 384Mi

k set resources deploy/orbit -n space -c orbit --requests=cpu=250m,memory=192Mi --limits=cpu=1,memory=384Mi
k describe node worker-2 | grep -A8 'Allocated resources'   # requests and limits already committed

• A Pod that fits no node by requests stays Pending with Insufficient cpu or Insufficient memory in its events, even if the nodes are idle in practice. Scheduling never looks at actual usage.
• cpu: 1 and cpu: 1000m are equal. memory: 1G is about 7% less than 1Gi.

QoS classesCopy Anchor Link

k get pod orbit-7d9c -n space -o jsonpath='{.status.qosClass}{"\n"}'

• QoS is computed from the spec at creation and can't change afterwards, not even by in-place resize.
• Setting only limits gives requests equal to limits, so a Pod whose every container sets only CPU and memory limits is Guaranteed.

LimitRange: per-object defaults and boundsCopy Anchor Link

apiVersion: v1
kind: LimitRange
metadata:
  name: container-bounds
  namespace: space
spec:
  limits:
  - type: Container
    defaultRequest:         # injected when a container sets no request
      cpu: 100m
      memory: 128Mi
    default:                # injected when a container sets no limit
      cpu: 500m
      memory: 256Mi
    min:
      cpu: 50m
      memory: 64Mi
    max:
      cpu: "2"
      memory: 1Gi
  - type: PersistentVolumeClaim
    max:
      storage: 20Gi

• Applied by the LimitRanger admission plugin when a Pod is created. Existing Pods are untouched, so restart the workload to see the defaults.
• A container that asks for more than max (or less than min) is rejected with a Forbidden error.
• type: Pod bounds the sum across containers; maxLimitRequestRatio caps limit divided by request.

ResourceQuota: namespace totalsCopy Anchor Link

k create quota space-quota -n space \
  --hard=requests.cpu=4,requests.memory=8Gi,limits.cpu=8,limits.memory=16Gi,pods=30,services.loadbalancers=0
k describe quota space-quota -n space      # Used vs Hard

• With a quota on requests.cpu, every new Pod must set a CPU request (on limits.memory, a memory limit, and so on), or admission rejects it. A LimitRange with defaults is the usual companion.
• scopeSelector limits a quota to some Pods: BestEffort, NotBestEffort, Terminating, NotTerminating, or a PriorityClass (for example "at most 2 Pods with priority critical-batch in this namespace").
• Quotas are checked at admission only. Lowering a quota below current use doesn't evict anything; it blocks new objects until usage drops.

The admission chain in shortCopy Anchor Link

• Requests pass authentication, authorization, then mutating admission (LimitRanger defaults, ServiceAccount, mutating webhooks), schema validation, then validating admission (ResourceQuota, Pod Security Admission, ValidatingAdmissionPolicy, validating webhooks). Any one can reject the object.
• Plugins are toggled on kube-apiserver with --enable-admission-plugins / --disable-admission-plugins. LimitRanger and ResourceQuota are on by default.
• Pod Security Admission is driven by namespace labels such as pod-security.kubernetes.io/enforce=restricted; a Pod that breaks the level is rejected at creation.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context drill-w5. In namespace quarry:

1. Make sure any container created without resources gets a request of 100m CPU and 128Mi memory and a limit of 300m CPU and 256Mi memory.
2. Limit the namespace to 10 Pods and a total of 2 CPU in requests.
3. Create Pod probe (image nginx:1.27-alpine, no resources in the spec) and confirm what it received.

apiVersion: v1
kind: LimitRange
metadata:
  name: quarry-defaults
  namespace: quarry
spec:
  limits:
  - type: Container
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    default:
      cpu: 300m
      memory: 256Mi

kubectl config use-context drill-w5
k apply -f quarry-limits.yaml
k create quota quarry-quota -n quarry --hard=pods=10,requests.cpu=2
k run probe -n quarry --image=nginx:1.27-alpine
k get pod probe -n quarry -o jsonpath='{.spec.containers[0].resources}{"\n"}'
# {"limits":{"cpu":"300m","memory":"256Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}
k describe quota quarry-quota -n quarry    # pods 1/10, requests.cpu 100m/2

Further readingCopy Anchor Link

• Resource management for Pods and containers
• Pod Quality of Service classes
• Limit ranges
• Resource quotas