Container logs

Exam tasks: 5.4 (manage and evaluate container output streams)

The decision: can kubectl logs reach the output you need (right container, right instance, right time window), or do you have to read it on the node, or first make the app's output visible at all?

Where container output goesCopy Anchor Link

• Only stdout and stderr reach kubectl logs. A file the app writes inside its filesystem doesn't.
• The kubelet keeps the current container's log and the log of the previous instance after a restart. When the Pod is deleted, its log files go too.
• The kubelet rotates logs: containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5) in the kubelet config. kubectl logs reads only the latest file.

kubectl logs, flag by flagCopy Anchor Link

Saving logs for the graderCopy Anchor Link

Tasks often say "write the log lines containing ERROR to /opt/out/errors.log".

k logs ledger-0 -c api -n vault | grep ERROR > /opt/out/errors.log
k logs ledger-0 -c api -n vault --previous > /opt/out/crash.log 2>&1
wc -l /opt/out/errors.log            # verify something was written

kubectl logs writes the log to stdout and its own warnings (such as the default-container notice) to stderr. Redirect only stdout unless the task asks for everything.

When kubectl logs can't helpCopy Anchor Link

• API server down: read files directly on the node: sudo tail -50 /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log, or sudo crictl logs <container-id>. See Troubleshooting the control plane.
• Node services aren't containers: kubelet and containerd log to the systemd journal. sudo journalctl -u kubelet --since "30 min ago", sudo journalctl -u containerd -e.
• The app writes to a file: add a sidecar that streams the file to its own stdout.

Streaming a log file through a sidecarCopy Anchor Link

A native sidecar is an init container with restartPolicy: Always. It starts before the app, keeps running alongside it, and doesn't block Pod completion.

apiVersion: v1
kind: Pod
metadata:
  name: tally
  namespace: vault
spec:
  initContainers:
  - name: log-tail
    image: busybox:1.37
    restartPolicy: Always            # makes it a sidecar
    command: ["sh", "-c", "exec tail -F /var/log/tally/app.log"]
    volumeMounts:
    - name: logs
      mountPath: /var/log/tally
  containers:
  - name: tally
    image: busybox:1.37
    command: ["sh", "-c", "while true; do echo \"$(date) tally tick\" >> /var/log/tally/app.log; sleep 5; done"]
    volumeMounts:
    - name: logs
      mountPath: /var/log/tally
  volumes:
  - name: logs
    emptyDir: {}

Then k logs tally -n vault -c log-tail -f. A sidecar under containers (the older pattern) works the same for logs; the native form is preferred when the Pod belongs to a Job, because it won't keep the Job from finishing.

Cluster-level logging patternsCopy Anchor Link

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context lab-logs. Pod quarry in namespace strata runs two containers. Save the log lines containing WARN from the previous instance of whichever container has restarted to /opt/drill/warn.log.

k get pod quarry -n strata \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\n"}{end}'
# example output: digger 0, sorter 4  -> sorter is the one restarting
k logs quarry -n strata -c sorter --previous | grep WARN > /opt/drill/warn.log

head -3 /opt/drill/warn.log
grep -vc WARN /opt/drill/warn.log     # 0: only WARN lines were saved

Further readingCopy Anchor Link

• Logging architecture
• kubectl logs
• Sidecar containers
• System logs