etcd backup and restore

Exam tasks: 1.4 (manage the lifecycle of Kubernetes clusters)

The decision: where is etcd listening, which certificates does etcdctl need to talk to it, and what must change in the static Pod manifest so the cluster runs from the restored data?

The two toolsCopy Anchor Link

Finding the endpoint and certificatesCopy Anchor Link

Never guess the paths: read them from the etcd static Pod manifest on the control plane node.

ssh cp-main
sudo grep -E 'listen-client-urls|cert-file|key-file|trusted-ca-file|data-dir' /etc/kubernetes/manifests/etcd.yaml

The server certificate works as a client certificate on kubeadm clusters. So do /etc/kubernetes/pki/etcd/healthcheck-client.crt and the API server's /etc/kubernetes/pki/apiserver-etcd-client.crt.

Saving a snapshotCopy Anchor Link

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /srv/etcd/snap-0412.db

sudo etcdutl --write-out=table snapshot status /srv/etcd/snap-0412.db

• sudo matters: the keys under /etc/kubernetes/pki are readable by root only.
• The snapshot comes from one member. On an HA cluster, any healthy member is fine.
• No etcdctl on the host? The etcd image ships it. Run it in the Pod and write under /var/lib/etcd, which is a hostPath mount, so the file appears on the node: k -n kube-system exec etcd-cp-main -- etcdctl --endpoints=... --cacert=... --cert=... --key=... snapshot save /var/lib/etcd/snap.db

Restoring on a kubeadm control planeCopy Anchor Link

ssh cp-main
sudo mkdir -p /root/mf-hold
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/mf-hold/

sudo etcdutl snapshot restore /srv/etcd/snap-0412.db --data-dir /var/lib/etcd-restored

sudo vi /etc/kubernetes/manifests/etcd.yaml

Change only the hostPath of the data volume. The container still sees /var/lib/etcd, so --data-dir and the volumeMounts stay as they are:

  volumes:
  - hostPath:
      path: /var/lib/etcd-restored      # was /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data

# wait until etcd is running again
sudo crictl ps --name etcd
sudo mv /root/mf-hold/kube-apiserver.yaml /etc/kubernetes/manifests/
# give the API server a minute, then
k get nodes
k get deploy -A           # objects from the snapshot time are back

• The kubelet restarts etcd as soon as the manifest changes. Editing a copy elsewhere does nothing.
• Restoring into a new directory keeps the old data as a fallback and avoids "data dir not empty" errors.
• Restarting kube-controller-manager, kube-scheduler and the kubelets afterwards is recommended, so nothing works from stale cached state.

Multi-member restoresCopy Anchor Link

On a multi-member cluster, restore the same snapshot on every member, each with its own identity, so the members form a new cluster together:

sudo etcdutl snapshot restore /srv/etcd/snap-0412.db \
  --name cp-main \
  --initial-cluster cp-main=https://10.20.0.11:2380,cp-b=https://10.20.0.12:2380,cp-c=https://10.20.0.13:2380 \
  --initial-advertise-peer-urls https://10.20.0.11:2380 \
  --data-dir /var/lib/etcd-restored

Take --name and the URLs from each node's etcd.yaml. For a single-member lab cluster you can leave them out.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

ssh cka-etcd-cp

1. Save a snapshot of the cluster's etcd to /opt/vault/etcd-pre.db.
2. Then restore the snapshot at /opt/vault/etcd-golden.db (already on the node) so the cluster runs from it. Use /var/lib/etcd-golden as the new data directory.

ssh cka-etcd-cp
sudo grep -E 'listen-client-urls|cert-file|key-file|trusted-ca-file' /etc/kubernetes/manifests/etcd.yaml

# 1. save
sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/vault/etcd-pre.db
sudo etcdutl --write-out=table snapshot status /opt/vault/etcd-pre.db

# 2. restore
sudo mkdir -p /root/mf-hold
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/mf-hold/
sudo etcdutl snapshot restore /opt/vault/etcd-golden.db --data-dir /var/lib/etcd-golden
sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-golden#' /etc/kubernetes/manifests/etcd.yaml
sudo grep -A2 'hostPath' /etc/kubernetes/manifests/etcd.yaml     # check only the etcd-data volume changed
sudo crictl ps --name etcd                                       # wait until Running
sudo mv /root/mf-hold/kube-apiserver.yaml /etc/kubernetes/manifests/

k get nodes
k get all -A | head          # objects match the golden snapshot, not the current state
sudo crictl ps | grep -E 'etcd|kube-apiserver'

Further readingCopy Anchor Link

• Operating etcd clusters for Kubernetes
• etcd disaster recovery
• Static Pods