Extension interfaces (CRI, CNI, CSI)
What the kubelet delegates to container runtimes, network plugins and storage drivers, where each one is configured on a node, and how to tell which plugin is installed.
Exam tasks: 1.7 (understand extension interfaces: CNI, CSI, CRI and so on)
The decision: which plugin owns this behaviour (running containers, giving Pods an IP, attaching a volume), and where on the node or in the API do you look to inspect or fix it?
What plugs in where
| Interface | Kubernetes asks it to | Called by | Lives on the node at |
|---|---|---|---|
| CRI (Container Runtime Interface) | Pull images, create Pod sandboxes, start and stop containers, stream logs and exec | kubelet, over gRPC | Socket /run/containerd/containerd.sock or /var/run/crio/crio.sock |
| CNI (Container Network Interface) | Attach a network interface and IP to a new Pod sandbox, remove it on deletion | Container runtime, by running a binary | Config /etc/cni/net.d/, binaries /opt/cni/bin/ |
| CSI (Container Storage Interface) | Create, attach, mount, resize and snapshot volumes | kubelet (node side) and sidecar controllers (cluster side), over gRPC | Node plugin socket under /var/lib/kubelet/plugins/ |
All three are out-of-tree: the vendor ships the plugin, Kubernetes ships only the interface. That's why swapping the runtime, network or storage doesn't need a new Kubernetes build.
CRI: the container runtime
- The kubelet's
containerRuntimeEndpoint(in/var/lib/kubelet/config.yaml) points at the runtime socket. kubeadm detects it at init and join, or takes--cri-socket. crictlspeaks CRI directly, so it works with any runtime. It's your tool when the API server is down.
sudo crictl ps -a # containers, including exited ones
sudo crictl pods # Pod sandboxes
sudo crictl logs <container-id>
sudo crictl images
cat /etc/crictl.yaml # runtime-endpoint: unix:///run/containerd/containerd.sock- A RuntimeClass picks a different OCI handler configured in the runtime (for example gVisor or Kata for
sandboxed workloads). Pods opt in with
spec.runtimeClassName.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: sandboxed
handler: runsc # must match a handler name in the containerd configLegacy: use containerd or CRI-O through the CRI instead
The kubelet's built-in Docker support (dockershim) was removed in v1.24. docker ps on a node shows nothing for
Kubernetes Pods on a containerd node. Use crictl. If Docker Engine must stay, it needs the external
cri-dockerd adapter.
CNI: Pod networking
- When a Pod sandbox starts, the runtime reads the lowest-sorted config file in
/etc/cni/net.d/and runs the plugin binaries from/opt/cni/bin/it names. The plugin creates the interface and allocates the IP. - Most plugins run a DaemonSet that writes that config and the binaries onto each node, plus an agent that programs routes, tunnels or eBPF.
- Until a config exists, the node reports
NetworkReady=falseand staysNotReady.
| Plugin | Data path | NetworkPolicy |
|---|---|---|
| Calico | BGP routing or VXLAN/IP-in-IP overlay, optional eBPF | Yes, plus its own extended policies |
| Cilium | eBPF; can replace kube-proxy | Yes, plus L7 policies |
| Flannel | VXLAN overlay | Not in a default install; needs an add-on policy engine |
| Cloud plugins (AWS VPC CNI, Azure CNI) | Pods get addresses from the cloud network | Varies, often with an add-on |
ls /etc/cni/net.d/ # which plugin wrote config
k get pods -n kube-system -o wide | grep -Ei 'calico|cilium|flannel'
k get ds -A # the CNI agent DaemonSetNetworkPolicies that do nothing
Kubernetes stores NetworkPolicy objects whatever the CNI, but enforcement is entirely the plugin's job. On a plain Flannel cluster, a default-deny policy is accepted and silently ignored. If policies "don't work", check the plugin before rewriting the YAML.
Two CNI configs on one node
If a second plugin's file sorts before yours in /etc/cni/net.d/ (for example a leftover 10-flannel.conflist
ahead of 20-cilium.conflist), new Pods get the wrong network. Remove leftovers after switching plugins.
CSI: storage drivers
- Controller plugin (a Deployment or StatefulSet) with sidecars:
external-provisionercreates volumes for PVCs,external-attacherhandles VolumeAttachments,external-resizerandexternal-snapshotterdo expansion and snapshots. - Node plugin (a DaemonSet) with
node-driver-registrar, which registers the driver's socket with the kubelet. The kubelet calls it to stage and mount volumes into Pods. - A StorageClass's
provisionerfield is the CSI driver name, for exampleebs.csi.aws.comor a name your driver publishes.
k get csidrivers # installed drivers and their capabilities
k get csinodes # which drivers registered on which node
k get volumeattachments # attach state per PV and node
k get sc # provisioner column = CSI driver nameExam signal
"A PVC stays Pending with a StorageClass set" often means the CSI driver named in provisioner isn't installed
or its controller isn't running. k get csidrivers and the controller Pod's logs answer it fast. "Pod stuck
ContainerCreating with a mount error" points to the node plugin on that node.
Legacy: use CSI drivers instead
In-tree volume plugins (awsElasticBlockStore, gcePersistentDisk, azureDisk, cinder and others) have been
migrated to CSI and removed from the kubelet. Old manifests using them are redirected to the CSI driver or fail if
it isn't installed. Write new StorageClasses against the CSI driver name.
Other extension points
| Mechanism | Extends |
|---|---|
| Device plugins | Advertise hardware such as GPUs as extended resources (nvidia.com/gpu) |
| Dynamic Resource Allocation | Claims for devices with parameters, through ResourceClaims and DeviceClasses |
| CustomResourceDefinitions and API aggregation | The API itself (see CRDs and operators) |
| Admission webhooks | Validation and mutation of API requests |
Scenarios
crictl talks straight to the runtime over the CRI, so it works without the API server and with any CRI runtime.
Docker isn't the runtime on current nodes, kubectl needs the API server, and on kubeadm clusters the API server
is a static Pod, not a systemd unit.
The API accepts NetworkPolicies regardless, but enforcement is the CNI plugin's job, and a default Flannel install doesn't do it. Calico or Cilium (or Flannel paired with a policy engine) would. kube-proxy and RBAC have nothing to do with policy enforcement, and an ingress default-deny doesn't need an Egress type.
Drill
kubectl config use-context cka-ext
Answer these about the cluster and write each answer on its own line in /opt/answers/plugins.txt:
(1) the CRI socket path the kubelet on node ext-w1 uses, (2) the CNI plugin installed, (3) the provisioner of the
default StorageClass.
ssh ext-w1
sudo grep containerRuntimeEndpoint /var/lib/kubelet/config.yaml
# or: ps -ef | grep [k]ubelet (look for --container-runtime-endpoint)
ls /etc/cni/net.d/
exit
k get pods -n kube-system -o wide | grep -Ei 'calico|cilium|flannel|weave'
k get sc # the (default) row, PROVISIONER columncat > /opt/answers/plugins.txt <<'EOF'
unix:///run/containerd/containerd.sock
cilium
csi.hostpath.example.io
EOF
cat /opt/answers/plugins.txtUse the values you actually find; the ones above are examples. If containerRuntimeEndpoint isn't in the config
file, check the kubelet's flags in /var/lib/kubelet/kubeadm-flags.env and the running command line.
Further reading
Helm and Kustomize
Installing, upgrading and rolling back Helm releases with values, inspecting charts and releases, and building Kustomize bases and overlays you apply with kubectl apply -k.
CRDs and operators
Writing a CustomResourceDefinition with a schema, versions and printer columns, working with custom resources through kubectl, and installing and checking an operator.