Asterrr's Handbook

Extension interfaces (CRI, CNI, CSI)

What the kubelet delegates to container runtimes, network plugins and storage drivers, where each one is configured on a node, and how to tell which plugin is installed.

Exam tasks: 1.7 (understand extension interfaces: CNI, CSI, CRI and so on)

The decision: which plugin owns this behaviour (running containers, giving Pods an IP, attaching a volume), and where on the node or in the API do you look to inspect or fix it?

What plugs in where

InterfaceKubernetes asks it toCalled byLives on the node at
CRI (Container Runtime Interface)Pull images, create Pod sandboxes, start and stop containers, stream logs and execkubelet, over gRPCSocket /run/containerd/containerd.sock or /var/run/crio/crio.sock
CNI (Container Network Interface)Attach a network interface and IP to a new Pod sandbox, remove it on deletionContainer runtime, by running a binaryConfig /etc/cni/net.d/, binaries /opt/cni/bin/
CSI (Container Storage Interface)Create, attach, mount, resize and snapshot volumeskubelet (node side) and sidecar controllers (cluster side), over gRPCNode plugin socket under /var/lib/kubelet/plugins/

All three are out-of-tree: the vendor ships the plugin, Kubernetes ships only the interface. That's why swapping the runtime, network or storage doesn't need a new Kubernetes build.

CRI: the container runtime

  • The kubelet's containerRuntimeEndpoint (in /var/lib/kubelet/config.yaml) points at the runtime socket. kubeadm detects it at init and join, or takes --cri-socket.
  • crictl speaks CRI directly, so it works with any runtime. It's your tool when the API server is down.
sudo crictl ps -a                 # containers, including exited ones
sudo crictl pods                  # Pod sandboxes
sudo crictl logs <container-id>
sudo crictl images
cat /etc/crictl.yaml              # runtime-endpoint: unix:///run/containerd/containerd.sock
  • A RuntimeClass picks a different OCI handler configured in the runtime (for example gVisor or Kata for sandboxed workloads). Pods opt in with spec.runtimeClassName.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: sandboxed
handler: runsc        # must match a handler name in the containerd config

Legacy: use containerd or CRI-O through the CRI instead

The kubelet's built-in Docker support (dockershim) was removed in v1.24. docker ps on a node shows nothing for Kubernetes Pods on a containerd node. Use crictl. If Docker Engine must stay, it needs the external cri-dockerd adapter.

CNI: Pod networking

  • When a Pod sandbox starts, the runtime reads the lowest-sorted config file in /etc/cni/net.d/ and runs the plugin binaries from /opt/cni/bin/ it names. The plugin creates the interface and allocates the IP.
  • Most plugins run a DaemonSet that writes that config and the binaries onto each node, plus an agent that programs routes, tunnels or eBPF.
  • Until a config exists, the node reports NetworkReady=false and stays NotReady.
PluginData pathNetworkPolicy
CalicoBGP routing or VXLAN/IP-in-IP overlay, optional eBPFYes, plus its own extended policies
CiliumeBPF; can replace kube-proxyYes, plus L7 policies
FlannelVXLAN overlayNot in a default install; needs an add-on policy engine
Cloud plugins (AWS VPC CNI, Azure CNI)Pods get addresses from the cloud networkVaries, often with an add-on
ls /etc/cni/net.d/                         # which plugin wrote config
k get pods -n kube-system -o wide | grep -Ei 'calico|cilium|flannel'
k get ds -A                                # the CNI agent DaemonSet

NetworkPolicies that do nothing

Kubernetes stores NetworkPolicy objects whatever the CNI, but enforcement is entirely the plugin's job. On a plain Flannel cluster, a default-deny policy is accepted and silently ignored. If policies "don't work", check the plugin before rewriting the YAML.

Two CNI configs on one node

If a second plugin's file sorts before yours in /etc/cni/net.d/ (for example a leftover 10-flannel.conflist ahead of 20-cilium.conflist), new Pods get the wrong network. Remove leftovers after switching plugins.

CSI: storage drivers

  • Controller plugin (a Deployment or StatefulSet) with sidecars: external-provisioner creates volumes for PVCs, external-attacher handles VolumeAttachments, external-resizer and external-snapshotter do expansion and snapshots.
  • Node plugin (a DaemonSet) with node-driver-registrar, which registers the driver's socket with the kubelet. The kubelet calls it to stage and mount volumes into Pods.
  • A StorageClass's provisioner field is the CSI driver name, for example ebs.csi.aws.com or a name your driver publishes.
k get csidrivers                   # installed drivers and their capabilities
k get csinodes                     # which drivers registered on which node
k get volumeattachments            # attach state per PV and node
k get sc                           # provisioner column = CSI driver name

Exam signal

"A PVC stays Pending with a StorageClass set" often means the CSI driver named in provisioner isn't installed or its controller isn't running. k get csidrivers and the controller Pod's logs answer it fast. "Pod stuck ContainerCreating with a mount error" points to the node plugin on that node.

Legacy: use CSI drivers instead

In-tree volume plugins (awsElasticBlockStore, gcePersistentDisk, azureDisk, cinder and others) have been migrated to CSI and removed from the kubelet. Old manifests using them are redirected to the CSI driver or fail if it isn't installed. Write new StorageClasses against the CSI driver name.

Other extension points

MechanismExtends
Device pluginsAdvertise hardware such as GPUs as extended resources (nvidia.com/gpu)
Dynamic Resource AllocationClaims for devices with parameters, through ResourceClaims and DeviceClasses
CustomResourceDefinitions and API aggregationThe API itself (see CRDs and operators)
Admission webhooksValidation and mutation of API requests

Scenarios

Scenario
The API server on a control plane node is down and kubectl times out. You need to see whether the kube-apiserver container is crash-looping and read its last error. Which tool works?
Scenario
A new cluster passes every connectivity test, but a default-deny ingress NetworkPolicy in namespace pay has no effect: Pods in other namespaces still reach pay's Pods. `ls /etc/cni/net.d` shows only 10-flannel.conflist. What's the cause?

Drill

kubectl config use-context cka-ext

Answer these about the cluster and write each answer on its own line in /opt/answers/plugins.txt: (1) the CRI socket path the kubelet on node ext-w1 uses, (2) the CNI plugin installed, (3) the provisioner of the default StorageClass.

Further reading

On this page