CRDs and operators

Exam tasks: 1.8 (understand CRDs, install and configure operators)

The decision: what does a CRD add to the API, what does it take for a custom resource to actually do something, and how do you install, inspect and configure the operator that makes it happen?

CRD, custom resource, controllerCopy Anchor Link

Writing a CRDCopy Anchor Link

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.vault.lab.io          # must be <plural>.<group>
spec:
  group: vault.lab.io
  scope: Namespaced                   # or Cluster
  names:
    plural: backups
    singular: backup
    kind: Backup
    shortNames: ["bk"]
  versions:
  - name: v1
    served: true
    storage: true                     # exactly one version is the storage version
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required: ["schedule", "target"]
            properties:
              schedule:
                type: string
              target:
                type: string
              keepLast:
                type: integer
                minimum: 1
                default: 7
            x-kubernetes-validations:
            - rule: "self.target.startsWith('pvc/')"
              message: "target must look like pvc/NAME"
          status:
            type: object
            properties:
              lastRun:
                type: string
    subresources:
      status: {}                      # .status is written through /status only
    additionalPrinterColumns:
    - name: Schedule
      type: string
      jsonPath: .spec.schedule
    - name: Last-Run
      type: string
      jsonPath: .status.lastRun

• apiextensions.k8s.io/v1 requires a structural schema for every version. Unknown fields are pruned unless you mark a subtree x-kubernetes-preserve-unknown-fields: true.
• x-kubernetes-validations holds CEL rules the API server checks on create and update. Defaults (default:) are applied on read and write.
• Several versions can be served, but only one is storage. Conversion between versions uses None (fields must be compatible) or a conversion webhook.

Working with custom resourcesCopy Anchor Link

apiVersion: vault.lab.io/v1
kind: Backup
metadata:
  name: ledger-nightly
  namespace: ledger
spec:
  schedule: "0 2 * * *"
  target: pvc/ledger-data

k get crd | grep vault.lab.io
k api-resources --api-group=vault.lab.io      # kind, short names, namespaced or not
k explain backup.spec                          # schema-driven docs, same as built-in kinds
k apply -f ledger-nightly.yaml
k get bk -n ledger                             # printer columns show up here
k get backups.vault.lab.io -A -o yaml          # fully qualified name avoids clashes
k describe backup ledger-nightly -n ledger     # status, events from the controller

Installing an operatorCopy Anchor Link

An operator install always gives you three things to verify:

1. CRDs registered: k get crd | grep <group>.
2. RBAC: a ServiceAccount with a ClusterRole that lets it watch its CRs and manage what it creates.
3. The controller Deployment running: k get pods -n <operator-ns>, and k logs deploy/<name> -n <operator-ns> when CRs don't reconcile.

• Configure the operator itself through its Deployment arguments, a ConfigMap or Helm values (for example which namespaces it watches). Configure what it manages through the custom resources.
• Install CRDs before CRs. Applying both in one file can fail on the first run because the CRD isn't established yet; rerunning the apply, or k wait --for=condition=Established crd/<name>, fixes it.
• Helm installs CRDs from a chart's crds/ folder only on first install and never upgrades or deletes them. Upgrade those CRDs with kubectl apply as the operator's docs describe.

ScenariosCopy Anchor Link

DrillCopy Anchor Link

kubectl config use-context cka-crd

1. An operator's install manifest is at /opt/op/beacon-operator.yaml. Install it and wait until its CRD beacons.signal.lab.io is established and the operator Pod is running in namespace beacon-system.
2. Create a Beacon named north in namespace relay with spec.intervalSeconds set to 30 and spec.endpoint set to http://relay-api.relay:8080.
3. Write the CRD's scope (Namespaced or Cluster) and its storage version to /opt/answers/beacon.txt.

k apply -f /opt/op/beacon-operator.yaml
k wait --for=condition=Established crd/beacons.signal.lab.io --timeout=60s
k get pods -n beacon-system                       # Running
k explain beacon.spec                             # confirm field names
k api-resources --api-group=signal.lab.io         # shortnames, namespaced

k create ns relay --dry-run=client -o yaml | k apply -f -
cat <<'EOF' | k apply -f -
apiVersion: signal.lab.io/v1
kind: Beacon
metadata:
  name: north
  namespace: relay
spec:
  intervalSeconds: 30
  endpoint: http://relay-api.relay:8080
EOF

k get crd beacons.signal.lab.io \
  -o jsonpath='{.spec.scope}{"\n"}{range .spec.versions[?(@.storage==true)]}{.name}{"\n"}{end}' \
  > /opt/answers/beacon.txt
cat /opt/answers/beacon.txt

k get beacon north -n relay -o yaml           # status filled in by the operator after a few seconds
k logs -n beacon-system deploy/beacon-operator --tail=20

Further readingCopy Anchor Link

• Custom resources
• Extend the Kubernetes API with CustomResourceDefinitions
• Operator pattern