CI/CD and container orchestration

Why this matters

The previous lesson described the code. This one is about what runs it and what runs on top of it: the pipeline that applies a change, and the orchestrator that keeps the result alive.

Both exist for the same reason. A person applying changes by hand is slow, inconsistent and unavailable at two in the morning. A pipeline applies the same steps every time and leaves a record; an orchestrator restarts a dead container before anyone has read the alert.

Kubernetes in particular has a reputation for depth that is entirely deserved, and the exam is not asking for that depth. It is asking whether you know what a pod is, what a service does, and where secrets and ConfigMaps fit — which is roughly the amount you need to be useful on a team that runs it.

The lesson

Pipelines

A pipeline is an automated sequence triggered by an event, almost always a push to a repository. The stages are conventional:

commit -> lint -> test -> build -> scan -> deploy to staging -> deploy to production

Any stage failing stops the pipeline, and nothing downstream runs. That is the whole value: a broken change cannot reach production because the machinery refuses to carry it there.

The tools are GitLab CI, GitHub Actions, Jenkins, Argo CD and Tekton, and their configuration is a file in the repository — which makes the pipeline itself infrastructure as code:

# .gitlab-ci.yml
stages: [test, build, deploy]

variables:
  IMAGE: registry.example.com/app

test:
  stage: test
  script:
    - ansible-lint playbooks/
    - python -m pytest

build:
  stage: build
  script:
    - podman build -t "$IMAGE:$CI_COMMIT_SHORT_SHA" .
    - podman push "$IMAGE:$CI_COMMIT_SHORT_SHA"

deploy:
  stage: deploy
  script:
    - ansible-playbook site.yml
  only:
    - main

Note only: main — the test and build stages run on every branch, and only a merge to main deploys. That single line is most of a deployment policy.

Variables are how a pipeline is parameterised without hard-coding anything: the image name, the target environment, the version. Every CI system provides built-in ones — CI_COMMIT_SHORT_SHA above tags the image with the exact commit that produced it, so a running container can always be traced back to its source. Secrets are supplied the same way, but marked protected and masked so they never appear in the log. A credential written into the pipeline file is a credential in the repository, which is to say disclosed.

Continuous integration is the first half: every change is merged and tested frequently, so integration problems surface in hours instead of at the end of a project. Continuous delivery means every passing build is deployable; continuous deployment means it is actually deployed, automatically. The distinction between the last two is one manual approval step, and which one you want is a judgement about blast radius rather than about tooling.

Shift left testing

Shift left testing means moving checks earlier — leftward along that pipeline diagram, and further still, onto the developer's machine.

The argument is cost. A configuration error caught by a pre-commit hook costs seconds. The same error caught in CI costs a pipeline run. In staging, it costs a deployment and someone's investigation. In production, it costs an outage, an incident review, and possibly a customer. The defect is identical; only the distance from where it was made has changed.

In practice, shifting left looks like:

# a pre-commit hook, running before the commit is even made
ansible-lint playbooks/
ruff check scripts/
tofu validate
gitleaks detect --no-banner        # a secret must never reach the history
shellcheck deploy.sh

gitleaks in that list is the clearest case. A key caught by a pre-commit hook was never committed. A key caught in CI is already in the history, already in every clone, and must be rotated — the pipeline can only tell you that you have a problem, not prevent it.

DevSecOps

DevSecOps is the same argument applied to security specifically: security review as a stage in the pipeline rather than a gate at the end.

The traditional model was a security review before release, which was adversarial, late, and routinely skipped under schedule pressure. DevSecOps builds the checks into the pipeline so they run on every change and block it automatically:

security:
  stage: test
  script:
    - trivy image --severity HIGH,CRITICAL "$IMAGE:$CI_COMMIT_SHORT_SHA"
    - trivy fs --scanners vuln,secret .
    - oscap xccdf eval --profile cis /usr/share/xml/scap/ssg-rhel9-ds.xml

Container image scanning, dependency scanning, secret scanning, static analysis, and compliance checks against a benchmark — each a stage, each able to fail the build. The cultural claim behind the name is that security is everyone's job rather than a department's; the mechanical claim is that a check which runs automatically on every change is worth more than a thorough review that happens twice a year.

GitOps

GitOps takes the idea one step further: the git repository is the single source of truth for what should be running, and an agent inside the cluster continuously reconciles reality to it.

The inversion is the important part. In a conventional pipeline, CI pushes to the cluster and therefore needs credentials for it. In GitOps, an agent — Argo CD or Flux — pulls from the repository:

conventional:  CI  --push credentials-->  cluster
GitOps:        CI --> repo  <--pull--  agent inside the cluster

What that buys:

• No cluster credentials in CI. The blast radius of a compromised pipeline drops enormously.
• Drift correction. Someone runs kubectl edit by hand at 3 a.m.; the agent notices the divergence from the repository and reverts it.
• Rollback is git revert. The previous state is a commit.
• The audit trail is the history. Who changed production, when, approved by whom — it is the pull request.

The discipline it demands is that manual changes do not survive, which is the point and is also a surprise the first time an emergency fix disappears under you. If it is not in the repository, it is not real.

State

State is what a system has actually got, as opposed to what you declared it should have, and it is the concept underneath everything in this lesson.

OpenTofu keeps a state file mapping the resources in your code to real objects at the provider — this aws_instance.web is that i-0abc123. Without it, a second apply could not tell "already exists" from "needs creating", and would build everything again.

Three consequences that matter operationally:

The state file contains secrets in plain text — generated passwords, private keys, connection strings — because it records resource attributes as they are. It must never be committed to git. Keep it in encrypted remote storage (an S3 bucket with encryption, or a state backend built for it).

Concurrent applies corrupt it, which is why remote backends provide locking. Two engineers applying at once without a lock is a genuinely bad day.

Losing it means OpenTofu no longer knows anything exists and will try to create duplicates of everything. tofu import can rebuild the mapping resource by resource, and it is tedious enough that "back up the state" is a rule rather than advice.

Kubernetes holds the equivalent in etcd, the cluster's key-value store, and the same rule applies: back it up, and treat it as the thing whose loss is unrecoverable.

APIs

An application programming interface is how all of this talks to anything else. Every tool in this lesson is ultimately an API client: OpenTofu's providers call cloud APIs, kubectl calls the Kubernetes API server, Ansible modules call whatever their target exposes.

Most are REST over HTTPS, exchanging JSON, and authenticated with a token:

curl -H "Authorization: Bearer $TOKEN" \
     -H "Accept: application/json" \
     https://api.example.com/v1/servers

curl -s https://api.example.com/v1/servers | jq '.items[].name'

kubectl get pods -o json | jq '.items[].metadata.name'

jq is the tool that makes an API usable from the shell, and it is worth learning alongside curl.

The HTTP verbs map onto operations predictably — GET reads, POST creates, PUT and PATCH update, DELETE removes — and the status codes are worth recognising: 200 success, 201 created, 401 not authenticated, 403 authenticated but not permitted, 404 not found, 429 rate limited, 500 the server's fault.

401 versus 403 is the useful distinction when something breaks: 401 means your token is wrong or expired, 403 means your token is fine and your permissions are not. They send you to entirely different places.

429 is worth respecting rather than retrying through. A script that hammers a rate-limited API gets slower, not faster, and may get the credential blocked.

Cloud-init

Cloud-init is the standard for configuring a machine at first boot. Rather than installing an operating system, a cloud instance launches a prepared image and cloud-init personalises it — hostname, users, SSH keys, packages, files, commands:

#cloud-config
hostname: web01
fqdn: web01.example.com

users:
  - name: admin
    groups: [wheel]
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nza... admin@workstation

package_update: true
packages:
  - nginx
  - chrony

write_files:
  - path: /etc/sysctl.d/99-tuning.conf
    content: |
      net.core.somaxconn = 4096

runcmd:
  - [systemctl, enable, --now, nginx]

The #cloud-config first line is required and is not a comment — without it the file is not recognised. Everything else is ordered: packages install before runcmd runs.

Two operational notes. Cloud-init runs once, on first boot, and marks itself done; re-running it needs cloud-init clean --logs and a reboot, which is a test-environment operation, not a production one. And when a new instance comes up without its configuration, the answer is in /var/log/cloud-init-output.log — which is the first place to look and is very often ignored in favour of guessing.

Cloud-init occupies the same slot as Kickstart from the installation lesson: Kickstart answers an installer, cloud-init configures an image that is already installed.

Kubernetes objects

Kubernetes is a control loop. You declare objects; controllers work continuously to make reality match. The objects on the objectives:

Pods are the smallest deployable unit — one or more containers that share a network namespace and can share volumes. Containers in a pod reach each other on localhost and are always scheduled onto the same node. Most pods hold one container; a second is usually a sidecar doing logging or proxying.

You do not create pods directly. A bare pod that dies stays dead. That is the single most useful thing to know about them.

Deployments are what you create instead. A Deployment declares how many replicas of a pod template should exist, and a controller maintains that number — replacing failures, and rolling out changes gradually:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.25
          ports:
            - containerPort: 80
          envFrom:
            - configMapRef:
                name: web-config
            - secretRef:
                name: web-secrets
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
      volumes:
        - name: content
          persistentVolumeClaim:
            claimName: web-content

A rolling update replaces pods a few at a time, so the application stays available throughout, and kubectl rollout undo returns to the previous version.

Services solve the problem that pods are ephemeral and their addresses change. A Service is a stable name and address in front of a changing set of pods, selected by label:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

Three types worth knowing. ClusterIP is the default — reachable only inside the cluster, which is right for a database. NodePort opens a port on every node. LoadBalancer asks the cloud provider for a real load balancer, and is how a service becomes reachable from the internet.

The selector is where this goes wrong in practice: a Service whose selector matches no pod's labels has no endpoints, accepts connections and answers nothing. kubectl get endpoints web showing <none> is the diagnosis, and it is almost always a label typo.

ConfigMaps hold non-sensitive configuration separately from the image, so the same image runs in staging and production with different settings:

kubectl create configmap web-config \
  --from-literal=LOG_LEVEL=info \
  --from-file=nginx.conf

Secrets hold sensitive values — passwords, tokens, TLS keys — and are mounted or injected the same way:

kubectl create secret generic web-secrets \
  --from-literal=DB_PASSWORD='...'
kubectl create secret tls web-tls --cert=tls.crt --key=tls.key

A Kubernetes Secret is base64-encoded, not encrypted. Anyone who can read the Secret object can read the value with base64 -d, and by default it sits in etcd unencrypted. Enable encryption at rest, restrict access with RBAC, and for anything serious use an external secrets manager. Treating base64 as protection is a genuine and common mistake, and it is exactly the kind of thing an exam question is built around.

Volumes provide storage, because a container's filesystem is disposable — the same argument as the container volumes lesson. emptyDir lasts as long as the pod; a PersistentVolumeClaim requests durable storage that survives the pod entirely, which is what a database needs.

kubectl get pods -o wide
kubectl describe pod web-7d4b8c9-x2k4p     # events -- the first diagnostic
kubectl logs -f web-7d4b8c9-x2k4p
kubectl exec -it web-7d4b8c9-x2k4p -- sh
kubectl apply -f deployment.yml
kubectl rollout status deployment/web
kubectl rollout undo deployment/web
kubectl get endpoints web

kubectl describe pod is where you look first. The events at the bottom say ImagePullBackOff (wrong image name or missing registry credentials), CrashLoopBackOff (the container starts and exits — check logs), or Pending with a scheduling message (no node has the resources). Those three cover most of what goes wrong.

Docker Swarm

Docker Swarm is the simpler alternative, built into Docker. It lost the orchestration argument to Kubernetes, but it is on the objectives and its vocabulary is worth knowing — partly because the concepts are the same with less machinery.

Nodes are the machines in the swarm: managers maintain cluster state and schedule work, workers run it.

docker swarm init
docker swarm join --token <token> manager:2377
docker node ls

A service in Swarm is the equivalent of a Kubernetes Deployment — the declaration of what should run and how many:

docker service create --name web --replicas 3 -p 80:80 nginx:1.25
docker service ls
docker service ps web
docker service logs web

Tasks are the individual container instances a service is made of. One task is one container on one node, and docker service ps web lists them — including failed ones and why, which makes it the Swarm equivalent of kubectl describe.

Scale changes the replica count:

docker service scale web=10
docker service update --image nginx:1.26 web    # rolling update

Networks in Swarm are overlay networks spanning every node, so a container on one machine reaches a service on another by name. Swarm also provides a routing mesh: a published port is reachable on every node, whichever one actually holds the container.

docker network create -d overlay appnet
docker service create --name api --network appnet myapi

That is the same overlay network type from the container networking lesson, and Swarm is where it is actually used on a single-vendor stack.

Docker/Podman Compose

Compose is not orchestration across machines — it runs a multi-container application on one host, and it is what most real work uses day to day.

The compose file describes the whole application:

# compose.yml
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s

  app:
    build: .
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      DB_HOST: db
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped

volumes:
  pgdata:

secrets:
  db_password:
    file: ./db_password.txt

docker compose up -d           # create and start everything
podman-compose up -d           # Podman's equivalent
docker compose down            # stop and REMOVE containers and networks
docker compose down -v         # ...and delete the volumes -- destroys data
docker compose ps
docker compose logs -f app     # follow one service
docker compose exec app sh
docker compose build --no-cache

Up and down are the two verbs. up -d creates the network, the volumes and every container, in dependency order. down removes the containers and the network but leaves the volumes, which is why your database survives a restart — and why down -v is the flag to be careful with, since it deletes them.

Logs from a compose application are aggregated: docker compose logs interleaves every service with a prefix, which is how you see that the app failed because the database was not ready. -f follows, and naming a service narrows it to one.

Two details that repay themselves. depends_on with condition: service_healthy waits for the health check rather than merely for the container to start — without it, "depends on" only means "start after", and an app that connects to a database immediately will still fail. And "127.0.0.1:8080:8080" binds to loopback rather than every interface, which is the exposure trap from the container networking lesson.

Compose is also the sensible place to start. A single-host application in a compose file is a real deployment, and if it later needs to scale across machines the file is a good description of what to build in Kubernetes.

On the exam

• A pipeline runs stages on a trigger and stops at the first failure; variables parameterise it, and secrets are injected, never committed.
• Shift left moves checks earlier because a defect costs more the further it travels. DevSecOps makes security a pipeline stage.
• GitOps: the repository is the source of truth and an in-cluster agent pulls it, so CI needs no cluster credentials and manual changes get reverted.
• State maps declared resources to real ones. OpenTofu's state file contains secrets in plain text, must not be committed, and needs locking.
• APIs are REST/JSON with tokens. 401 is authentication, 403 is authorisation; 429 means slow down.
• Cloud-init configures a cloud image on first boot; the file must begin #cloud-config, and it runs once.
• Pods are the smallest unit and are not created directly; Deployments maintain replicas and roll out updates; Services give a stable address, selected by label, with ClusterIP / NodePort / LoadBalancer.
• ConfigMaps for configuration, Secrets for sensitive values — and a Secret is base64, not encrypted. Volumes outlive the container; a PVC outlives the pod.
• kubectl describe pod first: ImagePullBackOff, CrashLoopBackOff, Pending.
• Docker Swarm: nodes are managers or workers, a service declares what runs, tasks are the individual containers, scale changes the count, overlay networks span nodes.
• Compose runs a multi-container app on one host. up creates, down removes containers and networks but keeps volumes — -v deletes them. logs aggregates every service.