Run a Kubernetes deployment and break its service

short · 35 min · Objective 4.1

Task

Deploy an application to a local Kubernetes cluster, confirm the control loop replaces a killed pod, and reproduce the commonest Service failure -- a selector that matches no pods -- then diagnose it the way you would in production.

Steps

  1. Apply a Deployment named web with three replicas of a simple web image, its pods labelled app: web -- the names the Verify uses -- and confirm kubectl get pods shows three running.
  2. Delete one pod with kubectl delete pod <name> and watch the control loop start a replacement within seconds. State why a bare Pod would not have come back.
  3. Add a Service named web selecting the deployment's pods by label, and confirm kubectl get endpoints lists three endpoints. Reach the app through it.
  4. Break it: change the Service's selector to a label no pod has. Confirm the Service object still looks healthy in kubectl get service web, that connections to it now fail -- refused at once under kube-proxy's default iptables mode, which rejects traffic to a Service with no endpoints -- and that kubectl get endpoints now shows <none>.
  5. Diagnose from the symptom: kubectl describe service and kubectl get endpoints together point at the label mismatch. Fix the selector and confirm endpoints return.
  6. Add a ConfigMap for non-secret settings and a Secret named web holding a key password (kubectl create secret generic web --from-literal=password=lab-only), mount both, and confirm the Secret is only base64-encoded -- decode it with kubectl get secret ... -o jsonpath and base64 -d.
  7. Roll out an image change and use kubectl rollout status and kubectl rollout undo to advance and revert it.

Verify

kubectl get deploy web -o jsonpath='{.status.readyReplicas}'    # 3
kubectl delete "$(kubectl get pod -l app=web -o name | head -1)"   # one pod, as in step 2
kubectl wait --for=condition=Ready pod -l app=web --timeout=60s >/dev/null
kubectl get pods -l app=web --no-headers | grep -c Running        # back to 3
kubectl get endpoints web -o jsonpath='{.subsets}' ; echo   # <none> when broken
kubectl get secret web -o jsonpath='{.data.password}' | base64 -d; echo   # plaintext

The endpoints going to <none> when the selector is wrong is the diagnosis to carry: a Service with no matching pods looks healthy and refuses every connection, and kubectl get endpoints is the one command that says why. (Deleting with -l app=web would take all three pods at once, not the one step 2 describes; kubectl wait replaces a fixed sleep that could count a replacement still starting.) The base64 decode proves a Secret is encoding, not encryption.

Notes

A bare Pod that dies stays dead; a Deployment maintains the replica count. That is the single most useful fact about Pods, and step 2 is where it becomes concrete -- the killed pod is replaced because the control loop is comparing desired state to actual and closing the gap.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.