Run a Kubernetes deployment and break its service

short · 35 min · Objective 4.1

Task

Deploy an application to a local Kubernetes cluster, confirm the control loop replaces a killed pod, and reproduce the commonest Service failure -- a selector that matches no pods -- then diagnose it the way you would in production.

Steps

  1. Apply a Deployment with three replicas of a simple web image, and confirm kubectl get pods shows three running.
  2. Delete one pod with kubectl delete pod <name> and watch the control loop start a replacement within seconds. State why a bare Pod would not have come back.
  3. Add a Service selecting the deployment's pods by label, and confirm kubectl get endpoints lists three endpoints. Reach the app through it.
  4. Break it: change the Service's selector to a label no pod has. Confirm the Service still accepts connections but answers nothing, and that kubectl get endpoints now shows <none>.
  5. Diagnose from the symptom: kubectl describe service and kubectl get endpoints together point at the label mismatch. Fix the selector and confirm endpoints return.
  6. Add a ConfigMap for non-secret settings and a Secret for a password, mount both, and confirm the Secret is only base64-encoded -- decode it with kubectl get secret ... -o jsonpath and base64 -d.
  7. Roll out an image change and use kubectl rollout status and kubectl rollout undo to advance and revert it.

Verify

kubectl get deploy web -o jsonpath='{.status.readyReplicas}'    # 3
kubectl delete pod -l app=web --field-selector status.phase=Running | head -1
sleep 5; kubectl get pods -l app=web --no-headers | grep -c Running   # back to 3
kubectl get endpoints web -o jsonpath='{.subsets}' ; echo   # <none> when broken
kubectl get secret web -o jsonpath='{.data.password}' | base64 -d; echo   # plaintext

The endpoints going to <none> when the selector is wrong is the diagnosis to carry: a Service with no matching pods accepts connections and answers nothing, and kubectl get endpoints is the one command that says so. The base64 decode proves a Secret is encoding, not encryption.

Notes

A bare Pod that dies stays dead; a Deployment maintains the replica count. That is the single most useful fact about Pods, and step 2 is where it becomes concrete -- the killed pod is replaced because the control loop is comparing desired state to actual and closing the gap.