The troubleshooting methodology
Listen to this lesson
This episode is a study companion for CompTIA Linux+ XK0-006 and is not produced by or endorsed by CompTIA.
Not a numbered objective, and assumed by four of them. Domain 5 is 22% of the exam, and objectives 5.2, 5.3, 5.4 and 5.5 each begin "Given a scenario, analyze and troubleshoot…" — which assumes a method without teaching one. CompTIA's own courseware devotes a chapter to it, the six-step model appears across A+, Network+ and Security+, and the exam's scenario questions are written against it. It is here because the four lessons after this one are much easier with it and because the habits below are what separate a fix from a guess.
Why this matters
Under pressure, with a system down and people asking, the temptation is to start changing things. It feels like progress. It is how a one-hour outage becomes a four-hour one, because after ninety minutes nobody can say what the system's configuration actually is any more.
A method is not bureaucracy. It is the thing that stops you from making the problem worse, and it is what lets you hand over cleanly at the end of a shift.
The lesson
The six steps
CompTIA's model, which the scenario questions are written against:
1. Identify the problem. Gather information. What exactly is failing, for whom, since when? What changed? Ask the user what they were doing, and check the logs, the monitoring and the change record.
The most valuable question is "what changed?" — because the overwhelming majority of failures follow a change: a deployment, a package update, a certificate expiry, a configuration edit, a full disk that has been filling for weeks. git log on your configuration repository, dnf history, and the deployment log are the first three places to look.
Be precise about the symptom. "The website is down" might be DNS, the load balancer, the application, the database, or one user's browser cache. Narrow it before theorising: can you reach it? From another network? Is it slow or refused? Every user or one?
2. Establish a theory of probable cause. A specific, falsifiable statement. Not "something is wrong with the network" but "the application cannot resolve db01 because the DNS server changed".
Question the obvious. It is on the objectives-adjacent list of habits for a good reason: the cable, the power, the typo, the expired certificate, the full disk, the service that is not running. Experienced people skip these because they feel beneath them, and then spend two hours on a sophisticated theory before discovering the service was stopped.
Start with the simple and likely, and work toward the complex and rare. If several theories fit, take the cheapest to test first, not the most interesting.
3. Test the theory. Confirm or eliminate it, and prefer a test that does not change anything:
systemctl status nginx # is it even running?
ss -tlnp | grep :443 # is it listening?
curl -sv https://localhost/ # does it answer locally?
dig db01.example.com # does the name resolve?
journalctl -u nginx --since "1 hour ago"
df -h && df -i # disk space AND inodes
If the theory is confirmed, go to step 4. If it is not, form a new theory — do not tweak the old one until it fits. And if you are out of theories, that is the point at which escalation is the correct move rather than a defeat.
4. Establish a plan of action. Decide what you will do before you do it, and include how you will undo it. For anything non-trivial that means: what is the change, what is the blast radius, what is the rollback, does it need a maintenance window, and who needs to know.
Back up whatever you are about to modify. cp sshd_config sshd_config.bak costs one second.
5. Implement the solution or escalate. Make the change — and change one thing at a time. Two simultaneous changes and a fixed system tells you nothing about which one fixed it; two changes and a worse system leaves you unable to back out cleanly.
This step includes escalation as a legitimate outcome, which is the part people read past.
6. Verify full functionality, and implement preventive measures. Confirm the actual user-visible symptom is gone, not just that your command returned zero. Then ask what stops it recurring: a monitor on the thing that filled, log rotation, a larger volume, a certificate renewal that is automated, a unit file with Restart=on-failure.
Then document findings and outcomes. This is the step that gets dropped under time pressure and is the one with the longest-lived value. The next person to see these symptoms — very possibly you, in eight months — needs the symptom, the cause, the fix, and the evidence that connected them. A ticket saying "fixed" is worthless; one saying "disk full because logrotate was not running after the 9.3 upgrade; re-enabled the timer; added a monitor at 85%" saves an hour next time.
Reproduce before you fix
Reproduce before you fix is the habit that prevents the most wasted work.
If you cannot make the problem happen on demand, you cannot know that you have fixed it. You will make a change, the symptom will not appear, and you will declare victory — and then it recurs on Thursday, because the symptom was intermittent and your change was irrelevant.
Reproduction also tells you the boundary of the problem, which is most of the diagnosis. Does it happen for every user or one? Every request or one in fifty? On both servers or one? With a fresh session or only an old one? Each answer eliminates a category of cause.
Where you genuinely cannot reproduce — a rare race, a failure only under production load — say so explicitly, and treat the fix as a hypothesis under observation rather than as done.
Change one thing at a time
Worth its own section because it is violated constantly, and always for the same reason: under pressure, three plausible fixes are faster to apply together than in sequence.
The costs are concrete. You do not learn which change mattered, so you cannot document a cause. You cannot revert cleanly, because you no longer know which of three to revert. And if the system gets worse, you have three suspects instead of one.
Make one change, verify, then decide. If a change does not help, undo it before trying the next one. Otherwise, an hour later, the configuration is an accumulation of failed hypotheses that nobody has a record of — and that accumulated state becomes the next incident.
Knowing when to escalate
Knowing when to escalate is a skill and not an admission. The signals:
- You are out of theories, or your last three have been wrong.
- The fix requires access or authority you do not have — a database migration, a firewall change, a vendor's support contract.
- The blast radius exceeds your authority. Restarting a production database is not a decision to make alone at 2 a.m.
- You have hit a time box. Set one at the start — "thirty minutes, then I bring in the network team" — and hold yourself to it. Without a time box, sunk cost keeps you going long past the point where a second pair of eyes would have solved it.
- The problem is outside your domain — a hardware fault, a provider outage, a bug that needs the vendor.
- Escalate immediately, without any of that, for a suspected security incident. Do not investigate a possible compromise alone; do not destroy evidence by "cleaning up". That is a different process with different rules, covered in the security troubleshooting lesson.
Escalating well means handing over what you know: the symptom, the timeline, what you have tested, what you have ruled out, what you changed. An escalation that arrives as "it's broken, can you look" wastes the responder's first twenty minutes repeating your work.
A worked example
The shape of it, end to end:
Identify. Users report the internal wiki is slow since roughly 09:00. Only the wiki; other services on the same host are fine. Confirmed by reproducing: page loads take 30 seconds. What changed?
dnf historyshows a security update at 03:40. The deployment log shows nothing else. Theory. The updated package restarted a service with different settings. Test.systemctl statusshows the service running.journalctlsince 03:40 shows repeated "cannot write to /var/lib/wiki/cache".df -hshows/varat 100%. Theory wrong, but the test found the real cause — which is why you run tests that gather information rather than tests that only confirm. New theory./varis full, so cache writes fail and every request falls back to a slow path.du -sh /var/* | sort -hshows/var/log/appat 40 GB. Plan. Compress and archive the old logs, confirm space, then fix rotation. Rollback: nothing destructive is being done. Implement. One change: archive logs older than 7 days. Space returns, page loads drop to under a second. Verify. Reproduce the original symptom — it is gone, from a user's browser, not just fromcurlon the host. Prevent.logrotatefor that directory was never configured; add it. Add a monitor at 85% on/var, because the disk had been filling for weeks and nothing said so. Document. Symptom, cause, fix, evidence, and the two preventive changes.
The lesson inside the example: the theory was wrong and the process still worked, because the test was chosen to produce information rather than to confirm a belief.
What to take away
- The six steps: identify the problem; establish a theory of probable cause; test the theory; establish a plan of action; implement the solution or escalate; verify full functionality — then implement preventive measures and document findings and outcomes.
- "What changed?" is the highest-yield question, because most failures follow a change.
- Question the obvious first — power, cable, service running, disk full, certificate expired, typo.
- Reproduce before you fix, or you cannot know the fix worked.
- Change one thing at a time, and undo a change that did not help before trying the next.
- Prefer tests that gather information over tests that only confirm your theory. A wrong theory with a good test still moves you forward.
- Escalate when you are out of theories, lack the authority, or hit your time box — and immediately for a suspected security incident.
- Document: symptom, cause, fix, evidence. The step most often skipped and the one with the longest life.
Practise what you just read
1. Which question yields the most in the first minutes of an incident?
Select one
Show answer
D. The overwhelming majority of failures follow a change: a deployment, a package update, a certificate expiry, a configuration edit, or a disk that has been filling for weeks. git log on the configuration repository, dnf history and the deployment log are the first three places to look, and they frequently end the investigation.
2. Three plausible fixes are available and the system is down. Why apply them one at a time?
Select one
Show answer
C. Applying them together saves minutes and costs you the cause, a clean rollback, and a documented fix. Undo a change that does not help before trying the next, or an hour later the configuration is an accumulation of failed hypotheses nobody has recorded -- and that accumulated state becomes the next incident.
3. When should a suspected security compromise be escalated?
Select one
Show answer
D. A suspected compromise is a different process with different rules. Investigating alone risks destroying evidence -- "cleaning up" a suspicious file removes the thing an incident responder needed -- and restoring service first may reinstall the attacker's access. Escalate immediately and preserve the system's state.
5 more questions on this objective are part of the full course.
Hands-on labs
Part of the free CompTIA Linux+ XK0-006 course — 48 lessons and 82 hands-on labs.