Answer a question with a pipeline

short · 25 min · Objective 1.5

Task

Given a web server access log, answer four questions using only the standard text tools -- no scripting language, no installing anything. The questions are the ones you are actually asked during an incident, and the pipelines that answer them are worth knowing by heart.

Steps

  1. Count the total number of requests.
  2. Produce a ranked list of client addresses by request count, most frequent first. Use the field extraction, sort, uniq -c, sort -rn idiom, and be able to say why the first sort is required.
  3. Count how many requests returned a 500 status, without counting a page whose name happens to contain 500.
  4. List the distinct status codes present, with a count for each.
  5. Show only the lines that are not 200 responses, using an inverted match.
  6. Extract the requested paths and find the three most requested.

Verify

cd ~/lab-tools
wc -l < access.log                                     # 400
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3
awk '$NF == 500' access.log | wc -l                    # 30
awk '{print $NF}' access.log | sort | uniq -c | sort -rn

The ranked address list must total 400 when its counts are summed, and the 500 count must be 30. If your 500 count is higher, you matched the string anywhere in the line rather than the status field -- which is exactly the class of error that makes a log analysis quietly wrong.

Notes

Step 3 is the point of the lab. grep -c 500 access.log gives a different and wrong answer, because it matches /page500 and any byte count containing 500. Matching a field, not a substring, is the difference between an answer and a number.