Build a log report with no scripting language

applied · 40 min · Objective 1.5

Task

Produce a short daily report from a web server log using only the standard text tools, then make it runnable from cron. The constraint -- no Python, no installing anything -- is realistic: on a hardened host it may be all you have, and the pipelines are worth having in your hands.

Steps

  1. Count total requests, and count them per hour by extracting the hour field.
  2. Produce the top ten client addresses by request count.
  3. Produce the count of each status code, ordered by frequency.
  4. Sum the bytes transferred. The shell only does integer arithmetic, so decide whether to use awk's accumulator or pipe to bc.
  5. Extract the requests that returned 500 and list the paths involved, with counts, so you can see whether the errors cluster on one path.
  6. Assemble the above into a script that prints a titled report, and make it safe to run from cron: absolute paths throughout, and a PATH set at the top.
  7. Add it to a crontab entry that runs at 06:00 daily and mails nothing on success.

Verify

cd ~/lab-report
awk 'END{print NR}' access.log                                   # 2000
awk '{print $NF}' access.log | awk '{s+=$1} END{print s}'         # byte total
awk '$(NF-1) == 500' access.log | wc -l                           # 54
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3 # 1000 10.0.1.1, 500 10.0.2.2, 250 10.0.0.3
awk '$(NF-1) == 500 {print $6}' access.log | sort | uniq -c | sort -rn | head -1   # 34 /p13
./report.sh | head -20
PATH=/usr/bin:/bin ./report.sh >/dev/null && echo "cron-safe"

The byte total is 8697000. Of the 54 errors, 34 are on /p13 -- that is the finding step 5 exists to surface, and the report should make it obvious.

The last line is the one that matters: running the script with cron's minimal PATH must still succeed. If it fails, you have a bare command somewhere that resolves only because your interactive PATH is richer -- which is exactly the failure that makes a script work for you and not at 06:00.

Notes

The 500 count in the third command uses the status field position, not a substring search. grep -c 500 answers 58 here, not 54, because four byte counts contain 500 too.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.