Build a log report with no scripting language

applied · 40 min · Objective 1.5

Task

Produce a short daily report from a web server log using only the standard text tools, then make it runnable from cron. The constraint -- no Python, no installing anything -- is realistic: on a hardened host it may be all you have, and the pipelines are worth having in your hands.

Steps

  1. Count total requests, and count them per hour by extracting the hour field.
  2. Produce the top ten client addresses by request count.
  3. Produce the count of each status code, ordered by frequency.
  4. Sum the bytes transferred. The shell only does integer arithmetic, so decide whether to use awk's accumulator or pipe to bc.
  5. Extract the requests that returned 500 and list the paths involved, with counts, so you can see whether the errors cluster on one path.
  6. Assemble the above into a script that prints a titled report, and make it safe to run from cron: absolute paths throughout, and a PATH set at the top.
  7. Add it to a crontab entry that runs at 06:00 daily and mails nothing on success.

Verify

cd ~/lab-report
awk 'END{print NR}' access.log                                   # 2000
awk '{print $NF}' access.log | awk '{s+=$1} END{print s}'         # byte total
awk '$(NF-1) == 500' access.log | wc -l                           # 117
./report.sh | head -20
PATH=/usr/bin:/bin ./report.sh >/dev/null && echo "cron-safe"

The last line is the one that matters: running the script with cron's minimal PATH must still succeed. If it fails, you have a bare command somewhere that resolves only because your interactive PATH is richer -- which is exactly the failure that makes a script work for you and not at 06:00.

Notes

The 500 count in the third command uses the status field position, not a substring search. grep -c 500 gives a different and wrong answer here, because paths and byte counts contain 500 too.