Parse a web server log into rows and count its failures

short · 40 min · Objective 1.2

Task

Turn the practice web server log into a table with one column per field, then count requests by status code. A log is only data once it is parsed.

Steps

  1. Read data/web.log line by line. Each line is in Common Log Format: client address, two dashes, a bracketed timestamp, the quoted request, the status code and the response size.
  2. Write lab/logs/requests.csv with header ip,timestamp,method,path,status,bytes, one row per log line, using Python's csv module so fields are quoted correctly.
  3. Count how many lines your parser could not read and record it in lab/logs/failures.txt as unparsed: N.
  4. Count requests by status and save them to lab/logs/status.csv with header status,requests.

Verify

These checks run in a POSIX shell with Python 3: Terminal on macOS or Linux, and on Windows Git Bash (it comes with Git for Windows) or WSL. On Windows, python3 is often a Microsoft Store placeholder that prints “Python was not found”; run alias python3=py in Git Bash first.

python3 -c "import csv; print(len(list(csv.DictReader(open('lab/logs/requests.csv')))), 'parsed requests')"
grep -E '^unparsed: ' lab/logs/failures.txt
sort -t, -k1,1 lab/logs/status.csv

5000 parsed requests and 0 unparsed lines, with 4494 responses of 200, 211 of 304, 198 of 404 and 97 of 500. The 500s are the server's own failures, a little under 2% of requests -- a figure you could not have read off the raw log by eye.

This is an independent study companion for CompTIA Data+ DA0-002 and is not produced by or endorsed by CompTIA.