Make logs survive an attacker with root

applied · 40 min · Objective 3.1

Before you start — what this lab needs

Two disposable virtual machines or containers: an origin that generates logs and a collector that receives them. Both must be disposable, and you will edit logs as root deliberately.

# collector and origin, each with rsyslog installed

Task

Forward authentication logs to a second host, then demonstrate the property that makes it worth doing: a root user on the origin can rewrite the local logs but cannot touch what has already left. Add log rotation that does not silently lose entries.

Steps

On the collector, enable rsyslog reception over TCP (the imtcp module and a listener) and confirm it is listening with ss -tlnp | grep 514.
On the origin, forward auth events to the collector by adding authpriv.* @@collector:514 and restarting rsyslog.
Generate authentication events on the origin -- a few failed logins with a bad password -- and confirm they arrive on the collector.
Now play the attacker: on the origin, as root, truncate the local auth log with : > /var/log/secure. Confirm the local record is gone.
Check the collector. The forwarded entries are still there. State plainly what this buys you and what it does not.
Configure logrotate for a busy application log with a postrotate reload, then demonstrate the failure mode without it: rotate a log while a process holds it open and show the process keeps writing to the rotated file until signalled.
Fix it with copytruncate and explain the small window of loss that option accepts in exchange for not needing to signal the process.

Verify

# on the collector
grep -c 'authentication failure' /var/log/from-origin.log     # non-zero
# on the origin, after truncating
wc -l < /var/log/secure                                        # small or zero
# the forwarded copy is unaffected -- compare the two counts
lsof -p "$(pgrep -f theapp)" | grep -c '(deleted)'             # 0 after copytruncate

The two counts telling different stories is the whole point: the origin's local record was destroyed and the collector's copy was not. That gap is where forensics lives after a compromise.

Notes

Forwarding does not make the logs tamper-proof -- an attacker who reaches the collector can still edit them there. It raises the cost: they now have to compromise two machines instead of one, and the second is one whose only job is to be hard to reach.