Run a local model on data that cannot leave

applied · 40 min · Objective 4.5

Task

Stand up a local model that runs entirely on your own hardware, and use it for the administration tasks the lesson endorses -- explaining a config line, drafting a regex, reviewing a script -- on data that policy forbids sending to a public service. The point is that "we cannot send this anywhere" is not the end of the tools, it is the case a local model is for.

Steps

  1. Read the install script before running it -- a curl | sh pattern is exactly what the lesson warns against, so read it first, then run it deliberately.
  2. Pull a small model: ollama pull llama3.1 (or a smaller one if memory is tight). Confirm with ollama list.
  3. Take a genuinely sensitive input -- a real sshd_config with internal hostnames, say -- and ask the LOCAL model to explain a directive. Confirm, by watching the network, that nothing left the machine.
  4. Draft a regex with it for a real log format, then verify the output against known data as in the short lab -- the local model is a drafting tool, and its output needs checking exactly as a hosted one's would.
  5. Have it review a script for injection risks and unquoted variables. Treat the review as a second opinion, not an authority.
  6. Disconnect the network entirely and confirm the model still works -- the property that makes it usable in an air-gapped or regulated environment.
  7. Contrast, in writing, what you would and would not send to a public service for the same tasks, and why the local model changes that calculus.

Verify

ollama list | grep -q llama && echo "model available locally"
# prove nothing leaves: run a query with the network down
nmcli networking off 2>/dev/null || ip link set eth0 down
ollama run llama3.1 "explain the sshd directive MaxStartups 10:30:60" | head -5
nmcli networking on 2>/dev/null || ip link set eth0 up

The query succeeding with networking OFF is the whole demonstration: the sensitive input never had anywhere to go, because the model is on the machine. That is what makes a local model the answer for data a public service may not receive.

Notes

A local model is weaker than the largest hosted ones, and for explaining a config line, drafting a regex or reviewing a script it is entirely adequate. The trade is capability for control, and in a regulated or air-gapped environment control is not a preference -- it is the only lawful option.