Diagnose a service you cannot reach

short · 30 min · Objective 1.4

Before you start — what this lab needs

A disposable virtual machine or container with a web server:

podman run -d --name lab-net --rm -p 8080:80 nginx:1.25

Work from the host, against the container.

Task

Work the layers in order against a service that is deliberately unreachable, and identify at which step the failure appears. Do it three times with three different causes, so that the sequence rather than the guess becomes the habit.

Steps

Confirm the healthy case first. Work up the layers: ip link, ip addr, ip route, then nc -zv localhost 8080 and curl -sf -o /dev/null -w '%{http_code}' http://localhost:8080/.
Cause one: stop the container. Repeat the sequence and note that the failure appears at nc, with connection refused, and that it is fast.
Cause two: restart it bound to loopback only, with -p 127.0.0.1:8080:80. From another machine on the network the port is unreachable while ss -tlnp on the host shows it listening. Confirm the bind address is what ss reports.
Cause three: run it on the normal port and add a firewall rule dropping inbound traffic to it. Note the failure is now a TIMEOUT rather than a refusal, and that the difference tells you a firewall is dropping silently.
Capture the difference: run tcpdump -n -i any port 8080 in another terminal during a refused connection and during a dropped one, and compare.
Use mtr to a public host and read the per-hop columns, identifying which hops rate-limit ICMP.

Verify

ss -tlnp | grep ':8080'                     # bind address: 0.0.0.0 vs 127.0.0.1
nc -zv localhost 8080 2>&1 | tail -1
timeout 5 nc -zv <host-ip> 8080 2>&1 | tail -1   # refused = fast, dropped = timeout
curl -sf -o /dev/null -w '%{http_code}\n' http://localhost:8080/

The distinction to be able to state at the end: refused is fast because something answered; timed out is slow because the packet vanished. That single observation separates "the service is down" from "something is dropping".

Notes

ss -tlnp showing 127.0.0.1 is the most under-diagnosed cause of "the firewall must be wrong". The firewall is fine; the service was never listening anywhere the client could reach.