Find why one host can reach a service and another cannot

applied · 45 min · Objective 5.3

Task

The hardest networking faults are the ones that work from one place and not another -- because "the network is down" is easy and "it works from the app server but not the load balancer" is where careers are spent. Reproduce this class and isolate the layer, using two hosts and a service between them.

Steps

  1. Establish the working baseline: from the client, curl the service and get a
  2. Capture ss -tlnp on the server showing the listener.
  3. Plant fault A -- firewall drop. Add a rule dropping the port on the server. From the client, distinguish the signature: a DROP gives a hang then timeout, a REJECT gives an immediate "Connection refused". Confirm with curl timing.
  4. Prove the layer without guessing: ping still works (ICMP allowed), the TCP handshake does not. nc -vz server 8080 times out while ping succeeds -> transport-layer block, not routing.
  5. Plant fault B -- the listener bound to 127.0.0.1. Now ss on the server shows the port open, curl localhost on the server works, and the client gets "Connection refused" instantly. Same symptom as REJECT, different cause.
  6. Separate B from the firewall: check the bind address in ss -tlnp (127.0.0.1:8080 vs 0.0.0.0:8080). The firewall was never involved.
  7. Plant fault C -- an MTU mismatch or a wrong route on one host only, so small requests work and large responses hang. Use ping -M do -s 1472 to find it.
  8. For each fault, write the single observation that a colleague could act on.

Verify

# from the client, classify by timing and message:
timeout 5 curl -s -o /dev/null -w '%{http_code}\n' http://server:8080 || echo "no answer"
nc -vz server 8080            # transport reachability, independent of HTTP
ss -tlnp | grep :8080         # run on the SERVER: bind address decides remote reach

The bind-address line settles the commonest false alarm: a service "open" in ss but bound to loopback is unreachable from every other host regardless of firewall state. Refused-instantly versus hang-then-timeout is the other decisive tell, and it separates a REJECT/closed port from a silent DROP.

Notes

The through-line is that identical symptoms have different causes at different layers, and only a layered check tells them apart. "Connection refused" is a reached host actively saying no (nothing listening, or a REJECT); a timeout is a packet that vanished (a DROP, a black-hole route, an MTU hole). Reading the failure mode is faster than reading logs.