Find why one host can reach a service and another cannot

applied · 45 min · Objective 5.3

Task

The hardest networking faults are the ones that work from one place and not another -- because "the network is down" is easy and "it works from the app server but not the load balancer" is where careers are spent. Reproduce this class and isolate the layer, using two hosts and a service between them.

Steps

  1. Establish the working baseline: from the client, curl the service and get a
  2. Capture ss -tlnp on the server showing the listener.
  3. Plant fault A -- firewall drop. Add a rule dropping the port on the server. From the client, distinguish the signature: a DROP gives a hang then timeout; a REJECT answers immediately, and HOW depends on what it rejects with -- iptables' default (icmp-port-unreachable) shows as "Connection refused", while firewalld's default rejection (icmp-host-prohibited, what you get on RHEL simply by not allowing the port) shows as "No route to host". Confirm with curl timing, and try both rejections.
  4. Prove the layer without guessing: ping still works (ICMP allowed), the TCP handshake does not. nc -vz server 8080 times out while ping succeeds -> transport-layer block, not routing.
  5. Plant fault B -- the listener bound to 127.0.0.1. Now ss on the server shows the port open, curl localhost on the server works, and the client gets "Connection refused" instantly. Same symptom as REJECT, different cause.
  6. Separate B from the firewall: check the bind address in ss -tlnp (127.0.0.1:8080 vs 0.0.0.0:8080). The firewall was never involved.
  7. Plant fault C -- an MTU mismatch or a wrong route on one host only, so small requests work and large responses hang. Use ping -M do -s 1472 to find it.
  8. For each fault, write the single observation that a colleague could act on.

Verify

# from the client, classify by timing and message:
timeout 5 curl -s -o /dev/null -w '%{http_code}\n' http://server:8080 || echo "no answer"
nc -vz server 8080            # transport reachability, independent of HTTP
ss -tlnp | grep :8080         # run on the SERVER: bind address decides remote reach

The bind-address line settles the commonest false alarm: a service "open" in ss but bound to loopback is unreachable from every other host regardless of firewall state. Instant answer versus hang-then-timeout is the other decisive tell, and it separates a REJECT or closed port from a silent DROP. Read the words too: "No route to host" from a host whose route you have just proven is almost always firewalld rejecting, not routing.

Notes

The through-line is that identical symptoms have different causes at different layers, and only a layered check tells them apart. "Connection refused" is a reached host actively saying no (nothing listening, or a REJECT); a timeout is a packet that vanished (a DROP, a black-hole route, an MTU hole). Reading the failure mode is faster than reading logs.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.