When DNS, DHCP or NTP breaks the cloud

Why this matters

CompTIA names these three services specifically, and they share a property that makes them worth a lesson: they are invisible when working and produce misleading symptoms when broken.

Nothing reports "DNS is down". It reports that an application cannot reach its database, that authentication is failing, that a deployment timed out. The underlying service is two layers below where the symptom appears, which is why these faults consume disproportionate time.

The defence is to know each one's signature. Time drift breaks authentication before anything else. Address exhaustion breaks only new instances. Stale DNS breaks some clients and not others. Recognising a signature turns a mysterious outage into a five-minute diagnosis.

The lesson

DNS faults: resolver configuration, split horizon, stale records and TTL

DNS is the most common of the three and has the widest range of failure modes.

Resolver configuration. Cloud virtual networks provide a resolver at a reserved address (objective 1.3). Instances get it via DHCP. It breaks when someone overrides it with an on-premises resolver that cannot resolve cloud names, when a hybrid setup has conditional forwarding configured in only one direction, or when a security group or network policy blocks port 53 — which is a real case in a default-deny egress design, and it breaks everything in a way that looks like total network failure.

Split horizon. The same name resolves differently depending on who asks — a private address internally, a public one externally. Legitimate and essential, and it produces confusing faults: it works from your laptop and not from the instance, or the reverse. Always resolve from the affected host.

The private-endpoint case, which is worth singling out because it is the most consequential and the quietest: a private endpoint exists but private DNS is not configured, so the service's name resolves to its public address. Traffic leaves via the NAT gateway and the internet instead of staying on the provider's network. Everything works — more slowly, at greater cost, and outside the network boundary an auditor was told about. Nothing alerts. This is the fault from objective 1.3 seen from the troubleshooting side.

Stale records and TTL. A record changed but clients still use the old value, because the old TTL had not expired — the mechanism from objective 2.3. Worse, some runtimes cache DNS for the life of the process, ignoring TTL entirely, so a long-running application can hold a stale address indefinitely while everything else has moved on. The signature is a subset of clients failing while others work, and restarting the affected process fixing it.

Propagation of a changed record takes up to the previous TTL, which is why lowering TTL days in advance of a planned change is the standard practice.

Diagnosis: resolve from the affected host, query the authoritative server directly to bypass caches, compare what different resolvers return, check the TTL on the answer, and confirm port 53 is permitted outbound.

DHCP option sets, address exhaustion, and the subnet that silently filled up

In a cloud, DHCP is managed by the platform and is less commonly misconfigured — but when it is, the effects are distinctive.

Option sets carry the DNS servers, domain name, NTP servers and other parameters handed to instances. A wrong DNS server here breaks name resolution fleet-wide, which is a common route into the DNS problems above. Changes usually apply on lease renewal or instance restart rather than immediately, which produces a partial fault: new and restarted instances behave one way, long-running ones another. That mixed state is a strong hint you are looking at an option-set change.

Address exhaustion is the classic, and its signature is unmistakable once you know it: existing instances are completely fine, and new ones cannot launch. An autoscaling group fails to add capacity during a peak; a deployment fails with a message about no available addresses; a container cluster cannot schedule new pods.

Why subnets fill up more easily than people expect:

• The provider reserves several addresses per subnet (objective 1.3), so a small subnet has fewer usable addresses than the arithmetic suggests.
• Every network interface consumes one, not every instance — and load balancers, NAT gateways, private endpoints and managed database instances all place interfaces in your subnets.
• Container networking can assign an address per pod, consuming them at a rate that surprises anyone sizing for instances.
• Rolling updates temporarily double the interface count.
• Terminated instances hold their addresses briefly.

The painful part is the fix: a subnet's range usually cannot be expanded. The remedies are to add an additional subnet and extend the workload into it, or to migrate — which is why the range-sizing decision in objective 1.3 is described there as effectively permanent. Monitor available addresses per subnet as a routine metric; it is a leading indicator with a long lead time.

Time drift, and the authentication and logging failures it causes first

NTP is the one people forget, and it has the most surprising failure mode: time problems present as security problems.

Because so much depends on clocks being close:

• Certificate validation fails if the clock is outside a certificate's validity window. A clock badly wrong produces "certificate not yet valid" or "expired" for a perfectly good certificate.
• Kerberos typically permits only about five minutes of skew, and beyond it authentication simply fails.
• Token and assertion validation — SAML assertions, OIDC tokens, signed API requests — all carry timestamps and validity windows, and cloud API calls are themselves signed with a timestamp. A skewed clock produces authentication failures against the cloud provider, which is a bewildering symptom if you do not know to check.
• MFA time-based codes are derived from the clock, so drift makes every code wrong.

And the second category, which affects your ability to diagnose anything:

• Logs from different systems cannot be correlated if their clocks disagree, which undermines every investigation, including the incident response in objective 6.3.
• Events appear out of order, or appear to precede their causes.
• Scheduled jobs run at the wrong time, or twice, or not at all.
• Distributed systems using timestamps for ordering or expiry behave incorrectly.

In a cloud, providers supply a time service at a link-local address, and instances use it by default — so drift is uncommon and usually caused by someone reconfiguring NTP, by egress rules blocking port 123 to an external time source, or by a virtual machine whose clock drifted badly while paused or migrated.

The diagnostic habit: when authentication fails across multiple unrelated systems at once, check the clock early. It is a ten-second check that resolves an otherwise confusing incident. Use the provider's time service, allow NTP egress, and monitor clock offset as a metric.

Provider-supplied services against your own, and the hybrid mix that conflicts

Each of these can be the provider's or yours, and the conflicts arise at the boundary.

Provider-supplied services are highly available, require no maintenance, and integrate with platform features — private endpoint resolution, instance naming, service discovery. They are the default and usually the right answer.

Self-managed services are used when you need features the platform does not offer, when a hybrid estate requires an existing directory's DNS, or when policy demands control.

The conflicts, which are what scenarios describe:

• Both configured, inconsistently. Some instances use the provider resolver, others a domain controller, and behaviour differs by instance for no visible reason.
• Conditional forwarding in one direction only. Cloud resolves on-premises names, on-premises cannot resolve cloud names, or the reverse. Forwarding must be configured both ways and is frequently only done once.
• Split-horizon mismatch, where the two views disagree about a name.
• Overridden option sets. The virtual network hands out the provider resolver and a configuration-management tool overwrites it inside the guest — and the guest's own configuration wins. The fleet then behaves inconsistently depending on whether that tool has run.
• Domain-joined instances that need the directory's DNS to find domain controllers, but the directory's DNS cannot resolve the provider's private endpoints. This is a genuinely common hybrid problem, and the answer is forwarding rules in both directions.
• Time source mismatch, where some instances use the provider's service and others an on-premises source that has itself drifted.

The principle: use the provider's services unless you have a stated reason not to, and where both exist, configure the forwarding and the precedence deliberately rather than letting each instance find its own answer.

Proving a shared service is the cause rather than assuming it because it is convenient

The closing discipline, and it cuts both ways. Shared services are a convenient explanation — "it must be DNS" is the reflex joke — and convenience is not evidence. Equally, they are genuinely the cause often enough that they must be eliminated early rather than late.

Prove it with cheap, decisive tests:

• For DNS: resolve the name from the affected host. If it fails, DNS. If it succeeds, connect to the returned address directly — if that works, DNS returned something wrong; if it also fails, it is connectivity (objective 6.2). Two commands, three outcomes, no ambiguity.
• For DHCP: can existing instances work while new ones fail? That pattern is address exhaustion or an option-set problem, and nothing else produces it so cleanly. Check available addresses in the subnet.
• For NTP: compare the host's clock against a known-good source. Instant, and it rules the whole class in or out.

Then apply the general rules:

• Check the blast radius. A shared service failing affects many unrelated things at once. A single application failing while everything else is fine is not a shared-service problem, however convenient that explanation is.
• Check what changed (objective 3.3). An option set modified, a resolver reconfigured, an egress rule tightened.
• Record what you eliminated, so nobody retests it.
• Escalate with evidence. "Name X resolves to Y from host Z at this time, and the authoritative server returns W" is actionable; "DNS is broken" is not.

And prevent recurrence by monitoring these services directly, because that is what converts them from invisible to visible: synthetic resolution checks for critical names, available addresses per subnet, clock offset per instance, and alerts on changes to DHCP option sets and resolver configuration. All four are cheap, and each one turns a confusing multi-hour incident into an alert that names its own cause.

What to take into the exam

• DNS: resolve from the affected host; split horizon means your laptop proves nothing. Private endpoint without private DNS resolves to the public address — everything works, slower, costlier and outside the boundary. Some runtimes cache DNS for the process lifetime, ignoring TTL.
• Blocked port 53 egress looks like total network failure.
• DHCP address exhaustion: existing instances fine, new ones cannot launch. Reserved addresses, load balancers, endpoints and per-pod addressing consume the range, and a subnet usually cannot be expanded.
• Option-set changes apply on renewal or restart, producing a mixed fleet.
• Time drift presents as authentication failure — certificates, Kerberos, tokens, signed API calls, MFA codes — and destroys log correlation. Check the clock when several unrelated systems fail to authenticate at once.
• Configure hybrid forwarding in both directions; guest-level resolver overrides beat the option set.
• Prove it with the two-command DNS test and the existing-versus-new DHCP pattern, and check blast radius before blaming a shared service.