System integration: APIs, webhooks and message queues

Why this matters

Cloud systems are assembled rather than built. A working application is a collection of services — some yours, some the provider's, some third-party — connected by integrations, and the integrations are where the interesting failures live.

The defining property is that an integration crosses a boundary you do not fully control. The other side can be slow, unavailable, changed without notice, or rate-limited. A design that assumes the far side always answers promptly and correctly will fail, and the failure will be intermittent, which is the worst kind to diagnose.

CompTIA's bullet is integrating systems for seamless cloud operations, and "seamless" is doing a lot of work in that sentence. This lesson is about what makes it so.

The lesson

REST APIs, authentication, rate limits and pagination as the everyday integration

REST over HTTPS is the default integration style: resources identified by URLs, manipulated with standard methods, with JSON as the usual representation.

What you need to get right in practice:

Authentication. How the caller proves who it is — a token, an API key, mutual TLS, or an OAuth flow. Within a cloud, prefer the platform's own identity (objectives 2.4 and 4.2) so there is no stored credential. For external APIs, the credential belongs in a secret manager and is fetched at run time.

Rate limits. Almost every API limits request rate, and the response is HTTP 429. The correct handling is exponential backoff with jitter: wait, doubling the interval, with a random component. The jitter matters more than it looks — without it, many clients throttled simultaneously retry simultaneously, and the coordinated retry is itself a load spike. Honour a Retry-After header where one is given.

Pagination. A collection endpoint returns a page at a time, and code that assumes the first page is everything works in testing and fails in production when the data grows. Follow pagination properly, and prefer cursor-based pagination over offsets where offered, since offsets skip or duplicate records when the underlying data changes mid-traversal.

Status codes. The distinction that decides retry behaviour: 4xx means the request was wrong — retrying identically will fail identically, so do not retry (except 429 and 408). 5xx means the server failed — retrying may succeed, so back off and retry.

Timeouts. Always set one. A client with no timeout waiting on an unresponsive service holds its connection and thread indefinitely, and enough of them exhausts the pool and takes down a service that is itself healthy. This is the most common way one service's outage becomes several.

Webhooks for push, polling for pull, and the reliability difference

Two ways to learn that something happened.

Polling asks repeatedly. Simple, entirely under your control, and works through firewalls with no inbound path. Its costs are latency — you learn at the next poll — and waste, since most polls find nothing. Polling frequently enough to be responsive usually means being rate-limited.

Webhooks push: the far side calls your endpoint when the event occurs. Immediate, efficient, and it requires you to operate a reachable, reliable HTTP endpoint.

Receiving webhooks correctly is the part people get wrong, and each item here is a real failure:

• Verify the signature. A webhook endpoint is a public URL that causes your system to act. Providers sign payloads; verify the signature and reject anything unsigned. An unauthenticated webhook receiver is an open door.
• Respond quickly, process asynchronously. Acknowledge immediately, put the payload on a queue, and do the work afterwards. Senders time out, and a slow receiver causes retries, which causes duplicates.
• Expect duplicates. Delivery is usually at-least-once, so the same event may arrive more than once. Handling must be idempotent, which is the next section.
• Expect out-of-order arrival. Use the event's own sequence or timestamp rather than arrival order.
• Handle the missed event. Webhooks are not guaranteed. A sender may retry for a while and then give up. The robust pattern is webhooks plus a periodic reconciliation sweep — the same belt-and-braces as objective 5.1's events-plus-schedule.

Message queues and topics, decoupling producers from consumers

Where direct calls couple two systems together, a queue puts a durable buffer between them.

• Queue (point-to-point). A producer writes a message; one consumer processes it. Work distribution.
• Topic (publish-subscribe). A publisher emits an event; every subscriber receives a copy. Notification and fan-out.

What this buys, and these are the exam's answers:

• Temporal decoupling. The consumer can be down; messages wait. The producer is unaffected.
• Load levelling. A spike is absorbed by the queue and drained at the consumer's pace, instead of overwhelming it. This is the answer to a scenario where a downstream system cannot cope with bursts.
• Independent scaling, with queue depth as the ideal autoscaling signal (objective 1.7) because it directly measures unserved work.
• Reliability, since a message is not removed until processing is acknowledged.

The mechanics worth knowing:

• Visibility timeout. A consumer takes a message and it becomes invisible to others for a period. If processing is not acknowledged in time, the message reappears and another consumer takes it. Set the timeout longer than the longest legitimate processing time, or slow messages are processed twice — an extremely common bug.
• Ordering is usually not guaranteed on standard queues. Ordered queues exist and cost throughput. Do not assume ordering.
• Delivery is at-least-once. Exactly-once is very hard and rarely what you are given, which brings us to idempotency again.
• Backlog is the health signal. A growing queue means consumers cannot keep up, and it is a leading indicator of an incident.

Retries, idempotency and dead-letter handling, because integrations fail midway

The three mechanisms that make an unreliable integration behave reliably.

Retries. Transient failures are normal, so retry — but with discipline: exponential backoff with jitter, a maximum attempt count, and only for retryable errors (5xx, timeouts, 429; not 4xx). Retrying a permanent failure forever is a way to convert one broken message into an outage.

Add a circuit breaker for the case where the far side is genuinely down: after repeated failures, stop calling for a period and fail fast. Without it, every request queues against a dead dependency, threads pile up, and the failure propagates upstream — the cascading-failure pattern that takes down systems which had no fault of their own.

Idempotency. Because delivery is at-least-once and retries create duplicates, processing the same message twice must be safe. Ways to achieve it:

• Make the operation naturally idempotent — "set the status to shipped" rather than "increment the count".
• Use an idempotency key: the sender includes a unique identifier, the receiver records processed identifiers and ignores repeats.
• Use a conditional write so a duplicate fails harmlessly.

This is the same property as objectives 2.2 and 5.1, applied at the message level, and it is the single most important idea in this lesson.

Dead-letter queues. A message that fails repeatedly must go somewhere. A dead-letter queue holds it after a configured number of attempts, so it stops blocking the main queue and is preserved for investigation. Two rules: alert on dead-letter depth, because a silent dead-letter queue is a place messages go to be forgotten; and be able to replay from it once the bug is fixed.

A related failure worth naming: the poison message that crashes its consumer. Without a dead-letter queue it is retried forever, killing consumer after consumer, and the queue never drains — a self-sustaining outage caused by one bad record.

Contracts and versioning, so one side can change without breaking the other

The last problem: both sides evolve, and they do so independently.

An interface is a contract, and once anything depends on it, changing it breaks things. The discipline is to distinguish backward-compatible changes from breaking ones:

• Compatible: adding an optional field, adding an endpoint, adding an enumeration value consumers are told to tolerate.
• Breaking: removing or renaming a field, changing a type, making an optional field required, changing the meaning of an existing value.

How to manage it:

• Version the interface — in the path, a header, or a media type — and support the old version while consumers migrate. Announce deprecation with a date, and monitor which consumers are still on the old version so you know when it is safe to remove.
• Be lenient in what you accept, strict in what you send. Consumers should ignore fields they do not recognise, so adding a field is not a breaking change. A consumer that rejects unknown fields makes every addition breaking.
• Use contract testing so both sides verify against the agreed interface in their own pipelines (objective 5.3), catching a breaking change before it ships rather than in production.
• Never break a contract silently. The failure appears in someone else's system, at a time they did not choose, and is diagnosed by people who do not know you made a change.

For asynchronous events the same rules apply to the event schema, and a schema registry is the common mechanism — it holds event definitions, enforces compatibility rules when a new version is registered, and refuses a change that would break existing consumers.

What to take into the exam

• 4xx: do not retry (except 429/408). 5xx: back off and retry. Always set a timeout — an unbounded wait exhausts the caller's pool and spreads the outage.
• 429 means rate-limited: exponential backoff with jitter, honouring Retry-After.
• Webhooks: verify the signature, acknowledge fast and process asynchronously, expect duplicates and out-of-order arrival, and reconcile periodically because delivery is not guaranteed.
• Queues decouple, level load and are the ideal autoscaling signal. Visibility timeout shorter than processing time causes duplicate processing. Standard queues do not guarantee ordering.
• At-least-once delivery makes idempotency mandatory — natural idempotency, an idempotency key, or a conditional write.
• Dead-letter queues catch poison messages; alert on their depth and be able to replay.
• Circuit breakers stop a dead dependency cascading upstream.
• Version interfaces, support the old one during migration, ignore unknown fields, and use contract tests.