Scaling horizontally and vertically, and knowing which you need

Why this matters

Objective 1.7 met scaling as a cost lever. This domain meets it as an operational discipline: not "which is cheaper" but "how do you run this safely in production, day after day, without dropping requests".

The operational side has failure modes the cost side never sees. Scaling in can drop live connections. Scaling out can overwhelm a shared dependency. A group can oscillate. A cache can be cold at exactly the moment demand arrives. These are the scenarios this domain asks about, and each has a specific answer.

The lesson

Vertical scaling, its ceiling, and the restart it usually requires

Vertical scaling changes the size of a single instance — more vCPU, more memory, faster storage.

Its operational properties:

• It usually requires a restart. Stop, change the type, start. That is downtime for anything without redundancy, which makes vertical scaling a planned change rather than a response to load.
• There is a ceiling. The largest instance type is a hard limit. A workload approaching it has nowhere left to go, and that is the moment to have already designed an alternative.
• The address may change on stop/start, which matters for anything hard-coded — and stopping and starting is also the remedy for a noisy neighbour (objective 1.2), so the two interact.
• Cost is not linear with benefit. Doubling the instance rarely doubles throughput, because something else — a lock, a single-threaded component, a downstream service — becomes the constraint.
• It may change licensing cost (objective 2.1), sometimes dramatically.

Vertical scaling remains the right answer for relational database primaries, for software that cannot be clustered, and as a fast remedy when a workload is genuinely undersized. It is the wrong answer to a demand curve, because you cannot do it every hour.

Horizontal scaling, and the statelessness it demands of the application

Horizontal scaling changes the number of instances behind a load balancer. It is the cloud-native answer: no ceiling worth worrying about, no downtime to adjust, and improved availability as a side effect because the failure of one instance is survivable.

The precondition is that any instance can serve any request. That requires:

• No local session state, or shared session state.
• No local file storage that other instances need.
• No singleton assumptions — a scheduled job that must run once cannot live on every instance without coordination. This is a genuinely common bug: three instances each firing the same nightly job.
• Tolerance of being terminated, at any moment, mid-request.
• Fast, unattended startup. An instance taking fifteen minutes to become useful cannot respond to a demand spike, which is what predictive scaling and pre-warmed capacity exist to work around.

Two things that must scale with the fleet and often do not: database connections, since a hundred instances each holding a pool can exhaust the database's connection limit — which is why connection pooling proxies exist — and licence counts for anything licensed per instance.

Session state, sticky sessions, and moving state out to a shared store

Session state is where horizontal scaling usually first goes wrong, because traditional applications kept it in memory.

Sticky sessions (session affinity) make the load balancer send a user back to the same instance. It makes an in-memory application work behind a load balancer, and it costs:

• Uneven load. Instances accumulate users and stay busy after a spike ends.
• Lost sessions on instance replacement. Scale-in, a rolling update, or a failure logs those users out — and rolling updates happen constantly.
• Poor scale-in behaviour, since removing any instance harms someone.

So sticky sessions are a migration crutch, and the real answer is to move the state out:

• Shared session store — an in-memory cache or a key-value store. Fast, and now the session survives any instance's death.
• Client-side sessions — a signed token carried by the client. No server-side state at all, with the trade that tokens are hard to revoke before expiry and must be kept small.
• Database-backed sessions — durable, slower, fine at modest scale.

The shared-store approach introduces a dependency that must itself be resilient: if every request touches the session store, it is now on the critical path and needs its own replication.

The same reasoning applies to uploaded files — object storage rather than local disk — and caches, which should be shared rather than per-instance so the hit rate does not collapse as the fleet grows.

Scheduled, reactive and predictive scaling against three different demand shapes

Three mechanisms, matched to three demand shapes.

Scheduled. Change capacity by clock or calendar. Correct when demand is predictable: office hours, a nightly batch, a known campaign, market open. It is also the cheapest reliable saving in most estates — scaling non-production to zero outside working hours.

Reactive. Respond to a measured signal crossing a threshold. The general case, and its quality depends on the signal (objective 1.7): queue depth and concurrency usually beat CPU. Its weakness is that it is inherently late — it responds after demand has arrived, and the new capacity takes time to become useful.

Predictive. Forecast from historical patterns and scale ahead of demand. Valuable where instances take minutes to become useful and demand rises faster than that. It needs a genuine repeating pattern; it cannot anticipate a novel event.

Most production systems use two or three together: a scheduled baseline that covers the known shape, reactive policies for variation, and a minimum that never drops below what a sudden arrival would need.

Two operational settings that decide whether it behaves:

• Cooldown / warm-up periods. After a scaling action, ignore the metric long enough for the change to take effect. Without this a group reacts to its own last action and oscillates, adding and removing instances continuously — which costs money and destabilises everything downstream.
• Minimum and maximum. The minimum protects against a cold start when traffic arrives. The maximum is the only thing between a traffic anomaly and an unbounded bill — and it is also a limit that will silently stop a legitimate scale-up, so it needs monitoring against the quotas from objective 2.4.

Scale-in safety: draining connections instead of terminating them

Scaling out is easy. Scaling in is where production breaks, and this is the most practically valuable part of the lesson.

Terminating an instance that is serving requests fails those requests. The correct sequence is connection draining (or deregistration delay):

1. Deregister the instance from the load balancer so no new requests arrive.
2. Wait for in-flight requests to complete, up to a configured timeout.
3. Then terminate.

Getting this right requires attention on both sides:

• Set the drain timeout longer than the longest legitimate request. A 30-second drain on a system with two-minute report generation cuts those reports off.
• The application must handle the shutdown signal — stop accepting work, finish what it has, exit. An application that ignores it is killed regardless of the drain setting.
• Long-lived connections need special handling. WebSockets and streaming connections may never end on their own; they need a server-initiated close so clients reconnect elsewhere.
• Workers must not lose work. A queue consumer terminated mid-message must either finish it or return it to the queue for redelivery — which is why the idempotency and retry rules in objective 5.4 matter here.

The related choice is which instance to remove. Termination policies can prefer the oldest instance (helpful, since it is furthest from the current image) or balance across zones (important, since scaling in without regard to zones can leave everything in one and destroy the resilience the spread was for).

Finally, scale in more slowly than you scale out. Asymmetric policies — aggressive out, conservative in — cost a little in instance hours and avoid both oscillation and the case where demand returns immediately after capacity was removed.

What to take into the exam

• Vertical: restart, hard ceiling, may change the licence cost. A planned change, not a response to load.
• Horizontal needs statelessness: no local sessions or files, no singleton jobs, tolerance of termination, fast startup. Watch database connection limits as the fleet grows.
• Sticky sessions are a crutch that causes uneven load and lost sessions on replacement. Move state to a shared store.
• Scheduled for known patterns, reactive for variation, predictive for slow startup. Combine them.
• Cooldowns prevent oscillation; minimums prevent cold starts; maximums cap the bill and can silently block a legitimate scale-up.
• Scale-in must drain: deregister, wait longer than the longest request, then terminate — and the application must handle the shutdown signal.
• Scale in more slowly than you scale out.