State, drift, and why the console is the enemy of IaC

Why this matters

This is the lesson where infrastructure as code stops being a filing system and starts having operational consequences. State and drift are where real estates get into trouble, and they produce the scenarios the exam likes: an apply that wants to destroy production, a fix that disappears overnight, two engineers applying at once, a resource the tool insists does not exist.

None of those are obvious from the tooling's documentation, and all of them are predictable once you understand what the tool actually knows.

The lesson

What a state file holds, why it is sensitive, and where it should live

A provider-neutral IaC tool keeps a state file: its record of which real resources correspond to which declarations in your code. It maps the logical name in your template to the actual identifier the provider assigned, and it caches the attributes last seen.

State exists because the tool must answer "does this already exist?" without a reliable, cheap way to search the whole estate. Provider-native tooling has the same concept, managed for you inside the service, which is one of its advantages.

Three properties follow, and each one is examinable.

State is authoritative about ownership. If it says a resource exists, the tool manages it. If state is lost, the tool believes nothing exists and will try to create everything again — producing duplicates or failing on name collisions. A lost state file does not destroy infrastructure; it orphans it, which is nearly as bad because nothing manages it any more.

State is sensitive. It contains resource identifiers, configuration details, network layout, and frequently secrets in plain text — a generated database password, a key, a connection string. Treat the state file with the same care as a credential store: encrypted at rest, access-controlled, and never in a source repository. Committing state to git is a real and common mistake with real consequences.

State must be shared and protected. It should live in remote backend storage — object storage with versioning and encryption enabled — not on a laptop. Versioning matters: a corrupted or truncated state can be rolled back to the previous version, which has rescued many people.

State locking, and the corruption that two simultaneous applies cause

If two people apply at the same time against the same state, both read it, both make changes, and both write it back. The second write overwrites the first's record. The resources created by the first apply now exist and are not in state — orphaned — and the file may be internally inconsistent.

State locking prevents this: the backend takes a lock for the duration of an operation, and a second apply waits or fails with a clear message. Use a backend that supports it; it is not optional for a team.

Operational points worth knowing:

• A crashed run can leave a stale lock. There is a force-unlock operation, and it must only be used once you are certain no apply is running. Breaking a live lock causes exactly the corruption locking prevents.
• Applying from a pipeline rather than laptops makes concurrent applies structurally unlikely, which is a better fix than relying on people to coordinate.
• Separate state per environment and per stack. One giant state file is slow, has a huge blast radius, and serialises every team behind one lock. This is the operational reason for the blast-radius splitting in the previous lesson.

Configuration drift: the manual fix at 2am that the next apply destroys

Drift is divergence between what the code says and what actually exists. It comes from one place: something changed reality without changing the code.

The usual causes are sympathetic rather than careless:

• An incident at 2am, fixed by hand in the console because that was fastest.
• A permission widened temporarily for a support case and never narrowed.
• Another tool, an autoscaler, or a platform feature changing something.
• A provider-side default change on a resource attribute.

Then the next apply runs. The tool compares desired state to actual, finds the manual change, and reverts it — because that is precisely what it is for. The emergency fix disappears, often without anyone connecting the two events, and the incident recurs.

There is a worse version. If the manual change was structural — someone recreated a resource by hand — the tool may decide the resource it knew about is gone and plan to destroy and recreate what is there now. An apply intended to change a tag proposes to replace a database. This is why the plan output is read before every apply, and why the word replace is the one to look for.

The cultural fix is the only durable one: the console is for reading, not writing. Emergency changes are legitimate, and the rule that makes them safe is that the code is updated to match the same day, before the next apply. Where console access to write must exist, restrict it to break-glass identities that are monitored (objective 4.2).

Detecting drift deliberately rather than discovering it during an incident

Waiting for the next apply to reveal drift means discovering it at the worst moment. Detect it on purpose instead.

• Scheduled plan runs. Run a plan nightly against every environment and alert if it is not empty. A non-empty plan when nobody has changed the code means reality moved. This is cheap, high-signal, and the single most useful practice in the lesson.
• Native drift detection, where the provider or tool offers it as a feature.
• Continuous posture scanning, which looks for policy violations rather than code divergence — related but not the same. Objective 6.5 covers it.

Expect and manage benign drift. Some attributes legitimately change outside your code: an autoscaling group's current instance count, tags applied by a platform service, values the provider manages. Left alone they make every plan noisy, and a noisy plan is one nobody reads — which is how real drift hides. The tooling offers ways to ignore specific attributes, and using them is how you keep the signal meaningful.

When drift is found, decide explicitly between two outcomes: revert it (apply, restoring the declared state) or adopt it (update the code to match, if the change was correct). Both are fine. Leaving it undecided is not.

Importing existing resources, and closing the loop on a console-built estate

Most estates were not built with IaC, which raises the practical question of how to adopt it without rebuilding everything.

Import brings an existing resource under management: you write a declaration matching the resource, then tell the tool to associate that declaration with the real object's identifier. It does not create anything; it adds the mapping to state.

Doing it well:

1. Write the declaration first, as accurately as you can.
2. Import the resource into state.
3. Run a plan. It should be empty. A non-empty plan means your code and reality disagree, and applying it would change the live resource — which is precisely the danger of importing carelessly.
4. Adjust the code until the plan is empty, then commit.

Step 3 is the whole discipline. Importing and applying without checking the plan is how people modify production while trying to document it.

Two practical notes: importing large estates by hand is slow, and there are tools that generate configuration from existing resources to give you a starting point — useful, and their output still needs reading. And not everything needs importing. A pragmatic adoption path is to import the long-lived foundations — networking, identity, shared data services — and rebuild the ephemeral things from code as they are next replaced, which gets most of the benefit for much less work.

What to take into the exam

• State maps declarations to real resources. Losing it orphans infrastructure rather than destroying it.
• State contains secrets in plain text. Encrypted remote backend with versioning; never in source control.
• State locking prevents the corruption of concurrent applies. Force-unlock only when certain nothing is running.
• Drift is reality changing without the code changing, and the next apply reverts it — or worse, plans a replacement. Read the plan.
• Detect drift on a schedule with nightly plans; ignore genuinely benign attributes so the signal stays readable.
• Import, then plan until it is empty, then commit. A non-empty plan after import means applying would change the live resource.