Terraform State Troubleshooting Runbook: Locking, Drift, Imports, and Safe Recovery

Overview

Terraform state is the record Terraform uses to map configuration addresses to real infrastructure objects. It may be stored locally or in a remote backend, and it can include provider-specific identifiers and sensitive values. A healthy workflow keeps state access controlled, changes reviewed, and recovery actions documented.

When a plan looks wrong, first separate three different conditions:

• State access problem: Terraform cannot read, write, or lock the state.
• Configuration problem: the desired resources, arguments, provider settings, or workspace are not what you intended.
• Infrastructure mismatch: the real resource changed outside Terraform, was deleted, was recreated, or was never represented correctly in state.

Do not begin with terraform state rm, manual state-file editing, or a forced unlock. Those actions can remove Terraform's record without changing the actual infrastructure. Treat state recovery as a controlled change: identify the workspace and backend, preserve evidence, confirm ownership, and make one reversible decision at a time.

For a broader delivery context, compare these checks with the CI/CD pipeline checklist. State commands should be protected by the same review and access controls as other production changes.

Checklist by scenario

Scenario 1: Terraform reports a state lock

1. Stop concurrent Terraform runs, including CI jobs, scheduled plans, and another operator's terminal session.
2. Confirm the exact backend, workspace, account, and repository revision. A lock in one workspace must not be treated as a lock in another.
3. Read the lock metadata and identify the operation, owner, timestamp, and lock ID if the backend exposes them.
4. Check whether the original process is still running or whether it failed while leaving a stale lock.
5. If an active operation exists, wait or coordinate with its owner. Do not force-unlock an active run.
6. Only after confirming the lock is stale, use the backend-supported unlock process and the exact lock ID. Record who approved it and why.
7. Run a read-only review, such as terraform plan, before applying anything. Investigate any unexpected replacement or deletion.

Decision rule: an unknown lock owner is not proof that the lock is stale. Investigate the runner, job history, and backend before removing it.

Scenario 2: The plan shows unexpected drift

1. Save the plan output and note the workspace, commit, variable set, provider versions, and identity used.
2. Check whether the resource was changed manually, by another automation system, by a platform process, or by a recovery operation.
3. Verify that the data source values, environment variables, secrets, region, account, and provider aliases are correct.
4. Inspect the resource in the cloud control plane and compare its relevant attributes with both configuration and state.
5. Decide whether the external change should be restored, accepted into configuration, or intentionally ignored with a narrowly scoped lifecycle rule.
6. Apply only after the proposed action matches the approved outcome.

Drift detection is most useful when it produces a decision, not merely a warning. If the external change is legitimate, update the configuration and review the resulting plan. If it is not legitimate, restore it through the normal Terraform workflow where possible.

Scenario 3: Importing an existing resource

1. Confirm the resource is supported by the provider and identify the exact provider-specific import identifier.
2. Write the destination resource address first, including module and instance notation when applicable.
3. Add or validate the resource configuration before importing. Start with required arguments and attributes that affect identity or replacement behavior.
4. Use an import block when your team wants the import recorded as configuration, or use the CLI import workflow where that is the established practice.
5. Run terraform plan after import and remove every unexplained change before applying.
6. Document ownership, dependencies, naming differences, and any attributes that are intentionally managed elsewhere.

A minimal configuration template looks like this:

import {
  to = module.network.example_resource.main
  id = "provider-specific-resource-id"
}

resource "example_resource" "main" {
  # Add the arguments required by the provider and your policy.
  # Validate with plan before applying any changes.
}

The address and ID above are placeholders. Use the provider documentation and your resource inventory to determine the real values; never infer an import ID from a display name alone.

Scenario 4: An orphaned resource or address mismatch

1. Determine whether the object exists in the provider and whether Terraform state contains it under another address.
2. Check for a module rename, resource rename, moved block, changed key, or changed count/index expression.
3. If the object should remain managed, use an address move or state move approach rather than recreating it.
4. If the object should no longer be managed, confirm that removing it from state is intentional and understand that this does not delete the real object.
5. Run a plan afterward and review dependent resources for replacements.

What to double-check

Before any state operation, use this compact safety table:

Also double-check sensitive-data handling. Do not paste state, plans, tokens, credentials, or provider responses into public issue trackers or unsecured chat. Review access to the backend and CI logs as part of your cloud security checklist.

If a plan discrepancy appears only in automation, compare the local and CI working directories, Terraform versions, lock file, environment variables, backend configuration, credentials, and workspace selection. A plan generated against the wrong account can look like widespread drift.

Common mistakes

• Force-unlocking first: this can allow two writers to operate against the same state.
• Using -lock=false as a workaround: it may hide contention rather than solve it and should not become a routine production practice.
• Assuming drift means state corruption: many differences are caused by legitimate external changes, wrong variables, or a different provider context.
• Importing without configuration: the next plan may propose changes simply because required arguments are missing or incorrect.
• Deleting state to fix a bad plan: removing an address does not remove the infrastructure and can cause Terraform to lose ownership context.
• Editing state manually without a recovery plan: malformed or incomplete state can make later operations harder to diagnose.
• Skipping a backup or version check: before sensitive recovery work, identify the backend's available state versions or snapshots and confirm how restoration would work.

When recovery is complex, pause and involve the infrastructure owner. Capture the last successful run, the first failing run, relevant plan output, backend information, and the intended final ownership model.

When to revisit

Revisit this runbook before seasonal planning cycles, infrastructure migrations, workspace reorganizations, provider or Terraform upgrades, and major CI/CD workflow changes. It is also useful after a new team member takes ownership of a workspace, after a backend migration, or when resources are moved between modules or accounts.

Keep a short workspace recovery record with the backend type, workspace naming convention, lock procedure, approval path, state-version retention approach, and contacts for active automation. Update command examples when your Terraform version, backend behavior, provider import syntax, or internal workflow changes.

During an incident, use this final sequence: stop concurrent writers, identify the exact context, preserve evidence, classify the problem, choose the smallest safe state action, plan, review, and only then apply. That sequence is the reusable control that prevents a temporary state issue from becoming an infrastructure outage.