Secrets Rotation Runbook: Rotate Cloud Credentials, API Keys, and Certificates Without Downtime

Overview

Secret rotation is more than generating a replacement value. A safe change must identify every consumer, introduce the new credential, move traffic or authentication to it, verify that the old value is no longer accepted, and preserve a controlled rollback path. The exact commands differ between cloud providers, secret stores, identity systems, and deployment platforms, but the operating sequence is broadly consistent.

Prefer a managed secret store or workload identity integration over placing long-lived credentials in source code, container images, shell history, tickets, or CI logs. Applications should read secrets at runtime or during a controlled deployment, and access should be limited to the workloads that need it. For Kubernetes workloads, this may involve a secret-store integration, a mounted secret, or an application deployment that reloads configuration without exposing values in manifests or logs.

Before beginning, define the change boundary. Record the secret owner, purpose, consumers, environment, provider, expiration or review date, rotation method, and rollback condition. A small inventory is often more useful than an unstructured list:

• Secret: descriptive name and system of record.
• Type: API key, access token, database password, signing key, private key, or certificate.
• Consumers: services, jobs, pipelines, integrations, and human operators.
• Change method: dual-key overlap, in-place replacement, identity binding, or certificate renewal.
• Validation: health checks, authentication tests, error-rate checks, and provider-side activity.
• Retirement evidence: revocation result, failed-use test where safe, and audit record.

For broader controls around IAM, logging, APIs, and secret storage, use the Cloud Security Checklist for Developers alongside this runbook.

Checklist by scenario

API keys and service credentials

1. Confirm the current key's owner, scope, allowed services, source restrictions, and consumers.
2. Check whether the provider supports two active keys or an equivalent overlap period. If it does, create the replacement before disabling the original.
3. Store the new value in the approved secret store. Do not paste it into chat, an issue, a deployment manifest, or a local file that is not protected.
4. Update one non-production consumer first, then run an authentication and functional test.
5. Roll the change through production consumers using the normal deployment or configuration workflow. Keep the old key available only for the planned overlap window.
6. Check request success rates, authentication failures, latency, and provider activity. A successful startup alone does not prove that every background job or integration has changed.
7. Disable or revoke the old key. Record the timestamp, operator, affected systems, and provider response.
8. Where it is safe, perform a controlled negative test showing that the old key cannot authenticate. Never conduct this test against a live customer workflow if it could create side effects.

Database passwords and application secrets

1. Identify every application, migration job, worker, dashboard, and emergency procedure that uses the credential.
2. Confirm how applications load configuration: startup only, periodic refresh, sidecar synchronization, or deployment-triggered reload.
3. Create a replacement credential with the same required privileges, or use the database's supported overlap mechanism.
4. Deploy the new value to a canary or limited workload. Test connection establishment, reads, writes, transactions, and connection-pool recovery as appropriate.
5. Update remaining workloads and verify that old processes are not still using stale configuration.
6. Revoke the former password or credential after the overlap period. Monitor connection errors and failed authentication attempts afterward.

Certificates and private keys

1. Inventory the certificate chain, private key, hostname coverage, termination point, dependent services, and renewal owner.
2. Confirm the renewal method and whether the platform reloads certificates automatically or requires a restart, reload, or rollout.
3. Generate or request the replacement certificate and protect the private key throughout the process.
4. Validate the certificate before deployment: subject or SAN coverage, validity dates, intended use, chain completeness, and key-to-certificate match.
5. Deploy to a test endpoint or one production instance where possible. Test both the normal hostname and any important alternate paths.
6. Complete the rollout and verify the served certificate from outside the application host. For a TLS endpoint, a provider-neutral check can begin with openssl s_client -connect example.com:443 -servername example.com; inspect the output without copying private material into logs or tickets.
7. Retain the former certificate only for the documented rollback period, then remove its private key from active systems and record the retirement.

Employee, vendor, or workload offboarding

1. List direct credentials, shared keys, tokens, certificates, automation accounts, and recovery paths associated with the person or vendor.
2. Disable identity access first when immediate containment is required.
3. Rotate shared secrets that the departing party could have accessed, even if there is no evidence of misuse.
4. Review access logs, recent key use, pipeline variables, and integrations for unexpected activity.
5. Confirm that replacement ownership, alerts, and renewal reminders are assigned to an active team or service identity.

What to double-check

Most rotation failures happen at system boundaries rather than during secret generation. Before disabling an old value, check the following:

• Hidden consumers: scheduled jobs, disaster-recovery environments, scripts, integration platforms, and rarely used administrative tools.
• Configuration precedence: environment variables, mounted files, defaults, and cached values may not be updated in the same way.
• Deployment behavior: determine whether a rollout restarts processes, refreshes a sidecar, or leaves existing connections unchanged.
• Permissions: match the replacement's scope to the intended workload; do not use rotation as a reason to grant broader access.
• Observability: make authentication failures searchable and connect them to deployment, secret-store, and provider audit events. The OpenTelemetry Observability Guide can help structure related telemetry.
• Rollback: define whether rollback means restoring the previous secret, re-enabling the old key, reverting a deployment, or issuing another replacement. A rollback must not leave both credentials active indefinitely.
• Evidence: capture metadata, not secret values. Record who approved the change, what systems were updated, when the old credential was revoked, and which checks passed.

Common mistakes

• Replacing before mapping consumers: an apparently unused credential may belong to a monthly job or recovery workflow. Inventory first.
• Assuming a restart is enough: some workers, caches, connection pools, and long-running processes retain old values until explicitly reloaded.
• Deleting the old key too early: without an overlap window, a typo or missed consumer can become an outage. Use dual-key rotation when supported.
• Leaving both keys active: overlap is temporary, not the final state. Schedule and verify revocation.
• Testing only the health endpoint: authentication can succeed while a specific write path, queue consumer, or third-party operation is broken.
• Logging sensitive values: avoid debug output that prints headers, environment variables, connection strings, JWTs, or private-key material. If a secret appears in logs, treat it as exposed and rotate it again.
• Rotating without ownership: every secret needs a service owner, renewal method, and review date. Otherwise the next change becomes an emergency investigation.

For production changes, connect this runbook to deployment gates and rollback procedures. The CI/CD Pipeline Checklist provides a useful place to add secret validation without exposing values.

When to revisit

Review this runbook before seasonal planning cycles and whenever the underlying workflow changes. Revisit it immediately after a cloud-provider migration, secret-store replacement, identity-model change, certificate-authority change, new deployment platform, or incident involving leaked or misused credentials.

At each review, compare the inventory with deployed services, CI/CD variables, infrastructure code, Kubernetes workloads, and provider audit records. Confirm that owners and escalation contacts are current, rotation intervals still match the credential's risk and provider capabilities, and alerts cover failed authentication and upcoming certificate expiry. Refresh provider-specific commands only after testing them in a non-production environment.

Before the next rotation, turn the following into a change ticket:

1. Name the credential, owner, consumers, and planned window.
2. Choose the overlap and rollback method.
3. Define pre-change, in-change, and post-change validation.
4. Assign an operator and an observer for production work.
5. Record revocation evidence and close the task only after old-credential use has been ruled out.

Keep this page with your incident and infrastructure runbooks. Related guidance includes the Production Alert Runbook Template, Terraform State Troubleshooting Runbook, and Kubernetes Troubleshooting Checklist.