Kubernetes Upgrade Planning Checklist: Version Skew, Add-ons, Workloads, and Rollback

Overview

Kubernetes upgrades can affect more than the control plane. API behavior, admission policies, storage integrations, networking components, ingress controllers, observability agents, autoscalers, deployment tooling, and application manifests may all depend on the cluster version. A successful upgrade therefore starts with an inventory and ends with evidence that workloads still behave correctly.

Use this guide for a self-managed or managed cluster, adapting the commands and sequencing to your provider. Managed Kubernetes platforms often separate control-plane and node-pool operations, while self-managed clusters may require you to coordinate control-plane components, worker nodes, certificates, and infrastructure automation. In either case, read the provider or distribution-specific upgrade documentation before acting.

The central rule is simple: do not upgrade directly in production without first rehearsing the same change in a representative environment. A staging cluster should use comparable Kubernetes configuration, add-ons, admission rules, workload versions, and traffic patterns. If staging cannot reproduce production closely, document the differences and treat them as upgrade risks.

Define success before the change

• All critical APIs and workloads remain available.
• Nodes rejoin the cluster and report the expected status.
• Deployments can schedule, roll out, and roll back.
• Ingress, DNS, service discovery, storage, and network policies behave normally.
• Metrics, logs, traces, alerts, and audit records remain available.
• You can restore service using a documented recovery path.

Checklist by scenario

1. Before choosing the target version

• Record the current Kubernetes version, node versions, operating systems, container runtime, and cluster distribution.
• Confirm the supported upgrade path. Some environments require sequential minor-version upgrades rather than a direct jump.
• Review the target release notes and removal or deprecation notices for APIs your workloads use.
• Export or otherwise preserve the cluster configuration needed to recreate important resources. Treat backups as incomplete until a restoration test demonstrates that they work.
• List critical namespaces, workloads, persistent volumes, custom resources, admission webhooks, operators, and externally managed dependencies.
• Confirm that your infrastructure-as-code configuration, deployment manifests, and policy definitions are compatible with the target version.

Useful inventory commands include kubectl version, kubectl get nodes -o wide, and kubectl api-resources. Command output varies by client and distribution, so save the results with the change record rather than relying on memory.

2. Check version skew and add-on compatibility

Version skew describes the supported difference between Kubernetes components, clients, and dependencies. It is not a license to mix arbitrary versions. Check the compatibility guidance for the exact Kubernetes release, your provider, and each installed add-on.

Pay particular attention to admission webhooks and custom resource definitions. A webhook that cannot reach its service, uses an incompatible API, or applies unexpectedly broad rules can block deployments and cluster operations. Review failure policies, certificates, service endpoints, and timeout settings before the maintenance window.

3. Validate workloads in staging

1. Apply manifests and verify that deprecated or removed API versions are not used.
2. Roll out representative Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs.
3. Test readiness and liveness probes, graceful termination, disruption budgets, and topology constraints.
4. Exercise ingress routes, internal service calls, DNS resolution, egress controls, and network policies.
5. Attach existing and new persistent volumes, then test application-level reads and writes.
6. Generate normal and elevated traffic to observe scheduling, autoscaling, and resource limits.
7. Confirm that logs, metrics, traces, events, and alerts still reach their destinations.
8. Test a deployment rollback and, where appropriate, a database or stateful-service recovery procedure.

For broader diagnosis, keep the Kubernetes troubleshooting checklist available during the rehearsal. Use the OpenTelemetry observability guide to verify that a version change has not interrupted traces, metrics, or logs.

4. Prepare the production maintenance window

• Assign an upgrade owner, application owners, an observer for monitoring, and a decision-maker for rollback.
• Freeze unrelated infrastructure and application changes for the agreed period.
• Capture baseline error rates, latency, saturation, node health, scheduling failures, and business-level checks.
• Confirm backup freshness and identify the exact restoration procedure and responsible operator.
• Prepare communication messages for start, progress, completion, delay, and recovery.
• Verify access to the cloud console, cluster credentials, infrastructure repository, logs, metrics, and incident channel.
• Define a stop condition, such as sustained application errors, failed node replacement, broken storage operations, or loss of observability.

5. Execute in controlled stages

Follow the upgrade sequence documented by your platform. A common pattern is to upgrade the control plane, update or replace worker nodes, and then validate workloads between stages. For node changes, use a controlled drain policy that respects PodDisruptionBudgets and accounts for singleton or stateful workloads. Never assume that a successful node drain proves that the application is healthy; verify service behavior after rescheduling.

After each stage, check kubectl get nodes, kubectl get pods -A, recent warning events, workload rollout status, ingress responses, storage operations, and monitoring signals. Avoid combining the Kubernetes upgrade with unrelated changes such as a networking redesign, major application release, or storage migration.

What to double-check

API deprecations and manifests

Search repositories, Helm charts, operators, and generated manifests for API versions that may be deprecated or removed in the target release. Validate rendered output rather than checking only source templates. Admission policies and custom resources deserve the same review because they may not appear in ordinary application deployment files.

Stateful workloads and data protection

Confirm replication, backup completion, volume snapshots where supported, and restore instructions. A cluster backup is not automatically an application-consistent database backup. Coordinate with database owners and test recovery at a useful level of detail, including credentials, connection strings, DNS, and application startup order.

Scheduling and capacity

Draining nodes temporarily reduces capacity. Check requests, limits, affinity rules, taints, tolerations, topology spread constraints, and autoscaler limits. Make sure replacement nodes have the required labels, routes, security settings, device plugins, and storage capabilities.

Security and access

Review authentication, authorization, service accounts, workload identity, secrets access, network policies, and audit logging. Keep credentials out of tickets and command history where possible. The cloud security checklist for developers provides a useful companion review for IAM, secrets, containers, and logging.

Rollback versus recovery

A Kubernetes rollback plan is not always a simple version downgrade. Control-plane downgrades may be unsupported, state may have changed, and add-ons may have migrated resources or configuration. Define separate options: stop the rollout, restore workloads to their previous application versions, replace failed nodes, restore data, or use a provider-supported cluster recovery process. Choose the safest supported action based on the failure, rather than forcing a downgrade.

Common mistakes

• Checking only the Kubernetes version: Add-ons, operators, clients, node images, and policies can be the actual source of incompatibility.
• Skipping a representative rehearsal: A small or differently configured test cluster may hide capacity, storage, or admission issues.
• Ignoring deprecated APIs: An application can appear healthy until its next deployment requires an unavailable API version.
• Draining without a capacity plan: PodDisruptionBudgets and scheduling constraints can leave replicas pending or reduce redundancy.
• Treating green nodes as a complete validation: Node readiness does not prove that traffic, storage, autoscaling, or observability work.
• Combining too many changes: Separate version upgrades from application, network, and storage redesigns so failures remain diagnosable.
• Assuming rollback is automatic: Write and rehearse a recovery procedure before the maintenance window.

For incidents during or after the change, use a structured production response process. The production alert runbook template can help connect metrics, logs, and traces to clear actions. If infrastructure state is managed with Terraform, review the Terraform state troubleshooting runbook before modifying state during an upgrade.

When to revisit

Revisit this checklist before every cluster upgrade, not only during scheduled planning cycles. Update it when Kubernetes releases a new minor version, your managed provider changes its supported upgrade path, an add-on changes its compatibility range, or your platform adopts a new CNI, CSI driver, ingress controller, operator, policy engine, or observability stack.

Also refresh the checklist after an upgrade review, incident, failed rehearsal, or significant workload change. Record what was tested, which assumptions proved wrong, how long each stage took, and which alerts helped or failed to help. That record turns the next Kubernetes upgrade from a one-off event into a repeatable platform operation.

Action for your next upgrade: create a version and add-on inventory, select a representative staging target, run the workload validation sequence, document supported recovery options, and schedule production only after the decision-maker can identify both the success criteria and the stop conditions.