Kubernetes pod failures become easier to resolve when you treat the pod status as a starting clue rather than a diagnosis. This decision tree gives you a namespace-safe command sequence, symptom-to-cause checks, and a repeatable review routine for Pending, CrashLoopBackOff, ImagePullBackOff, probe failures, configuration errors, and OOMKilled containers.
Overview
Begin with the workload, namespace, and recent events. Avoid jumping directly to a restart or deleting the pod: those actions can remove useful evidence and may cause the same failure to recur.
Set the namespace explicitly in every command. Replace production and payments-api with values from your cluster:
kubectl get pods -n production
kubectl describe pod payments-api-7f8c6d9b7f-x2k4m -n production
kubectl get events -n production --sort-by='.lastTimestamp'
kubectl logs payments-api-7f8c6d9b7f-x2k4m -n production --all-containers
kubectl logs payments-api-7f8c6d9b7f-x2k4m -n production --previous
The current logs show what the container is doing now. The --previous option is especially useful when a container has restarted and the current instance has not produced much output. For a deployment-level view, check the controller and replica status as well:
kubectl get deployment payments-api -n production
kubectl describe deployment payments-api -n production
kubectl get replicaset -n production
kubectl rollout status deployment/payments-api -n production
Use this decision tree as a first pass:
- Is the pod Pending? Inspect scheduling events, resource requests, node selectors, taints, affinity, and persistent volume claims.
- Is it waiting with ImagePullBackOff or ErrImagePull? Check the image reference, registry access, pull credentials, and node connectivity.
- Is it restarting with CrashLoopBackOff? Read current and previous logs, then inspect the command, arguments, configuration, exit code, and probes.
- Was it terminated with OOMKilled? Compare the container memory limit with actual usage and determine whether the application or the limit needs attention.
- Is the pod Running but unavailable? Check readiness probes, service selectors, endpoints, ports, and application-level health.
What to track
A useful troubleshooting record connects the visible status to the underlying evidence. Capture these fields during an incident or deployment review:
| Signal | What it may indicate | Next check |
|---|---|---|
| Pending | No suitable node, insufficient requested resources, an unsatisfied constraint, or an unresolved volume | kubectl describe pod, node capacity, taints, affinity, and PVC events |
| ImagePullBackOff | Invalid image name or tag, unavailable registry, missing credentials, or authorization failure | Pod events, imagePullSecrets, registry path, and image accessibility |
| CrashLoopBackOff | The process exits repeatedly, often because of configuration, startup, dependency, or probe problems | Current and previous logs, exit code, command, arguments, and probe configuration |
| OOMKilled | The container exceeded its memory limit or the node experienced memory pressure | Container resources, usage metrics, application allocation, and node conditions |
| Running but not Ready | The readiness probe fails or the application is not serving the expected endpoint | Probe path and port, application logs, service endpoints, and network policy |
Pending pods
For a Pending pod, the events at the end of kubectl describe pod usually provide the most direct clue. Messages about insufficient CPU or memory point to scheduling capacity or oversized requests. Messages about taints, node affinity, or selectors point to placement rules. A pod can also remain Pending because its persistent volume claim has not bound.
kubectl get nodes
kubectl describe node worker-1
kubectl get pvc -n production
kubectl describe pvc data-payments -n production
Distinguish a resource request from a limit: requests influence scheduling, while limits constrain runtime usage. Increasing a limit will not make a pod schedulable if its request still cannot fit on an eligible node.
ImagePullBackOff and ErrImagePull
Read the exact event rather than relying on the short status label. Confirm the registry hostname, repository path, tag or digest, and the credentials referenced by imagePullSecrets. If the image is private, verify that the service account used by the pod can access the required secret in the same namespace. Avoid embedding registry credentials in a manifest or posting them in an incident channel.
For build-side investigation, compare the deployment image with the image produced by your pipeline. A valid local build does not prove that the cluster can pull the image. Review the Docker image troubleshooting and hardening guide when the problem may originate in image creation, tagging, or runtime assumptions.
CrashLoopBackOff and probe failures
CrashLoopBackOff is a backoff state, not the root cause. Check the terminated container's exit code and reason:
kubectl get pod payments-api-7f8c6d9b7f-x2k4m -n production -o jsonpath='{.status.containerStatuses[*].state}'
kubectl get pod payments-api-7f8c6d9b7f-x2k4m -n production -o jsonpath='{.status.containerStatuses[*].lastState}'
Common investigation paths include a missing environment variable, an invalid configuration file, a failed database connection, a migration that cannot complete, an incorrect container command, or an application that exits after doing its one-time task. If the process stays alive but the pod is repeatedly restarted, inspect liveness and startup probes. A probe can fail because the path, port, scheme, initial delay, timeout, or expected response does not match the application.
Use a startup probe when an application legitimately needs more time to initialize, and keep readiness separate from liveness: readiness controls traffic eligibility, while liveness is intended to identify an unhealthy process. Change probe thresholds only after confirming the application behavior; relaxing a probe can hide a real startup or dependency problem.
OOMKilled and resource pressure
OOMKilled means the container was terminated after exceeding its memory limit. First verify whether the limit is consistently too low or whether usage grows unexpectedly. Compare container metrics with the configured request and limit, then inspect application heap settings, caches, batch sizes, concurrency, and large payload handling.
kubectl top pod payments-api-7f8c6d9b7f-x2k4m -n production --containers
kubectl describe pod payments-api-7f8c6d9b7f-x2k4m -n production
kubectl describe node worker-1
The availability of resource metrics depends on the cluster's monitoring setup. If kubectl top is unavailable, use the metrics system already approved for your environment. Do not automatically raise limits: a larger limit may defer the failure, increase node pressure, or make the workload harder to schedule.
Cadence and checkpoints
Troubleshooting works better when you track recurring signals instead of treating every failed rollout as an isolated event. Add the following checks to deployment reviews and operational runbooks:
- Every deployment: record rollout status, unavailable replicas, warning events, image identifier, and readiness duration.
- Weekly or during active operations: review CrashLoopBackOff counts, restart increases, Pending duration, failed probes, and image-pull errors by namespace.
- Monthly or quarterly: compare resource requests and limits with observed usage, review recurring node-pressure events, and remove obsolete configuration or image tags from runbooks.
- Before cluster upgrades: validate workload behavior, admission policies, add-ons, and version compatibility. Use the Kubernetes upgrade planning checklist to organize the review.
For each recurring failure, preserve the pod manifest, relevant events, a short log excerpt, the deployment revision, and the remediation. Redact tokens, passwords, certificates, and sensitive request data before storing the record. This creates a small troubleshooting dataset that helps distinguish a new regression from a familiar operational condition.
How to interpret changes
Look for changes in frequency, duration, and scope. One failed pod after a configuration change suggests a workload or deployment issue. Many pods failing across namespaces at the same time suggests a shared dependency, registry, node pool, admission rule, or cluster-level condition.
Correlate the pod timeline with deployment revisions and cluster events:
kubectl rollout history deployment/payments-api -n production
kubectl get pods -n production -o wide
kubectl get events -A --sort-by='.lastTimestamp'
Interpret the most common patterns carefully:
- Pending time increases after a node migration: inspect capacity, taints, affinity, and volume attachment rather than changing application code first.
- Restarts begin after a new image: compare the image's command, environment, filesystem, and startup behavior with the previous revision.
- Readiness failures rise without process exits: examine dependency latency, probe timeouts, service ports, and application health responses.
- OOMKilled appears after traffic increases: compare memory per request, concurrency, cache growth, and payload size before selecting a new limit.
- Image pull errors affect only new nodes: check node connectivity, registry access, runtime configuration, and credentials available to the workload.
Make one controlled change at a time when possible. After the change, verify the pod condition, events, logs, rollout status, and service behavior. A pod becoming Running is not sufficient if it remains unready or if traffic still fails through the service.
When to revisit
Revisit this troubleshooting workflow after Kubernetes version upgrades, changes to the container runtime or node pools, registry or identity changes, new admission policies, and major changes to probes or resource settings. Kubernetes behavior and supported debugging options can change across releases, so validate commands against the version and distribution used by your team.
Use this practical closeout checklist during the next incident:
- Write down the namespace, pod name, deployment revision, and first observed time.
- Capture
kubectl describe pod, sorted namespace events, current logs, and previous logs when restarts occurred. - Classify the failure as scheduling, image pull, process exit, probe, resource, configuration, or dependency related.
- Check whether the symptom affects one pod, one workload, one node pool, or the wider cluster.
- Apply the smallest safe remediation, then confirm rollout, readiness, service endpoints, and application behavior.
- Record the cause, evidence, fix, and a prevention task in the runbook.
For broader service-level diagnosis, pair this pod workflow with the Kubernetes troubleshooting checklist and a production alert runbook. If credentials or configuration exposure is part of the incident, consult the cloud security checklist for developers before sharing diagnostic output.