Kubernetes troubleshooting is easier when you move from symptoms to evidence in a consistent order. This reusable checklist helps you diagnose pods, deployments, services, ingress, scheduling, DNS, and rollout failures with focused kubectl commands, verification steps, and low-risk fixes.
Overview
Before changing a workload, establish its scope and current state. Confirm the cluster context, namespace, recent events, and the relationship between the application’s Deployment, ReplicaSet, Pods, Service, and Ingress. Many Kubernetes incidents become harder to resolve when an operator inspects one object in isolation or applies a restart before collecting evidence.
Start with commands that are safe to run and useful across most incidents:
kubectl config current-context
kubectl get namespaces
kubectl get pods -A
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get deploy,rs,svc,ingress -n <namespace>
Replace placeholders with the affected context, namespace, and resource names. If you manage multiple clusters, verify the context every time before making a change. A command aimed at the wrong cluster can create a second incident while you are investigating the first.
Use this workflow as a diagnostic sequence rather than a list of commands to run blindly: observe, form a likely cause, verify it, then make the smallest safe change. Record the relevant output as you go so another operator can follow the reasoning.
Checklist by scenario
1. Pods are in CrashLoopBackOff or repeatedly restarting
Inspect:
kubectl get pod <pod> -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers
kubectl logs <pod> -n <namespace> --previous
Likely causes: an application startup error, an invalid environment variable, a missing configuration or secret, a failed dependency connection, a bad command or argument, or a liveness probe that kills the container before it is ready.
Verify: check the container’s exit code and termination reason, read the previous container’s logs, and inspect the Events section in describe. Compare the configured command, image tag, mounted volumes, environment variables, and probe paths with the application’s expectations.
Safe fix: correct the faulty configuration or image reference through the normal deployment workflow. Avoid deleting the pod before collecting logs; a replacement may remove useful evidence. If the application needs more startup time, review its startup and readiness probes rather than simply increasing every timeout.
2. Pods remain Pending
Inspect:
kubectl get pods -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get nodes
kubectl describe node <node>
Likely causes: insufficient CPU or memory, node selectors or affinity rules that match no nodes, taints without a matching toleration, an unavailable PersistentVolumeClaim, or scheduling constraints that cannot be satisfied.
Verify: read the scheduler Events on the pod. Compare the pod’s resource requests with allocatable capacity, check node labels and taints, and inspect storage claims separately:
kubectl get pvc -n <namespace>
kubectl describe pvc <claim> -n <namespace>
Safe fix: correct an inaccurate selector, add the required toleration only when the workload should run on that node group, or address the storage provisioning issue. Do not remove resource requests merely to force scheduling; that can shift the failure to runtime resource contention.
3. A Deployment rollout is stuck or failing
Inspect:
kubectl rollout status deployment/<deployment> -n <namespace>
kubectl rollout history deployment/<deployment> -n <namespace>
kubectl describe deployment <deployment> -n <namespace>
kubectl get rs,pods -l app=<label> -n <namespace>
Likely causes: new pods cannot schedule, containers fail during startup, readiness probes never succeed, an image cannot be pulled, or the Deployment selector and pod labels do not describe the intended set of pods.
Verify: determine whether the new ReplicaSet created any pods and whether those pods became Ready. Check image pull messages, probe failures, and available versus desired replicas. Inspect the rendered manifest if the Deployment is managed by a package or GitOps workflow.
Safe fix: correct the release configuration and redeploy. If the previous revision is known to be healthy and the current release presents a material impact, use kubectl rollout undo deployment/<deployment> according to your change process. Treat rollback as a stabilization step, then investigate the failed revision separately.
4. A Service is not reachable
Inspect:
kubectl get svc <service> -n <namespace> -o wide
kubectl describe svc <service> -n <namespace>
kubectl get endpoints <service> -n <namespace>
kubectl get endpointslice -l kubernetes.io/service-name=<service> -n <namespace>
Likely causes: the Service selector matches no pods, target ports do not match the container’s listening port, pods are not Ready, the client is using the wrong Service type or port, or a NetworkPolicy blocks traffic.
Verify: compare the Service selector with pod labels, then compare port, targetPort, and the application’s actual listening address. An empty endpoint list usually points to selector or readiness problems. Test from a temporary diagnostic pod in the relevant namespace when appropriate:
kubectl run netcheck --rm -it --restart=Never --image=busybox -- sh
Safe fix: correct labels or port mappings in the source manifest. Review NetworkPolicies before changing them; use the existing isolation model as the guide. For practical isolation patterns, see the Kubernetes Network Policy examples.
5. Ingress returns an error or routes to the wrong backend
Inspect:
kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get svc -n <namespace>
kubectl get pods -n <controller-namespace>
Likely causes: an incorrect host or path rule, a missing Service, a mismatched Service port, an ingress class mismatch, TLS configuration problems, or an unhealthy ingress controller.
Verify: confirm the request host and path, inspect controller events and logs, and test the backend Service directly from inside the cluster. If direct Service traffic works but external traffic fails, focus on the Ingress object, controller, DNS, TLS, or external load-balancer layer.
Safe fix: correct one routing element at a time and retest with the intended host header. Avoid editing the controller’s generated configuration directly unless that is explicitly part of its supported operating procedure.
6. DNS lookup fails inside the cluster
Inspect:
kubectl get pods -n kube-system
kubectl get svc -n kube-system
kubectl logs -n kube-system -l k8s-app=kube-dns
Likely causes: an incorrect Service name or namespace, a pod-level resolver configuration issue, unhealthy cluster DNS components, or a NetworkPolicy blocking DNS traffic.
Verify: run a lookup from the affected namespace and compare a short Service name with its fully qualified name, such as service.namespace.svc. Check the pod’s /etc/resolv.conf and confirm that DNS pods are running and have endpoints.
Safe fix: correct the application’s hostname or namespace reference first. If the failure is systemic, investigate DNS deployment health, service endpoints, node networking, and policy rules before restarting components.
What to double-check
- Namespace and context: confirm both before every read or write operation.
- Labels and selectors: Deployments, Services, NetworkPolicies, and monitoring rules depend on exact label matches.
- Readiness versus liveness: readiness controls traffic; liveness can restart a container. A probe that is too aggressive can create a restart loop.
- Rendered configuration: inspect the final manifest produced by Helm, Kustomize, or a delivery system, not only the source values.
- Events and logs together: Events explain scheduling and control-plane decisions; application logs explain process behavior.
- Recent changes: compare the incident start time with image, configuration, secret, policy, node, and ingress changes.
- Resource requests and limits: verify that scheduling assumptions and runtime capacity are both realistic.
Common mistakes
Deleting resources before investigating. A restart may temporarily hide the cause and remove logs. Capture state first.
Assuming a running pod is healthy. A pod can be Running while its application is not Ready, its endpoint is absent, or its process is returning errors.
Testing only from outside the cluster. External failures can obscure whether the problem is the application, Service, Ingress, DNS, or load-balancing layer. Test each boundary independently.
Changing several variables at once. A simultaneous probe, image, Service, and policy change makes verification difficult and increases rollback risk.
Using forceful fixes as routine procedure. Scaling, deleting, bypassing policies, or removing resource settings can stabilize one symptom while creating capacity, security, or reliability problems elsewhere.
Ignoring the delivery system. A manual fix may be overwritten by CI/CD or GitOps reconciliation. Make the durable correction in the system that owns the manifest. For release hygiene, pair this checklist with the CI/CD pipeline best practices guide.
When to revisit
Keep this checklist with the team’s operational runbooks and review it before seasonal planning cycles, major platform upgrades, or changes to deployment workflows. Revisit it whenever the cluster version, ingress implementation, service mesh, network-policy model, storage platform, observability stack, or CI/CD tooling changes. Those changes can alter command output, resource ownership, failure modes, or the safest recovery procedure.
After a significant incident, add the exact symptom, decisive command, confirmed cause, and approved fix to the relevant scenario. Remove steps that no longer apply and mark commands that are unsafe in production. A short, maintained checklist is more useful than a long runbook that no one trusts.
For the next incident, follow this practical sequence: verify context and namespace; capture status, Events, and logs; identify the failing boundary; test the smallest hypothesis; apply one controlled change; confirm recovery through readiness and user-facing behavior; then record what changed. That sequence turns Kubernetes troubleshooting from trial and error into a repeatable engineering practice.
