Kubernetes Ingress troubleshooting becomes much faster when you follow the request path from DNS to the controller, Service, and Pod. This guide provides a reusable checklist for diagnosing 404, 502, TLS, hostname, health-check, and network-policy problems without changing multiple layers at once.
Overview
An Ingress is a set of HTTP or HTTPS routing rules. It does not usually process traffic by itself; an Ingress controller watches those rules and configures a proxy or load balancer to handle requests. A typical request follows this path:
Client
|
| DNS lookup for app.example.com
v
Public load balancer or controller service
|
| TLS termination and hostname/path matching
v
Ingress controller
|
| Service port selection
v
Kubernetes Service
|
| Endpoint and readiness selection
v
Ready application Pod
Each hop can produce a different symptom. A DNS problem may prevent the connection entirely. A TLS problem may occur before an HTTP response exists. A 404 often indicates that the request did not match the expected host or path, although the exact response depends on the controller. A 502 or 503 commonly points to an unavailable, incorrectly addressed, or unhealthy backend.
Start by identifying the boundary where the request fails. Record the exact URL, hostname, path, HTTP method, response code, response headers, and whether the request works when sent directly to the Service. This evidence is more useful than changing annotations or restarting Pods immediately.
Checklist by scenario
1. DNS or connection failure
- Resolve the hostname from the client network:
dig +short app.example.comornslookup app.example.com. - Confirm that the returned address belongs to the intended load balancer or Ingress entry point.
- Check whether the record is an A record, AAAA record, or CNAME and verify that the chosen address is supported by your environment.
- Test connectivity to the expected ports, normally 80 and 443, without assuming that a successful DNS lookup proves the route works.
- Inspect the controller Service and its external address with
kubectl get svc -A.
If DNS points to an old controller, an old load balancer, or the wrong environment, Kubernetes resources may look correct while users reach a different endpoint.
2. 404 or the default backend response
- Inspect the complete Ingress object:
kubectl describe ingress <name> -n <namespace>. - Compare the requested hostname with the Ingress
spec.rules[].hostvalue, including spelling and the exact domain suffix. - Check the path, path type, and whether the client is sending a trailing slash or a different URL prefix.
- Send a request with an explicit Host header to the controller address:
curl -i -H 'Host: app.example.com' http://<address>/health. - Review controller logs for rejected, ignored, or unmatched configuration.
Host matching is especially easy to misread when testing with an IP address. An HTTP request made directly to the load balancer IP may not include the hostname required by the rule. For HTTPS, use a hostname-aware test such as curl --resolve app.example.com:443:<ip> https://app.example.com/.
3. TLS certificate or HTTPS errors
- Confirm that the Ingress has a TLS section for the hostname and references the intended Secret.
- Verify that the Secret exists in the same namespace as the Ingress:
kubectl get secret <tls-secret> -n <namespace>. - Check the certificate names and expiry without exposing the private key:
kubectl get secret <tls-secret> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName. - Test the certificate served to clients:
openssl s_client -connect app.example.com:443 -servername app.example.com. - If cert-manager is used, inspect the Certificate, CertificateRequest, Order, and Challenge resources where applicable.
A valid Secret does not guarantee that the controller is serving it. The hostname used by the client, the TLS entry, certificate names, and controller configuration must all align. For safe credential and certificate rotation practices, see the Secrets Rotation Runbook.
4. 502, 503, or upstream connection errors
- Confirm that the Service exists and exposes the port named by the Ingress:
kubectl get svc <service> -n <namespace> -o yaml. - Compare the Ingress backend port with the Service port. The Ingress normally targets the Service port, not the container port.
- Check endpoints or EndpointSlices:
kubectl get endpointslice -l kubernetes.io/service-name=<service> -n <namespace>. - Inspect selector labels and confirm that ready Pods match them:
kubectl get pods -n <namespace> --show-labels. - Review readiness probes, Pod logs, and recent events. A Pod can be running but excluded from traffic because it is not ready.
- Test the Service from a temporary diagnostic Pod to separate application and Ingress problems:
kubectl run netcheck --rm -it --restart=Never --image=curlimages/curl -- curl -sv http://<service>.<namespace>.svc.cluster.local:<port>/health.
If the direct Service request fails, focus on the application, Service selector, port mapping, readiness, or network policy before changing the Ingress. The Kubernetes Pod Troubleshooting Decision Tree can help when the backend Pods are unhealthy.
5. Network policy or health-check failure
- Identify the source identity used by the controller or load-balancer health checker.
- Review NetworkPolicy ingress rules in the application namespace.
- Ensure the policy permits the controller namespace, Pod labels, or node path required by the deployment model.
- Confirm that the health-check path returns the expected status without authentication if the checker cannot authenticate.
- Check whether the application listens on the address and protocol expected by the Service.
Do not assume that a successful request from a developer workstation proves that the controller can reach the backend. Test from an appropriate in-cluster location and compare source, destination, port, and protocol.
What to double-check
Use this short decision tree after collecting evidence:
Does DNS resolve to the intended entry point?
No -> Correct the DNS record or controller exposure.
Yes
|
Does TLS negotiate for the requested hostname?
No -> Check certificate names, Secret, TLS rule, and controller logs.
Yes
|
Does the host and path match an Ingress rule?
No -> Correct Host, path, pathType, or test headers.
Yes
|
Does the Service have ready endpoints?
No -> Fix selectors, readiness, Pods, or application health.
Yes
|
Can the controller reach the Service port?
No -> Check port mapping, protocol, NetworkPolicy, and controller events.
Yes
|
Inspect application routing, redirects, timeouts, and response behavior.
Capture the configuration in one place while troubleshooting:
kubectl get ingress,svc,pods -n <namespace> -o widekubectl describe ingress <name> -n <namespace>kubectl get events -n <namespace> --sort-by=.lastTimestampkubectl logs -n <controller-namespace> deploy/<controller-deployment>
Controller annotations are implementation-specific. Verify the annotation name, accepted value, scope, and version against the controller documentation used by your cluster. Avoid copying an annotation from one controller into another and assuming equivalent behavior.
Ingress remains common, but teams may evaluate the Gateway API for newer designs. Treat migration as a separate change: map listeners, routes, backends, ownership, TLS configuration, and policy behavior before replacing working resources. Validate both paths in a non-production environment and define rollback steps.
Common mistakes
- Testing the IP without the Host header: this can produce a default 404 even when hostname routing is correct.
- Confusing Service ports and container ports: the Ingress targets the Service interface, while the Service maps to the container.
- Checking only Pod status: “Running” does not necessarily mean a Pod is ready or serving the expected protocol.
- Reading only the application logs: controller events and access logs often show whether the request reached the backend at all.
- Changing several layers at once: simultaneous DNS, certificate, annotation, and deployment changes make the original failure difficult to isolate.
- Exposing sensitive data while debugging: redact tokens, cookies, private keys, and authorization headers from command output and incident notes. For related API and token workflows, review the JWT Debugging and Validation Guide.
When to revisit
Keep this checklist with the service runbook and revisit it before a domain change, certificate renewal, controller upgrade, Kubernetes upgrade, new NetworkPolicy, or backend port change. It is also useful after a load-balancer migration, a change in health-check behavior, or a move from Ingress toward Gateway API.
At each review, confirm that example commands still match the installed controller, API versions, namespace layout, and observability setup. Record the known controller Service, log location, test endpoint, TLS Secret ownership, and rollback procedure. During an incident, begin with one reproducible request, trace it from DNS through the backend, and change only the layer supported by the evidence. That habit turns Kubernetes Ingress troubleshooting from trial and error into a repeatable operational workflow.