4 common Kubernetes Pods Error and Debugging
Why do Kubernetes Pods fail?
The two most common reasons for Kubernetes pod failure is
- The container inside the pod doesn’t start, which we also call a startup failure.
- The application code fails after the container start, also known as runtime failure.
Five most common commands to debug Kubernetes error
- kubectl get pods: This command is used to list all pods
- kubectl describe pod <pod name>: This command is used to check events related to pods
- kubectl logs: To check the pod logs
- kubectl exec -it <pod name>: This will run bash directly in the pod
- kubectl get events: To show the cluster level events
1: ImagePullBackOff error
Reason: Kubernetes isn’t able to retrieve the image for the container. The main reason behind that is that the image name provided in the pod spec is invalid. The tag specified in the image doesn’t exist, or the specified image in the private registry and Kubernetes cluster doesn’t have access to that.
Fix: Correct the image name or tag. Add the credentials to your private registry secret to fix the private registry issue.
- To replicate the issue
kubectl run nginx --image=nginx:2.0
- Check the pod status
# kubectl get pods
nginx 0/1 ImagePullBackOff 0 6s
- Use describe command
kubectl describe pod nginx
Warning Failed 8m22s (x4 over 9m56s) kubelet Failed to pull image "nginx:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:2.0": failed to resolve reference "docker.io/library/nginx:2.0": docker.io/library/nginx:2.0: not found
- Go to the DockerHub and check the tag associated with nginx image https://hub.docker.com/_/nginx?tab=tags
- As you can see, the latest tag is present in the DockerHub(or depending upon which tag I want to use for my application). After using the latest tag, I can see my pod is running.
$ kubectl run nginx1 --image=nginx:latest
pod/nginx1 created
$ kubectl get pods nginx1
NAME READY STATUS RESTARTS AGE
nginx1 1/1 Running 0 10s
- To replicate the issue check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/Imagepullbackoff.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/Imagepullbackoff.yaml
2: CreateContainerConfigError
Reason: There could be many reasons, but this blog is trying to mount the volume for the secret which doesn’t exist.
Fix: Create the secret
- Check the pod status
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
my-config-pod 0/1 CreateContainerConfigError 0 4s
- Run the describe command on the pod
kubectl describe pod my-config-pod
Warning Failed 13s (x3 over 27s) kubelet Error: configmap "mytest-config" not found
- To create the configmap
$ kubectl create configmap mytest-config --from-literal=name=Prashant
configmap/mytest-config created
- Check the status of pod now
$ kubectl get pod my-config-pod
NAME READY STATUS RESTARTS AGE
my-config-pod 1/1 Running 0 13s
- To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/CreateContainerConfigError.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/CreateContainerConfigError.yaml
3: ContainerCreating Error
Reason: There could be many reason but in the above blog its trying to mount volume for the secret which doesn’t exist.
Fix: Create the secret
- Check the status of pod
$ kubectl get pod my-secret-pod
NAME READY STATUS RESTARTS AGE
my-secret-pod 0/1 ContainerCreating 0 52s
- Run the describe command on the pod
kubectl describe pod my-secret-pod
Warning FailedMount 30s (x7 over 62s) kubelet MountVolume.SetUp failed for volume "my-secret-volume" : secret "mytop-secret" not found
- Create the secret
kubectl create secret generic mytop-secret --from-literal=key1=supersecret
- Check the status of pod now
$ kubectl get pod my-secret-pod
NAME READY STATUS RESTARTS AGE
my-secret-pod 0/1 Completed 0 4m27s
- To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/containercreating.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/containercreating.yaml
4. CrashLoopBackOff Error
CrashLoopBackOff ErrorReason: This error occurs once the container is started and then crashes due to an application in the container’s application that prevents it from starting; you miss configured the container due to a typo mistake, or CMD/ENTRYPOINT is missing, or liveness probe failed too many times.
Fix: Fix the application or misconfigured command
- Check the pod status
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
mytest-demo 0/1 CrashLoopBackOff 3 (20s ago) 69s
- Run the describe command on the pod
kubectl describe pod mytest-demo
Message: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "printen": executable file not found in $PATH: unknown
- As you can see there is a typo in the command and it should be printenv(rather then printen)
- To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/crashloopbackoff.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/crashloopbackoff.yaml
Tag:devops, docker, kubernetes