4 common Kubernetes Pods Error and Debugging

Why do Kubernetes Pods fail?

The two most common reasons for Kubernetes pod failure is

• The container inside the pod doesn’t start, which we also call a startup failure.
• The application code fails after the container start, also known as runtime failure.

Five most common commands to debug Kubernetes error

• kubectl get pods: This command is used to list all pods
• kubectl describe pod <pod name>: This command is used to check events related to pods
• kubectl logs: To check the pod logs
• kubectl exec -it <pod name>: This will run bash directly in the pod
• kubectl get events: To show the cluster level events

1: ImagePullBackOff error

Reason: Kubernetes isn’t able to retrieve the image for the container. The main reason behind that is that the image name provided in the pod spec is invalid. The tag specified in the image doesn’t exist, or the specified image in the private registry and Kubernetes cluster doesn’t have access to that.

Fix: Correct the image name or tag. Add the credentials to your private registry secret to fix the private registry issue.

• To replicate the issue

kubectl run nginx --image=nginx:2.0

• Check the pod status

# kubectl get pods
nginx                       0/1     ImagePullBackOff   0          6s

• Use describe command

kubectl describe pod nginx

Warning  Failed     8m22s (x4 over 9m56s)   kubelet            Failed to pull image "nginx:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:2.0": failed to resolve reference "docker.io/library/nginx:2.0": docker.io/library/nginx:2.0: not found

• Go to the DockerHub and check the tag associated with nginx image https://hub.docker.com/_/nginx?tab=tags
• As you can see, the latest tag is present in the DockerHub(or depending upon which tag I want to use for my application). After using the latest tag, I can see my pod is running.

$ kubectl run nginx1 --image=nginx:latest

pod/nginx1 created

$ kubectl get pods nginx1

NAME     READY   STATUS    RESTARTS   AGE

nginx1   1/1     Running   0          10s

• To replicate the issue check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/Imagepullbackoff.yaml

kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/Imagepullbackoff.yaml

2: CreateContainerConfigError

Reason: There could be many reasons, but this blog is trying to mount the volume for the secret which doesn’t exist.

Fix: Create the secret

• Check the pod status

$ kubectl get pod

NAME             READY   STATUS                       RESTARTS   AGE

my-config-pod     0/1     CreateContainerConfigError   0          4s

• Run the describe command on the pod

kubectl describe pod my-config-pod
Warning  Failed     13s (x3 over 27s)  kubelet            Error: configmap "mytest-config" not found

• To create the configmap

$ kubectl create configmap mytest-config --from-literal=name=Prashant

configmap/mytest-config created

• Check the status of pod now

$ kubectl get pod my-config-pod

NAME            READY   STATUS    RESTARTS   AGE

my-config-pod   1/1     Running   0          13s

• To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/CreateContainerConfigError.yaml

kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/CreateContainerConfigError.yaml

3: ContainerCreating Error

Reason: There could be many reason but in the above blog its trying to mount volume for the secret which doesn’t exist.

Fix: Create the secret

• Check the status of pod

$ kubectl get pod my-secret-pod

NAME            READY   STATUS              RESTARTS   AGE

my-secret-pod   0/1     ContainerCreating   0          52s

• Run the describe command on the pod

kubectl describe pod my-secret-pod
Warning  FailedMount  30s (x7 over 62s)  kubelet            MountVolume.SetUp failed for volume "my-secret-volume" : secret "mytop-secret" not found

• Create the secret

kubectl create secret generic mytop-secret --from-literal=key1=supersecret

• Check the status of pod now

$ kubectl get pod my-secret-pod

NAME            READY   STATUS      RESTARTS   AGE

my-secret-pod   0/1     Completed   0          4m27s

• To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/containercreating.yaml

kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/containercreating.yaml

4. CrashLoopBackOff Error

CrashLoopBackOff ErrorReason: This error occurs once the container is started and then crashes due to an application in the container’s application that prevents it from starting; you miss configured the container due to a typo mistake, or CMD/ENTRYPOINT is missing, or liveness probe failed too many times.

Fix: Fix the application or misconfigured command

• Check the pod status

$ kubectl get pod

NAME                 READY   STATUS             RESTARTS      AGE

mytest-demo          0/1     CrashLoopBackOff   3 (20s ago)   69s

• Run the describe command on the pod

kubectl describe pod mytest-demo

Message:      failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "printen": executable file not found in $PATH: unknown

• As you can see there is a typo in the command and it should be printenv(rather then printen)
• To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/crashloopbackoff.yaml

kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/crashloopbackoff.yaml