• About WordPress
    • WordPress.org
    • Documentation
    • Support
    • Feedback
  • Log In
  • Register
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Have any question?

101daysofdevops@gmail.com
RegisterLogin
101DaysofDevops
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Blog

  • Home
  • Blog
  • Blog
  • 4 common Kubernetes Pods Error and Debugging

4 common Kubernetes Pods Error and Debugging

  • Posted by lakhera2020
  • Date April 20, 2022
  • Comments 0 comment

Why do Kubernetes Pods fail?

The two most common reasons for Kubernetes pod failure is

  • The container inside the pod doesn’t start, which we also call a startup failure.
  • The application code fails after the container start, also known as runtime failure.

Five most common commands to debug Kubernetes error

  • kubectl get pods: This command is used to list all pods
  • kubectl describe pod <pod name>: This command is used to check events related to pods
  • kubectl logs: To check the pod logs
  • kubectl exec -it <pod name>: This will run bash directly in the pod
  • kubectl get events: To show the cluster level events

1: ImagePullBackOff error

Reason: Kubernetes isn’t able to retrieve the image for the container. The main reason behind that is that the image name provided in the pod spec is invalid. The tag specified in the image doesn’t exist, or the specified image in the private registry and Kubernetes cluster doesn’t have access to that.

Fix: Correct the image name or tag. Add the credentials to your private registry secret to fix the private registry issue.

  • To replicate the issue
kubectl run nginx --image=nginx:2.0
  • Check the pod status
# kubectl get pods
nginx 0/1
ImagePullBackOff 0 6s
  • Use describe command
kubectl describe pod nginx
Warning  Failed     8m22s (x4 over 9m56s)   kubelet            Failed to pull image "nginx:2.0": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:2.0": failed to resolve reference "docker.io/library/nginx:2.0": docker.io/library/nginx:2.0: not found
  • Go to the DockerHub and check the tag associated with nginx image https://hub.docker.com/_/nginx?tab=tags
  • As you can see, the latest tag is present in the DockerHub(or depending upon which tag I want to use for my application). After using the latest tag, I can see my pod is running.
$ kubectl run nginx1 --image=nginx:latest
pod/nginx1 created
$ kubectl get pods nginx1
NAME     READY   STATUS    RESTARTS   AGE
nginx1   1/1     Running   0          10s
  • To replicate the issue check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/Imagepullbackoff.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/Imagepullbackoff.yaml

2: CreateContainerConfigError

Reason: There could be many reasons, but this blog is trying to mount the volume for the secret which doesn’t exist.

Fix: Create the secret

  • Check the pod status
$ kubectl get pod
NAME             READY   STATUS                       RESTARTS   AGE
my-config-pod     0/1     CreateContainerConfigError   0          4s
  • Run the describe command on the pod
kubectl describe pod my-config-pod
Warning Failed 13s (x3 over 27s) kubelet Error: configmap "mytest-config" not found
  • To create the configmap
$ kubectl create configmap mytest-config --from-literal=name=Prashant
configmap/mytest-config created
  • Check the status of pod now
$ kubectl get pod my-config-pod
NAME            READY   STATUS    RESTARTS   AGE
my-config-pod   1/1     Running   0          13s
  • To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/CreateContainerConfigError.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/CreateContainerConfigError.yaml

3: ContainerCreating Error

Reason: There could be many reason but in the above blog its trying to mount volume for the secret which doesn’t exist.

Fix: Create the secret

  • Check the status of pod
$ kubectl get pod my-secret-pod
NAME            READY   STATUS              RESTARTS   AGE
my-secret-pod   0/1     ContainerCreating   0          52s
  • Run the describe command on the pod
kubectl describe pod my-secret-pod
Warning FailedMount 30s (x7 over 62s) kubelet MountVolume.SetUp failed for volume "my-secret-volume" : secret "mytop-secret" not found
  • Create the secret
kubectl create secret generic mytop-secret --from-literal=key1=supersecret
  • Check the status of pod now
$ kubectl get pod my-secret-pod
NAME            READY   STATUS      RESTARTS   AGE
my-secret-pod   0/1     Completed   0          4m27s
  • To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/containercreating.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/containercreating.yaml

4. CrashLoopBackOff Error

CrashLoopBackOff ErrorReason: This error occurs once the container is started and then crashes due to an application in the container’s application that prevents it from starting; you miss configured the container due to a typo mistake, or CMD/ENTRYPOINT is missing, or liveness probe failed too many times.

Fix: Fix the application or misconfigured command

  • Check the pod status
$ kubectl get pod
NAME                 READY   STATUS             RESTARTS      AGE
mytest-demo          0/1     CrashLoopBackOff   3 (20s ago)   69s
  • Run the describe command on the pod
kubectl describe pod mytest-demo
Message:      failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "printen": executable file not found in $PATH: unknown
  • As you can see there is a typo in the command and it should be printenv(rather then printen)
  • To replicate the issue, check the GitHub code https://github.com/100daysofdevops/kubernetes_debugging/blob/main/crashloopbackoff.yaml
kubectl create -f https://raw.githubusercontent.com/100daysofdevops/kubernetes_debugging/main/crashloopbackoff.yaml

Tag:devops, docker, kubernetes

  • Share:
author avatar
lakhera2020

Previous post

My road to Gremlin Chaos Engineering Practitioner Certificate
April 20, 2022

Next post

Debugging Performance Issue using SAR
April 21, 2022

You may also like

Am I reading the iostat command output correctly?
25 April, 2022

Iostat command came from the same sysstat family package # rpm -qf `which iostat` sysstat-11.7.3-6.el8.x86_64 It mainly read data from /proc/diskstats # cat /proc/diskstats 259 0 nvme1n1 147 0 6536 …

Debugging Performance Issue using SAR
21 April, 2022

What is SAR? SAR is a utility used to collect and report system activity. It collects data relating to most core system functions and writes those metrics to binary data …

My road to Gremlin Chaos Engineering Practitioner Certificate
16 October, 2021

Chaos Engineering is one field that always draws my attention. I came to know about it after I heard about the Netflix Simian Army toolkit https://github.com/Netflix/SimianArmy . At an initial glance, it’s …

Leave A Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Am I reading the iostat command output correctly?
  • Debugging Performance Issue using SAR
  • 4 common Kubernetes Pods Error and Debugging
  • My road to Gremlin Chaos Engineering Practitioner Certificate
  • My road to Certified Kubernetes Security Specialist (CKS)

Recent Comments

  • lakhera2020 on Debugging Performance Issue using SAR
  • Anonymous on Debugging Performance Issue using SAR
  • Pety on Day 2 – MetalLB Load Balancer for Bare Metal Kubernetes
  • akashambasta on Day 1 – AWS IAM User
  • rd on 100 Days of AWS

 

101daysofdevops@gmail.com

  • Home
  • About Us
  • Courses
  • Blog

© 101daysofdevops. All rights reserved.

Login with your site account

Lost your password?

Not a member yet? Register now

Register a new account

Are you a member? Login now