Common Kubernetes Errors

1. ErrImagePull or ImagePullBackOff

The two most frequent issues are specifying the incorrect container image and  attempting to use private images / third party images without supplying registry credentials.

We can check the pods and the error status with the below commands:

$ kubectl get pods

$ kubectl describe pod <pod_name>

Error message: "fail" with ErrImagePull: "Error: image <pod_name> not found"

Solution:

In addition to problems with network connectivity, there are three main offenders:

• The image tag is wrong.

• The image is not real (or is in a different registry)

• Kubernetes lacks authorization to download that image.

2. Liveness/Readiness Probe Failure

Kubernetes will destroy your container and start over if the Liveness Probe fails. No traffic will be delivered to that Pod until it becomes Ready if the Readiness Probe fails, making that Pod unavailable as a Service endpoint.

Error Message:

Killing     Killing container with docker id 0fb5f1a56ea0: pod "liveness-pod_fail(d75469d8-f090-11e6-bd01-42010af0012c)" container "test-container" is unhealthy, it will be killed and re-created.
  Warning Unhealthy   Liveness probe failed: Get http://url/healthz: dial tcp <ip-address>: getsockopt: connection refused
 Warning Unhealthy   Readiness probe failed: Get http://url/healthz: dial tcp <ip-address>: getsockopt: connection refused

Solution:

Three scenarios are most likely:

• Your probes are no longer accurate. Has the URL for health changed?

• Your probes are overly sensitive. Does it take a while for your application to launch or respond?

• The Probe - no longer receives a proper response from your application. Is your database configured incorrectly

3. Secret/Config Missing

Best practises for Kubernetes advise using ConfigMaps or Secrets to pass application run-time configuration. This information could consist of API endpoints, database credentials, or other configuration flags.

Developers frequently make the error of creating Deployments that refer to ConfigMap or Secret properties that are either missing or completely nonexistent.

Code snippet:

  containers:
    - name: test-container
      image: <image-url>
      command: [ "/bin/sh", "-c", "env" ]
      env:
        - name: SPECIAL_KEY
          valueFrom:
            configMapKeyRef:
              name: new-config-name
              key: config.key

  containers:
    - name: test-container
      image: <image-url>
      command: [ "/bin/sh", "-c", "env" ]
      volumeMounts:
        - mountPath: /etc/secret/
          name: mysecret
  restartPolicy: Never
  volumes:
    - name: mysecretvolume
      secret:
        secretName: mysecret

Errors:

Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "test-container" with RunContainerError: "GenerateRunContainerOptions: configmaps \"special-config\" not found"

Warning FailedMount MountVolume.SetUp failed for volume with: secrets "myothersecret" not found

Solution:

The new-config-name ConfigMap that the Pod is trying to access is not present in this namespace. The Pod should restart and pull in the runtime data when we create the ConfigMap.

Make a copy of mysecret that has the required secure credentials to solve this issue. The container will start appropriately once mysecret has been established.

4. CrashLoopBackOff

This tells us that Kuberenetes is trying to launch this Pod, but one or more of the containers is crashing or getting killed.

We can start by looking at our application logs. You may view the application logs using kubectl logs assuming you are sending your application logs to stdout.

kubectl logs <pod-name>

Errors:

$ kubectl describe pod <pod-name>
Name:        <name>
Namespace:    fail
Node:        ****************************************
Start Time:    Fri, 22 Jul 2022 14:20:29 -0500
Labels:        pod-template-hash=2443551393
        run=crasher
Status:        Running
IP:        10.0.0.74
Controllers:    **************
Containers:
  crasher:
    Container ID:    docker://*************************************************
    Image:        **********************
    Image ID:        docker://*************************************************
    Port:        
    State:        Terminated
      Reason:        Error
      Exit Code:    1
      Started:        Fri, 22 Jul 2022 14:20:29 -0500
      Finished:        Fri, 22 Jul 2022 14:20:29 -0500
    Last State:        Terminated
      Reason:        Error
      Exit Code:    1
      Started:        Fri, 22 Jul 2022 14:20:29 -0500
      Finished:        Fri, 22 Jul 2022 14:20:29 -0500
    Ready:        False
    Restart Count:    5
    

Solution:

This Pod is being terminated, according to Kubernetes, since the application inside the container crashed. We can observe that the application's Exit Code is 1, specifically. Another error that could appear is OOMKilled. This is a typical occurrence.

5. Memory issue - CPU/Memory exceeds

Administrators of Kubernetes clusters can restrict the amount of CPU or memory allotted to Pods and Containers. If you are an application developer and are unaware of the constraints, you might be shocked when your deployment fails.

 spec:
      containers:
        - name: test-container
          image: nginx
          resources:
            requests:
              memory: 5Gi

Errors: maximum memory usage per Container is 100Mi, but request is 5Gi

Solution:

Running kubectl describe limitrange will allow you to see the current namespace limits.

You currently have three options:

• Request a limit increase from your cluster administrator.

• Your Deployment's Request or Limit parameters should be decreased.

• Change the boundaries by editing the limits