IBM Cloud Docs
Why do images fail to pull from registry with ImagePullBackOff or authorization errors?

Why do images fail to pull from registry with ImagePullBackOff or authorization errors?

Virtual Private Cloud Classic infrastructure

When you deploy a workload that pulls an image from IBM Cloud Container Registry, your pods fail with an ImagePullBackOff status.

oc get pods

NAME         READY     STATUS             RESTARTS   AGE
<pod_name>   0/1       ImagePullBackOff   0          2m

When you describe the pod, you see authentication errors similar to the following.

oc describe pod <pod_name>

Failed to pull image "<region>.icr.io/<namespace>/<image>:<tag>" ... unauthorized: authentication required
Failed to pull image "<region>.icr.io/<namespace>/<image>:<tag>" ... 401 Unauthorized

Failed to pull image "registry.ng.bluemix.net/<namespace>/<image>:<tag>" ... unauthorized: authentication required
Failed to pull image "registry.ng.bluemix.net/<namespace>/<image>:<tag>" ... 401 Unauthorized

...
Failed to pull image "<image>:<tag>" ... Manifest for <image>:<tag> not found

Your cluster uses an API key that is stored in an image pull secret to authorize the cluster to pull images from IBM Cloud Container Registry, or the image with the specific tag does not exist in the repository.

By default, new clusters have image pull secrets that use API keys so that the cluster can pull images from any regional icr.io registry for containers that are deployed to the default Red Hat OpenShift project.

  1. Verify that you use the correct name and tag of the image in your deployment YAML file.

    ibmcloud cr images

  2. Check your pull traffic and storage quota. If the limit is reached, free up used storage or ask your registry administrator to increase the quota.

    ibmcloud cr quota

  3. Get the pod configuration file of a failing pod, and look for the imagePullSecrets section.

    oc get pod <pod_name> -o yaml

    Example output

    ...
imagePullSecrets:
- name: all-icr-io
...

  4. If no image pull secrets are listed, set up the image pull secret in your project.

    1. Verify that the default project has icr-io image pull secrets for each regional registry that you want to use. If no icr-io secrets are listed in the project, use the ibmcloud oc cluster pull-secret apply --cluster <cluster_name_or_ID> command to create the image pull secrets in the default project.
      oc get secrets -n default | grep "icr-io"
    2. Copy the all-icr-io image pull secret from the default Red Hat OpenShift project to the project where you want to deploy your workload.
    3. Add the image pull secret to the service account for this Red Hat OpenShift project so that all pods in the project can use the image pull secret credentials.

  5. If image pull secrets are listed in the pod, determine what type of credentials you use to access IBM Cloud Container Registry.

Troubleshooting image pull secrets that use API keys

If your pod configuration has an image pull secret that uses an API key, check that the API key credentials are set up correctly.

The following steps assume that the API key stores the credentials of a service ID. If you set up your image pull secret to use an API key of an individual user, you must verify that user's IBM Cloud IAM permissions and credentials.

  1. Find the service ID that API key uses for the image pull secret by reviewing the Description. The service ID that is created with the cluster is named cluster-<cluster_ID> and is used in the default Red Hat OpenShift project. If you created another service ID such as to access a different Red Hat OpenShift project or to modify IBM Cloud IAM permissions, you customized the description.

    ibmcloud iam service-ids

    Example output

    UUID                Name               Created At              Last Updated            Description                                                                                                                                                                                         Locked
ServiceId-aa11...   <service_ID_name>  2019-02-01T19:01+0000   2019-02-01T19:01+0000   ID for <cluster_name>                                                                                                                                         false
ServiceId-bb22...   <service_ID_name>  2019-02-01T19:01+0000   2019-02-01T19:01+0000   Service ID for IBM Cloud Container Registry in Kubernetes cluster <cluster_name> namespace <project>                                                                                                                                         false

  2. Verify that the service ID is assigned at least an IBM Cloud IAM Reader service access role policy for IBM Cloud Container Registry. If the service ID does not have the Reader service access role, edit the IAM policies. If the policies are correct, continue with the next step to see if the credentials are valid.

    ibmcloud iam service-policies <service_ID_name>

    Example output

    Policy ID:   a111a111-b22b-333c-d4dd-e555555555e5
Roles:       Reader
Resources:
              Service Name       container-registry
              Service Instance
              Region
              Resource Type      namespace
              Resource           <registry_namespace>

  3. Check if the image pull secret credentials are valid.

    1. Get the image pull secret configuration. If the pod is not in the default project, include the -n option.

      oc get secret <image_pull_secret_name> -o yaml [-n <project>]

    2. In the output, copy the base64 encoded value of the .dockerconfigjson field.

      apiVersion: v1
kind: Secret
data:
  .dockerconfigjson: eyJyZWdp...==
...

    3. Decode the base64 string. For example, on OS X you can run the following command.

      echo -n "<base64_string>" | base64 --decode

      Example output

      {"auths":{"<region>.icr.io":{"username":"iamapikey","password":"<password_string>","email":"<name@abc.com>","auth":"<auth_string>"}}}

    4. Compare the image pull secret regional registry domain name with the domain name that you specified in the container image. By default, new clusters have image pull secrets for each regional registry domain name for containers that run in the default Red Hat OpenShift project. However, if you modified the default settings or are using a different Red Hat OpenShift project, you might not have an image pull secret for the regional registry. Copy an image pull secret for the regional registry domain name.

    5. Log in to the registry from your local machine by using the username and password from your image pull secret. If you can't log in, you might need to fix the service ID.

      docker login -u iamapikey -p <password_string> <region>.icr.io
      1. Re-create the cluster service ID, IBM Cloud IAM policies, API key, and image pull secrets for containers that run in the default Red Hat OpenShift project.
        ibmcloud oc cluster pull-secret apply --cluster <cluster_name_or_ID>
      2. Re-create your deployment in the default Red Hat OpenShift project. If you still see an authorization error message, repeat Steps 1-5 with the new image pull secrets. If you still can't log in, open an IBM Cloud Support case.

    6. If the login succeeds, pull an image locally. If the command fails with an access denied error, the registry account is in a different IBM Cloud account than the one your cluster is in. Create an image pull secret to access images in the other account. If you can pull an image to your local machine, then your API key has correct permissions, but the API setup in your cluster is not correct.

      docker pull <region>icr.io/<namespace>/<image>:<tag>

    7. Check that the pull secret is either referenced directly from the deployment or from the service account that the deployment uses. If you still can't resolve the issue, contact support.