Resolving permission and credential errors

When you try to make changes to your cluster, such as creating or deleting worker nodes or other components, you receive an error message similar to one of the following examples.

The infrastructure authentication credentials are not authorized for the request.

We were unable to connect to your Softlayer account.
Creating a standard cluster requires that you have either a
Pay-As-You-Go account that is linked to an IBM Cloud infrastructure
account term or that you have used the Kubernetes service
CLI to set your Infrastructure API keys.

'Item' must be ordered with permission.

The worker node instance '<ID>' can't be found. Review '<provider>' infrastructure user permissions.

The worker node instance can't be found. Review '<provider>' infrastructure user permissions.

The worker node instance can't be identified. Review '<provider>' infrastructure user permissions.

The IAM token exchange request failed with the message: <message>
IAM token exchange request failed: <message>

The cluster could not be configured with the registry. Make sure that you have the Administrator role for Container Registry.

The infrastructure credentials that are set for the region and resource group are missing the appropriate infrastructure permissions, or the credentials are not recognized.

There are multiple reasons why this can occur.

• You do not have the required infrastructure permissions.

• The resource group and region are mismatched.

  • Credentials and the IAM API key are set to a region and a resource group. The region is specified when you run the ibmcloud ks credential set command. The resource group applied is whichever resource group is targeted when you run this command. If you do not explicitly target a resource group with the ibmcloud target command before you run ibmcloud ks credential set, it is possible that the API key's resource group is not the one you expected, resulting is mismatched credentials that are not recognized.

• Credentials were added to or removed from the cluster.

  • If you created a cluster with a linked IBM Cloud infrastructure account and then later added or removed credentials with the ibmcloud ks credential set or ibmcloud ks credential unset, the credentials might not match the specifications for the linked account. This can result in the credentials being unrecognized.

The account owner must set up the infrastructure account credentials properly. The credentials depend on what type of infrastructure account you are using.

Before you begin, Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster..

1. Identify what user credentials are used for the region and resource group's infrastructure permissions.

   1. Check the API key for a region and resource group of the cluster.

      ibmcloud ks api-key info --cluster <cluster_name_or_ID>
      

      Example output

      Getting information about the API key owner for cluster <cluster_name>...
      OK
      Name                Email
      <user_name>         <name@email.com>
      

   2. Check if the classic infrastructure account for the region and resource group is manually set to use a different IBM Cloud infrastructure account.

      ibmcloud ks credential get --region <us-south>
      

      Example output if credentials are set to use a different classic account. In this case, the user's infrastructure credentials are used for the region and resource group that you targeted, even if a different user's credentials are stored in the API key that you retrieved in the previous step.

      OK
      Infrastructure credentials for user name <1234567_name@email.com> set for resource group <resource_group_name>.
      

      Example output if credentials are not set to use a different classic account. In this case, the API key owner that you retrieved in the previous step has the infrastructure credentials that are used for the region and resource group.

      FAILED
      No credentials set for resource group <resource_group_name>.: The user credentials could not be found. (E0051)
      

2. Validate the infrastructure permissions that the user has.

   1. List the suggested and required infrastructure permissions for the region and resource group.

      ibmcloud ks infra-permissions get --region <region>
      

      For console and CLI commands to assign these permissions, see Classic infrastructure roles.

   2. Make sure that the infrastructure credentials owner for the API key or the manually set account has the correct permissions. You can change the API key or manually set infrastructure credentials owner for the region and resource group.

3. Try again to perform the infrastructure operation, such as deleting the cluster or worker node. If you still run into the permissions or credentials error, review these additional troubleshooting pages.

   1. If the worker node is not removed, review the State and Status fields and the common issues with worker nodes to continue debugging.
   2. If you manually set credentials and still can't see the cluster's worker nodes in your infrastructure account, you might check whether the cluster is orphaned.

4. If the issue persists, gather the following information to submit to IBM Cloud support. Save the outputs from each command. Make sure that you have the correct resource group targeted with the ibmcloud target -g <resource_group> command.

   1. API key info.

      ibmcloud ks api-key info --cluster <cluster_name_or_id>
      

   2. Account details.

      ibmcloud target
      

   3. Credential details for the expected region and resource group.

      ibmcloud ks credential get --region <region>
      

   4. Infrastructure permissions details.

      ibmcloud ks infra-permissions get --region <region>
      

5. [Open an issue with IBM Cloud support](/docs/containers?topic=containers-get-help. Be sure to include all the information and command outputs gathered in the previous step.

Error: SoftLayer_Exception_User_Customer_Unauthorized: Invalid API key