Why does my cluster upgrade fail due to Pod Security upgrade prerequisites?
When you upgrade your IBM Cloud Kubernetes Service from Kubernetes version 1.24 to version 1.25, the upgrade fails and you see an error message similar to the following example.
Version update cancelled. CAE009: Cannot complete cluster master operations because the cluster does not pass Pod Security upgrade prerequisites. Reason: [ClusterRoleBinding 'restricted-psp-user' does not have expected subjects]. For more information, see the troubleshooting docs: 'https://ibm.biz/master_pod_security_upgrade_iks_125'
The Kubernetes PodSecurityPolicy
admission controller was removed in Kubernetes 1.25 and replaced with a new Pod Security Admission controller.
To safely upgrade IBM Cloud Kubernetes Service clusters from version 1.24 to version 1.25, the cluster PodSecurityPolicies
(PSP) and associated role-based access control must satisfy the following prerequisites.
- No PSPs beyond the 5 IBM Cloud defined PSPs can exist.
- The IBM Cloud defined cluster role bindings that give all users and service accounts authority to use the IBM Cloud defined privileged and restricted PSPs must exist.
These prerequisites ensure that the cluster's version 1.24 PodSecurityPolicy
configuration is equivalent to version 1.25 Pod Security Admission configuration and the upgrade and switch to Pod Security Admission does not break existing
applications. Note these prerequisite do not preclude use of third party pod security providers.
Before you begin
- Review the 1.25 version information and update actions.
- Review the Migrating from PSPs to Pod Security Admission guide.
If you are not ready to migrate to Pod Security Admission, you can clear the status message by performing a cluster master refresh.
If you already performed the Pod Security Admission upgrade prerequisite actions, the error message indicates an action that you might have missed or an unexpected change to IBM Cloud Kubernetes Service defined resources that you need to address. Complete the following steps based on the message you are seeing.
Could not get PodSecurityPolicies
- Run the following to command to get your PSPs.
kubectl get podsecuritypolicies
- If there is no error, try again to upgrade the cluster master.
Found non-IBM PodSecurityPolicy
There are additional PSPs that need to be removed.
Before you upgrade your cluster to version 1.25, verify that only the following PSPs exist.
ibm-privileged-psp
ibm-anyuid-psp
ibm-anyuid-hostpath-psp
ibm-anyuid-hostaccess-psp
ibm-restricted-psp
-
List your PSPs.
kubectl get podsecuritypolicies
-
If the additional PodSecurityPolicy is no longer in use, delete it. If there are additional pod security policies, review the migration guide.
kubectl delete podsecuritypolicies PSP
-
Retry the cluster upgrade.
Could not get ClusterRoleBinding privileged-psp-user
This message indicates the privileged-psp-user
cluster role binding does not exist or that some other error prevented the upgrade operation from reading the resource.
- List the resource.
kubectl get clusterrolebindings privileged-psp-user
- If you get a
NotFound
error, create or update theprivileged-psp-user
cluster role binding.
ClusterRoleBinding privileged-psp-user
does not have expected roleRef
This error indicates an unexpected change and that the cluster role binding is referencing the wrong cluster role.
-
Delete the cluster role binding.
kubectl delete clusterrolebinding privileged-psp-user
-
Create the
privileged-psp-user
cluster role binding.
ClusterRoleBinding privileged-psp-user does not have expected subjects
If this change was intentional, you must first determine whether your apps can run with the cluster role binding. If they can, Create or update restricted-psp-user cluster role binding.
Could not get ClusterRoleBinding restricted-psp-user
This message indicates the restricted-psp-user
cluster role binding does not exist or some other error prevented the upgrade operation from reading the resource.
- Get the details of the
restricted-psp-user
cluster role binding.kubectl get clusterrolebindings restricted-psp-user
- If you get a
NotFound
error, create or update restricted-psp-user cluster role binding.
ClusterRoleBinding restricted-psp-user
does not have expected roleRef
This error indicates is an unexpected change and that the cluster role binding is referencing the wrong cluster role.
Complete the following steps to change the roleRef
.
- Delete the cluster role binding.
kubectl delete clusterrolebinding restricted-psp-user
- Create or update restricted-psp-user cluster role binding.
ClusterRoleBinding restricted-psp-user does not have expected subjects
If this change was intentional, you must first determine if applications can run properly with the cluster role binding.
Then, create or update restricted-psp-user cluster role binding.
Creating or updating the privileged-psp-user
ClusterRoleBinding
Create or update privileged-psp-user
cluster role binding by running the following command.
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: privileged-psp-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-privileged-psp-user
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:authenticated
EOF
Creating the restricted-psp-user
cluster role binding
Create or update the restricted-psp-user
ClusterRoleBinding by running the following command.
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: restricted-psp-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-restricted-psp-user
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:masters
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:nodes
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:authenticated
EOF