Why does OpenShift show the cluster version is not up to date?
Virtual Private Cloud Classic infrastructure Satellite
You see one or more of the following for an extended period of time.
-
The OpenShift Web Console displays a banner stating that the cluster is updating and the Cluster Settings page shows the current version has a status of
Partial
. -
When you run
ibmcloud oc cluster get --cluster <cluster_name>
command you see awarning
state and a status indicating cluster operators are down-level.State: warning Status: Some Cluster Operators are down-level and need to be updated, see 'https://ibm.biz/rhos_clusterversion_ts'
-
When you run
oc get clusterversion
command you see an earlier version.NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.27 True True 27h Unable to apply 4.12.3: an unknown error has occurred: MultipleErrors
-
You see that a cluster master upgrade was cancelled with a status similar to the following.
Version update cancelled. CAE012: Cannot complete cluster master operations because the current OpenShift clusterversion rollout is not complete. For more information, see the troubleshooting docs: 'https://ibm.biz/rhos_clusterversion_ts'
A cluster master update to a new patch version or an upgrade to a new minor version has not completed.
A cluster master update completes soon after updating the Cluster Version Operator. The Cluster Version Operator manages updates to various Cluster Operators, a process which continues after a cluster master update has completed.
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.11 True False 33h Cluster version is 4.13.11
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
console 4.13.11 True False False 2d3h
csi-snapshot-controller 4.13.11 True False False 2d4h
dns 4.13.11 True False False 2d4h
image-registry 4.13.11 True False False 2d4h
ingress 4.13.11 True False False 2d4h
insights 4.13.11 True False False 2d4h
kube-apiserver 4.13.11 True False False 2d4h
kube-controller-manager 4.13.11 True False False 2d4h
kube-scheduler 4.13.11 True False False 2d4h
kube-storage-version-migrator 4.13.11 True False False 2d4h
marketplace 4.13.11 True False False 2d4h
monitoring 4.13.11 True False False 2d4h
network 4.13.11 True False False 2d4h
node-tuning 4.13.11 True False False 32h
openshift-apiserver 4.13.11 True False False 2d4h
openshift-controller-manager 4.13.11 True False False 2d4h
openshift-samples 4.13.11 True False False 33h
operator-lifecycle-manager 4.13.11 True False False 2d4h
operator-lifecycle-manager-catalog 4.13.11 True False False 2d4h
operator-lifecycle-manager-packageserver 4.13.11 True False False 2d4h
service-ca 4.13.11 True False False 2d4h
storage 4.13.11 True False False 2d4h
If the cluster has just been upgraded, the cluster operators can take several minutes to upgrade. It is normal to see a status like the following with the number of completed steps increasing over time.
Working towards 4.13.11: 511 of 616 done (82% complete), waiting on network
If the cluster continues shows partial update for an extended period there is a problem in the cluster that is preventing the cluster operators from updating to the current version. This can be caused by many things, such as the following examples.
- Unhealthy cluster components such as worker nodes blocking the rollout of operator updates.
- Incorrect change to a cluster operator configuration.
- Conflicting changes to cluster configurations, such as RBAC changes that affect OpenShift components.
Begin by getting more detailed information about the cluster operator update progress and problems.
-
Run the following commands to get more information.
oc get clusterversion
oc get clusteroperators
oc describe clusteroperator NAME
-
Review the MESSAGE column in the command output to identify which cluster operators are currently blocking progress and give an indication of the problem. The way in which OpenShift updates operators can result in unexpected dependencies; for example, some cluster operator updates might be waiting for other operators to complete the step they are running now.
-
Check the worker node status using the
oc get nodes
command. Sometimes nodes are notReady
, which can be due to a lack of resources on a node or a problem withkube-proxy
orkubelet
. This can cause issues during an update.-
If you have recently changed the configuration of one of the incomplete operators, describe the operator for more details by running
oc describe clusteroperator NAME
.For exampleoc describe cluster-operator image-registry
might show an error that helps identify the problem. -
Continue checking the status and logs of the operator pods for further debugging of the issue. Sometimes the operators are degraded, but that does not mean the updates have failed or are not working.
-
-
If you are not able to identify or resolve the problems, or if the issue persists, contact support. Open a support case. In the case details, be sure to include any relevant log files, error messages, or command outputs.