Setting up cluster secret encryption
Virtual Private Cloud Classic infrastructure Satellite
After creating a cluster, you can protect Kubernetes secrets and any credentials stored in your secrets by enabling a key management service (KMS) provider, such as IBM® Key Protect for IBM Cloud® or Hyper Protect Crypto Services.
Enabling secret encryption from the CLI
You can enable a KMS provider, update the KMS provider instance, or update the root key through the CLI.
Setting up cross-account encryption by using a KMS in a different account is supported in the CLI or API.
-
Create a KMS instance and root key. If you want to use cross account KMS encryption, make sure to create the KMS and root key in the account whose KMS instance you want to use.
-
Get the ID of the KMS instance that you previously created.
ibmcloud oc kms instance ls
-
Get the ID of the root key that you previously created.
ibmcloud oc kms crk ls --instance-id <KMS_instance_ID>
-
Enable the KMS provider to encrypt secrets in your cluster. Fill in the options with the information that you previously retrieved. The KMS provider's private cloud service endpoint is used by default to download the encryption keys. To use the public cloud service endpoint instead, include the
--public-endpoint
option. The enablement process can take some time to complete.ibmcloud oc kms enable -c <cluster_name_or_ID> --instance-id <kms_instance_ID> --crk <root_key_ID> [--public-endpoint]
During the enablement, you might not be able to access the Kubernetes master such as to update YAML configurations for deployments.
-
Verify that the KMS enablement process is finished. The process is finished when that the Master Status is Ready and Key management service is enabled.
ibmcloud oc cluster get -c <cluster_name_or_ID>
Example output when the enablement is in progress
NAME: <cluster_name> ID: <cluster_ID> ... Master Status: Key management service feature enablement in progress.
Example output when the master is ready
NAME: <cluster_name> ID: <cluster_ID> ... Master Status: Ready (1 min ago) ... Key Management Service: enabled
After the KMS provider is enabled in the cluster, all cluster secrets are automatically encrypted.
-
Optional: Verify that your secrets are encrypted.
Do not delete root keys in your KMS instance, even if you rotate to use a new key. If you delete a root key that a cluster uses, the cluster becomes unusable, loses all its data, and can't be recovered. When you rotate a root key, you can't reuse a previous root key for the same cluster. Similarly, if you disable a root key, operations that rely on reading secrets fail. Unlike deleting a root key, however, you can reenable a disabled key to make your cluster usable again.
Enabling secret encryption from the console
You can enable a KMS provider, update the KMS provider instance, or update the root key through the IBM Cloud console.
-
Create a KMS instance and root key. If you want to use cross account KMS encryption, make sure to create the KMS and root key in the account whose KMS instance you want to use.
-
From the Red Hat OpenShift clusters console, select the cluster that you want to enable encryption for.
-
From the Overview tab, in the Summary > Key management service section, click Enable. If you already enabled the KMS provider, click Update.
-
Select the Key management service instance and Root key that you want to use for the encryption.
During the enablement, you might not be able to access the Kubernetes master such as to update YAML configurations for deployments.
-
Click Enable (or Update).
-
Verify that the KMS enablement process is finished. From the Summary > Master status section, you can check the progress. Example output when the enablement is in progress.
Master status KMS feature enablement in progress.
Example output when the master is ready.
Master status Ready
After the KMS provider is enabled in the cluster, all cluster secrets are automatically encrypted.
-
Optional: Verify that your secrets are encrypted.
Do not delete root keys in your KMS instance, even if you rotate to use a new key. If you delete a root key that a cluster uses, the cluster becomes unusable, loses all its data, and can't be recovered. When you rotate a root key, you can't reuse a previous root key for the same cluster. Similarly, if you disable a root key, operations that rely on reading secrets fail. Unlike deleting a root key, however, you can reenable a disabled key to make your cluster usable again.
Rotating the root key for your cluster
To rotate the root key that is used to encrypt your cluster, repeat the steps to enable KMS encryption. When you rotate a root key, you can't reuse a previous root key for the same cluster.
You can rotate the root key manually from your KMS instance. This action automatically re-enables KMS in your cluster with the new root key. To manually rotate your keys, see your KMS provider docs.
- IBM® Key Protect for IBM Cloud®: See Rotating your keys.
- Hyper Protect Crypto Services: See Rotating root keys manually.
Verifying secret encryption
After you enable a KMS provider in your Red Hat OpenShift on IBM Cloud cluster, you can verify that your cluster secrets are encrypted by disabling the root key. When you disable the root key, the cluster can no longer decrypt the secrets and becomes unusable, which signifies that your secrets were encrypted.
Make sure that you have the IBM Cloud IAM Administrator platform and Manager service access role for the cluster.
-
To check that KMS encryption is enabled, verify that the Key Management Service status is set to
enabled
in the output of the following command.ibmcloud oc cluster get -c <cluster_name_or_ID>
-
Verify that you can list the secrets in your cluster.
oc get secrets --all-namespaces
-
In your KMS instance, disable the root key that is used to encrypt your cluster. If you encrypted your cluster with a KMS and CRK from a different account, the CRK can only be disabled from the account where it is located.
-
Wait for the cluster to detect the change to your root key.
-
Try to list your secrets. You get a timeout error because you can no longer connect to your cluster. If you try to set the context for your cluster by running
ibmcloud oc cluster config
, the command fails.oc get secrets --all-namespaces
Example output
Unable to connect to the server: dial tcp 169.48.110.250:32346: i/o timeout
-
Check that your cluster is in a warning state. Your cluster remains in this state and is unusable until you enable your root key again.
ibmcloud oc cluster get -c <cluster_name_or_ID>
-
In your KMS instance, enable the root key so that your cluster returns to a normal state and becomes usable again.