Setting up Secrets Manager in your Kubernetes Service cluster

When you integrate IBM Cloud Secrets Manager with your IBM Cloud Kubernetes Service cluster, you can centrally manage Ingress subdomain certificates and other secrets.

About Secrets Manager

With Secrets Manager, you can use a single service to manage your secrets and control who has access to them. A Secrets Manager instance is not automatically provisioned in your cluster. However, you can use a single Secrets Manager instance across multiple clusters, and a single cluster can have more than one instance.

What functionality can I gain with Secrets Manager?

With Secrets Manager, you can:

Create managed Kubernetes secrets with Ingress TLS certificates included.
Create Kubernetes secrets of any type by using the CRN of any Secrets Manager instance you own.
Automatically update your secrets in your cluster on a regular basis.
Track the expiration dates of your certificates from the IBM Cloud console.
Control who has access to your secrets by creating secret groups for approved users.

Note that to have your secrets automatically updated, you must register at least one Secrets Manager instance to your cluster. For more information, see Registering your Secrets Manager instance to your cluster.

Secrets Manager FAQ

Keep the following points in mind when using Secrets Manager.

What types of secrets are supported with Secrets Manager?: Secrets Manager supports IAM credentials, key-value secrets, user credentials, arbitrary secrets, and Kubernetes secrets. For Kubernetes secrets, Secrets Manager supports both TLS and non-TLS (Opaque) secret types. With TLS secrets, you can specify one certificate CRN. With non-TLS secrets, you can specify multiple fields to pull non-certificate secrets. If you do not specify a secret type when you create a secret, TLS is applied by default. For more information on supported secrets, see Working with secrets of different types.
Are secrets that are stored in a registered Secrets Manager instance automatically updated?: Yes. If you have a Secrets Manager instance registered to your cluster, the secrets on the cluster are automatically updated with the values from Secrets Manager once a day. These updates are made using the value of the secret from the corresponding CRN.
Are my secrets automatically updated if I do not create and register a Secrets Manager instance?: If you do not have a Secrets Manager instance registered to your cluster, your default Ingress secrets continue to update automatically every 90 days and are applied to your cluster. However, any secrets you created that reference the default Ingress secret are not automatically updated.; Example scenario: You have a default Ingress certificate in the default namespace. You run the ibmcloud ks ingress secret create command and reference the CRN of the default Ingress certificate to mirror the certificate in the istio-system namespace. Without a Secrets Manager instance, the default Ingress certificate in the default namespace automatically updates. However, you are responsible for regularly updating the certificate in the istio-system namespace with the kubectl commands or another rotation method.
I created secrets that reference the default Ingress certificate, but I have not created and registered a Secrets Manager instance. How do I manage my secrets?: If you don't register a Secrets Manager instance, IBM Cloud Kubernetes Service only automatically updates the default Ingress secret. You are responsible for managing any other secrets by using kubectl commands or another rotation method. If you have any secrets that reference the default Ingress certificate, you should remove them by using ibmcloud ks ingress secret rm.
What is the difference between the ibmcloud ks ingress instance CLI commands and the ibmcloud ks ingress secret CLI commands?: There are two sets of CLI commands that work directly with Secrets Manager instances in IBM Cloud Kubernetes Service: the ibmcloud ks ingress secret commands and the ibmcloud ks ingress instance commands. The ibmcloud ks ingress instance commands are used to manage your Secrets Manager instances. The ibmcloud ks ingress secret commands are used to manage your Ingress secrets that are stored in a Secrets Manager instance or secrets that are written directly to the cluster.

Setting up your Secrets Manager instance

Follow the steps to set up Secrets Manager in your cluster.

Enable service-to-service communication

Integrating Secrets Manager with your IBM Cloud Kubernetes Service cluster requires service-to-service communication authorization. Follow the steps to set up the authorization. For additional info, see Integrations for Secrets Manager.

In the IBM Cloud console, click Manage > Access (IAM).
Click Authorizations.
Click Create.
In the Source service list, select Kubernetes Service.
Select the option to scope the access to All resources.
In the Target service list, select Secrets Manager.
Select the option to scope the access to All resources.
In the Service access section, check the Manager option.
Click Authorize.

Create a Secrets Manager instance

To create a Secrets Manager instance in the CLI or the UI, refer to the Secrets Manager documentation. It might take several minutes for you Secrets Manager instance to fully provision.

When you create a Secrets Manager instance, it is not provisioned directly in your cluster. You must register your new Secrets Manager instance with your cluster in the next step.

Register your Secrets Manager instance to your cluster

Follow the steps to register your Secrets Manager instance to your cluster.

Get the CRN of the Secrets Manager instance. In the output, the CRN is in the ID row.

ibmcloud resource service-instance <instance_name>

Example output

Name:                  my-secrets-manager-instance 
ID:                    crn:v1:bluemix:public:secrets-manager:us-south:a/1aa111aa1a11111aaa1a1111aa1aa111:111a1111-11a1-111a-1111-1a1a1a1111a1:
GUID:                  111a1111-11a1-111a-1111-1a1a1a1111a1 
Location:              us-south   
Service Name:          secrets-manager   
Service Plan Name:     standard   
Resource Group Name:   default   
State:                 active   
Type:                  service_instance   
Sub Type:                 
Created at:            2022-06-08T12:46:45Z   
Created by:            user@ibm.com   
Updated at:            2022-06-08T12:54:45Z

Register the instance to your cluster. Specify the instance CRN found in the previous step.

If you want to register an instance to a cluster and also set it as the default instance, include the --is-default option. Otherwise, you can set a default instance with the ibmcloud ks ingress instance default set command.
```
ibmcloud ks ingress instance register --cluster <cluster_name_or_id> --crn <instance_crn> [--is-default]
```

Verify that the Secrets Manager instance was registered to the cluster.

ibmcloud ks ingress instance ls --cluster <cluster_name_or_id>

Example output

Name                                Type              Is Default   Status    Secret Group   CRN   
my-secrets-manager-instance         secrets-manager   false        created   default        crn:v1:bluemix:public:secrets-manager:us-south:a/1aa111aa1a11111aaa1a1111aa1aa111:111a1111-11a1-111a-1111-1a1a1a1111a1::

You can specify a Secrets Manager instance and a secret group when you create a cluster with the ibmcloud ks cluster create classic or ibmcloud ks cluster create vpc-gen2 commands. Use the --sm-instance option to register an instance to the cluster and the --sm-group option to specify a secret group that can access the secrets on the cluster. See Registering a Secrets Manager instance when creating a cluster.

Set a default Secrets Manager instance and regenerate your secrets

When you set a default Secrets Manager instance, all new Ingress subdomain certificates are stored in that instance.

Run the command to set the new default instance. You can optionally specify a secret group that is allowed access to the secrets in the instance.
```
ibmcloud ks ingress instance default set --cluster <cluster_name_or_id> --name <instance_name> --secret-group <secret_group_id>
```
Regenerate your secrets. Any secrets that are managed by IBM, such as your default Ingress secrets, are uploaded to the new default instance. These secrets are automatically updated and the CRN is changed to reference the Secrets Manager instance.
1. List the nlb-dns subdomains in your cluster.
```
ibmcloud ks nlb-dns ls --cluster <cluster_name_or_id>
```
2. For each subdomain in your cluster, run the command to regenerate your IBM-managed secrets. This updates the CRN of these secrets to reference the CRN of the new default Secrets Manager instance.
  
  Regenerating your secrets is rate-limited to five times per week. Follow the steps in this document carefully, as repeating them might cause you to reach the limit. If you do not regenerate your secrets, or if you have reached the limit, your secrets are uploaded to your Secrets Manager instance at the next renewal cycle.
```
ibmcloud ks nlb-dns secret regenerate --cluster <cluster_name_or_id> --nlb-subdomain <nlb_subdomain>
```
3. Verify that your default Ingress secrets regenerated. In the output, the CRN of the default Ingress secrets should contain secrets-manager.
  
  It might take several minutes for your secrets to regenerate. During this process, the Status column in the output says regenerating and switches to created when the regeneration is complete.
```
ibmcloud ks ingress secret ls --cluster <cluster_name_or_id>
```
  Example output
```
Name                               Namespace  CRN                                                                               Expires On                 Domain                                                   Status    Type   
secret-11111aa1a1a11aa1111111-000  default    crn:v1:bluemix:public:secrets-manager:us-south:a/1aa111aa1:secret:a111aa11-11a1   2022-12-21T19:52:29+0000   secret-11111aa1a1a.us-south.containers.appdomain.cloud   created   TLS   
secret-22222aa2a2a22aa2222222-000  default    crn:v1:bluemix:public:secrets-manager:us-south:a/2aa222aa2:secret:a222aa22-22a2   2022-12-21T19:52:29+0000   secret-22222aa2a2a.us-south.containers.appdomain.cloud   created   TLS    
```

Controlling access to your secrets with secret groups

With Secrets Manager, you can use secret groups to control who has access to the secrets in your cluster. A secret group can be assigned to an IAM access group so that only select users or service IDs can access the secrets within the secret group. For more information, see Organizing your secrets.

Registering a Secrets Manager instance when creating a cluster

If you are creating a new Classic or VPC cluster, you can register an existing Secrets Manager instance and secret group to the cluster during creation. Secrets in the cluster are stored in the Secrets Manager instance and applied to the secret group.

The Secrets Manager instance registered during cluster create does not automatically become the default Secrets Manager instance. You must still set the default instance manually.

If you create a cluster in the CLI with the ibmcloud ks cluster create classic or ibmcloud ks cluster create vpc-gen2, you can specify a Secrets Manager instance or secret group with the following command options:

--sm-instance: Use this option to register a Secrets Manager instance to the cluster by specifying the instance CRN. To find the CRN of a Secrets Manager instance, run ibmcloud resource service-instance <name_of_instance> or navigate to your resource list in the UI and click on the instance.
--sm-group: Use this option to specify the ID of the secret group. To find the secret group ID, run ibmcloud secrets-manager secret-groups.

If you create a cluster in the UI, follow these steps to specify a Secrets Manager instance or secret group:

In the Integrations section of the cluster create page, select the option to enable Secrets Manager.
From the Secrets Manager instance drop down menu, select the instance you want to register to the cluster. If no instances are available, create one.
From the Secrets Manager group drop down menu, select the secret group you want to apply.
Create the cluster.
Check that the Secrets Manager instance is registered to the cluster.
1. When your cluster is fully provisioned, click on the cluster to view the cluster details. Under Integrations, find the Secrets Manager heading and click Manage.
2. In the side panel, check that correct instance is listed under Registered Secrets Manager instances.
3. To register additional instances to the cluster, click Register instances.