Deploying an app that uses COS
Create a persistent volume claim (PVC) to provision IBM Cloud Object Storage for your cluster.
Depending on the settings that you choose in your PVC, you can provision IBM Cloud Object Storage in the following ways:
- Dynamic provisioning: When you create the PVC, the matching persistent volume (PV) and the bucket in your IBM Cloud Object Storage service instance are automatically created.
- Static provisioning: You can reference an existing bucket in your IBM Cloud Object Storage service instance in your PVC. When you create the PVC, only the matching PV is automatically created and linked to your existing bucket in IBM Cloud Object Storage.
Before you begin:
- Create and prepare your IBM Cloud Object Storage service instance.
- Create a secret to store your IBM Cloud Object Storage service credentials.
- Decide on the configuration for your IBM Cloud Object Storage.
To add IBM Cloud Object Storage to your cluster:
-
Create a configuration file to define your persistent volume claim (PVC). If you add your IBM Cloud Object Storage credentials to the default storage classes, do not list your secret in the PVC.
kind: PersistentVolumeClaim apiVersion: v1 metadata: name: <name> # Enter the name of the PVC. namespace: <namespace> # Enter the namespace where you want to create the PVC. The PVC must be created in the same namespace where you created the Kubernetes secret for your service credentials and where you want to run your pod. annotations: ibm.io/auto-create-bucket: "<true_or_false>" ibm.io/auto-delete-bucket: "<true_or_false>" ibm.io/bucket: "<bucket_name>" ibm.io/object-path: "<bucket_subdirectory>" ibm.io/quota-limit: "true/false" # Disable or enable a quota limit for your PVC. To use this annotation you must specify the -set quotaLimit=true option during installation. ibm.io/endpoint: "https://<s3fs_service_endpoint>" ibm.io/tls-cipher-suite: "default" ibm.io/secret-name: "<secret_name>" # The name of your Kubernetes secret that you created. ibm.io/secret-namespace: "<secret-namespace>" # By default, the COS plug-in searches for your secret in the same namespace where you create the PVC. If you created your secret in a namespace other than the namespace where you want to create your PVC, enter the namespace where you created your secret. ibm.io/add-mount-param: "<option-1>,<option-2>" # s3fs mount options ibm.io/access-policy-allowed-ips: "XX.XXX.XX.XXX, XX.XX.XX.XXX, XX.XX.XX.XX" # A csv of allow listed IPs. spec: accessModes: - ReadWriteOnce resources: requests: storage: <size> storageClassName: <storage_class>
ibm.io/auto-create-bucket
- Choose between the following options.
true
: When you create the PVC, the PV and the bucket in your IBM Cloud Object Storage service instance are automatically created. Choose this option to create a new bucket in your IBM Cloud Object Storage service instance. Note that the service credentials must have Writer permissions to automatically create the bucket.false
: Choose this option if you want to access data in an existing bucket. When you create the PVC, the PV is automatically created and linked to the bucket that you specify inibm.io/bucket
.ibm.io/auto-delete-bucket
- Choose between the following options.
true
: Your data, the bucket, and the PV is automatically removed when you delete the PVC. Your IBM Cloud Object Storage service instance remains and is not deleted. If you choose to set this option totrue
, then you must setibm.io/auto-create-bucket: true
andibm.io/bucket: ""
so that your bucket is automatically created with a name with the formattmp-s3fs-xxxx
.false
: When you delete the PVC, the PV is deleted automatically, but your data and the bucket in your IBM Cloud Object Storage service instance remain. To access your data, you must create a new PVC with the name of your existing bucket.ibm.io/bucket
- Choose between the following options.
- If
ibm.io/auto-create-bucket
is set totrue
: Enter the name of the bucket that you want to create in IBM Cloud Object Storage. If in additionibm.io/auto-delete-bucket
is set totrue
, you must leave this field blank to automatically assign your bucket a name with the formattmp-s3fs-xxxx
. The name must be unique in IBM Cloud Object Storage. - If
ibm.io/auto-create-bucket
is set tofalse
: Enter the name of the existing bucket that you want to access in the cluster. ibm.io/object-path
- Optional: Enter the name of the existing subdirectory in your bucket that you want to mount. Use this option if you want to mount a subdirectory only and not the entire bucket. To mount a subdirectory, you must set
ibm.io/auto-create-bucket: "false"
and provide the name of the bucket inibm.io/bucket
. ibm.io/quota-limit
- To use this annotation, you must specify the
--set quotaLimit=true
option during installation. If you want to use this annotation, but didn't specify--set quotaLimit=true
during installation, re-install the helm chart. - If
ibm.io/quota-limit
is set totrue
, your PVC will set a maximum amount of storage (in bytes) available for the bucket based on the sizestorage: <size>
that you specify. - If
ibm.io/quota-limit
is set tofalse
, the quota is not enforced on your PVC meaning that actual amount of storage in bytes might exceed thestorage: <size>
that you specified depending on your app. ibm.io/endpoint
- If you created your IBM Cloud Object Storage service instance in a location that is different from your cluster, enter the private or public cloud service endpoint of your IBM Cloud Object Storage service instance that you want to use. For
more information and an overview of available service endpoints, see Additional endpoint information. By default, the
ibmc
Helm plug-in automatically retrieves your cluster location and creates the storage classes by using the IBM Cloud Object Storage private cloud service endpoint that matches your cluster location. If your classic cluster is in a multizone metro, such asdal10
, the IBM Cloud Object Storage private cloud service endpoint for the multizone metro, for example Dallas. To verify that the service endpoint in your storage classes matches the service endpoint of your service instance, runoc describe storageclass < storageclassname>
. Make sure that you enter your service endpoint in the formathttps://< s3fs_private_service_endpoint>
for private cloud service endpoints, orhttp://< s3fs_public_service_endpoint>
for public cloud service endpoints. If the service endpoint in your storage class matches the service endpoint of your IBM Cloud Object Storage service instance, don't include theibm.io/endpoint
option in your PVC YAML file. ibm.io/add-mount-param
- Enter mount options for your s3fs Fuse volumes. For example
ibm.io/add-mount-param: "del_cache,retries=6"
. For a list of option, see the s3fs man pages ibm.io/access-policy-allowed-ips
- Enter a comma-separated list of IPs that can access your volumes. For example,
ibm.io/access-policy-allowed-ips: "XX.XXX.XX.XXX, XX.XX.XX.XXX, XX.XX.XX.XX
. storage
- In the spec resources requests section, enter a size for your IBM Cloud Object Storage bucket in gigabytes. The actual space that you use in IBM Cloud Object Storage might be different and is billed based on the pricing table. If you enabled quotas when you installed the plug-in , the quota for your bucket is equal to this size.
storageClassName
- Choose between the following options.
- If
ibm.io/auto-create-bucket: "true"
: Enter the storage class that you want to use for your new bucket. - If
ibm.io/auto-create-bucket: "false"
: Enter the storage class that you used to create your existing bucket. - If you manually created the bucket in your IBM Cloud Object Storage service instance or you can't remember the storage class that you used, find your service instance in the IBM Cloud dashboard and review the Class and Location of your existing bucket. Then, use the appropriate storage class. The IBM Cloud Object Storage API endpoint that is set in your storage class is based on the region that your cluster is in. If you want to access a bucket that is located in a different region than the one where your cluster is in, you must create your own storage class and use the appropriate API endpoint for your bucket.
ibm.io/secret-name
- Enter the name of the secret that holds the IBM Cloud Object Storage credentials that you created earlier. If you add your IBM Cloud Object Storage credentials to the default storage classes, you must not list secrets in the PVC. If you want to integrate Key Protect encryption when creating new buckets from PVCs in your cluster, you must include the root key CRN when creating your IBM Cloud Object Storage secret. Note that you cannot add Key Protect encryption to existing buckets.
secret-namespace
- By default, the COS plug-in searches for your secret in the same namespace where you create the PVC. If you created your secret in a namespace other than the namespace where you want to create your PVC, enter the namespace where you created your secret.
-
Create the PVC in your cluster.
oc apply -f filepath/pvc.yaml
-
Verify that your PVC is created and bound to the PV.
oc get pvc
Example output
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE s3fs-test-pvc Bound pvc-b38b30f9-1234-11e8-ad2b-t910456jbe12 8Gi RWO ibmc-s3fs-standard-cross-region 1h
-
Optional: If you plan to access your data with a non-root user, or added files to an existing IBM Cloud Object Storage bucket by using the console or the API directly, make sure that the files have the correct permission assigned so that your app can successfully read and update the files as needed.
-
To mount the PV to your deployment, create a configuration
.yaml
file and specify the PVC that binds the PV.apiVersion: apps/v1 kind: Deployment metadata: name: <deployment_name> labels: app: <deployment_label> spec: selector: matchLabels: app: <app_name> template: metadata: labels: app: <app_name> spec: containers: - image: <image_name> name: <container_name> securityContext: runAsUser: <non_root_user> volumeMounts: - name: <volume_name> mountPath: /<file_path> volumes: - name: <volume_name> persistentVolumeClaim: claimName: <pvc_name>
app
- In the metadata section, enter label for the deployment.
matchLabels.app
andlabels.app
- In the spec selector and in the spec template metadata sections, enter a label for your app.
image
- The name of the container image that you want to use. To list available images in your IBM Cloud Container Registry account, run
ibmcloud cr image-list
. name
- The name of the container that you want to deploy to your cluster.
runAsUser
- In the spec containers security context section, you can optionally set the run as user value.
mountPath
- In the spec containers volume mounts section, enter the absolute path of the directory to where the volume is mounted inside the container. If you want to share a volume between different apps, you can specify volume sub paths for each of your apps.
volumeMounts.name
- In the spec containers volume mounts section, enter the name of the volume to mount to your pod.
volumes.name
- In the volumes section, enter the name of the volume to mount to your pod. Typically this name is the same as
volumeMounts/name
. claimName
- In the volumes persistent volume claim section, enter the name of the PVC that binds the PV that you want to use.
Creating a deployment
After you create PVC and deployment configuration files, create the deployment in your cluster.
-
Create the deployment.
oc apply -f <local_yaml_path>
-
Verify that the PV is successfully mounted.
oc describe deployment <deployment_name>
The mount point is in the Volume Mounts field and the volume is in the Volumes field.
Volume Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-tqp61 (ro) /volumemount from myvol (rw) ... Volumes: myvol: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: mypvc ReadOnly: false
-
Verify that you can write data to your IBM Cloud Object Storage service instance by logging in to the app pod and writing data. Log in to the pod that mounts your PV.
oc exec <pod_name> -it bash
-
Navigate to your volume mount path that you defined in your app deployment.
-
Create a text file.
echo "This is a test" > test.txt
-
From the IBM Cloud® dashboard, navigate to your IBM Cloud Object Storage service instance.
-
From the menu, select Buckets.
-
Open your bucket, and verify that you can see the
test.txt
that you created.
Using object storage in a stateful set
If you have a stateful app such as a database, you can create stateful sets that use IBM Cloud Object Storage to store your app's data. Alternatively, you can use an IBM Cloud database-as-a-service, such as IBM Cloudant and store your data in the cloud.
Before you begin:
- Create and prepare your IBM Cloud Object Storage service instance.
- Create a secret to store your IBM Cloud Object Storage service credentials.
- Decide on the configuration for your IBM Cloud Object Storage.
To deploy a stateful set that uses object storage:
-
Create a configuration file for your stateful set and the service that you use to expose the stateful set. The following examples show how to deploy NGINX as a stateful set with three replicas, each replica with a separate bucket or sharing the same bucket.
Example to create a stateful set with three replicas, with each replica using a separate bucket.
apiVersion: v1 kind: Service metadata: name: nginx-v01 namespace: default labels: app: nginx-v01 # must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx-v01 # must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web-v01 namespace: default spec: selector: matchLabels: app: nginx-v01 # must match spec.template.metadata.labels in stateful set YAML and metadata.labels in service YAML serviceName: "nginx-v01" replicas: 3 template: metadata: labels: app: nginx-v01 # must match spec.selector.matchLabels in stateful set YAML and metadata.labels in service YAML spec: terminationGracePeriodSeconds: 10 containers: - name: nginx image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web volumeMounts: - name: mypvc mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: mypvc annotations: ibm.io/auto-create-bucket: "true" ibm.io/auto-delete-bucket: "true" ibm.io/bucket: "" ibm.io/secret-name: mysecret volume.beta.kubernetes.io/storage-class: ibmc-s3fs-standard-perf-cross-region volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "ibmc-s3fs-standard-perf-cross-region" resources: requests: storage: 1Gi
Example to create a stateful set with three replicas that share the same bucket
mybucket
.apiVersion: v1 kind: Service metadata: name: nginx-v01 namespace: default labels: app: nginx-v01 # must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML spec: ports: - port: 80 name: web clusterIP: None selector: app: nginx-v01 # must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML --- apiVersion: apps/v1 kind: StatefulSet metadata: name: web-v01 namespace: default spec: selector: matchLabels: app: nginx-v01 # must match spec.template.metadata.labels in stateful set YAML and metadata.labels in service YAML serviceName: "nginx-v01" replicas: 3 template: metadata: labels: app: nginx-v01 # must match spec.selector.matchLabels in stateful set YAML and metadata.labels in service YAML spec: terminationGracePeriodSeconds: 10 containers: - name: nginx image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web volumeMounts: - name: mypvc mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: mypvc annotations: ibm.io/auto-create-bucket: "false" ibm.io/auto-delete-bucket: "false" ibm.io/bucket: mybucket ibm.io/secret-name: mysecret volume.beta.kubernetes.io/storage-class: ibmc-s3fs-standard-perf-cross-region volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs spec: accessModes: [ "ReadOnlyMany" ] storageClassName: "ibmc-s3fs-standard-perf-cross-region" resources: requests: storage: 1Gi
name
-
Enter a name for your stateful set. The name that you enter is used to create the name for your PVC in the format:
<volume_name>-< statefulset_name>-<replica_number>
. serviceName
-
Enter the name of the service that you want to use to expose your stateful set.
replicas
-
Enter the number of replicas for your stateful set.
matchLabels
-
In the spec selector match labels section, enter all labels that you want to include in your stateful set and your PVC. Labels that you include in the
volumeClaimTemplates
of your stateful set are not recognized by Kubernetes. Instead, you must define these labels in thespec.selector.matchLabels
andspec.template.metadata.labels
section of your stateful set YAML. To make sure that all your stateful set replicas are into the load balancing of your service, include the same label that you used in thespec.selector
section of your service YAML. labels
-
In the spec metadata labels section, enter the same labels that you added to the
spec.selector.matchLabels
section of your stateful set YAML. terminationGracePeriodSeconds
-
Enter the number of seconds to give the
kubelet
to safely terminate the pod that runs your stateful set replica. For more information, see Delete Pods. VolumeClaimTemplates.name
-
In the spec volume claim templates metadata section, enter a name for your volume. Use the same name that you defined in the
spec.containers.volumeMount.name
section. The name that you enter here is used to create the name for your PVC in the format:<volume_name>-<statefulset_name>-<replica_number>
. ibm.io/auto-create-bucket
-
In the spec volume claim templates metadata section, set an annotation to configure how buckets are created. Choose between the following options:
- true: Choose this option to automatically create a bucket for each stateful set replica. Note that the service credentials must have Writer permissions to automatically create the bucket.
- false: Choose this option if you want to share an existing bucket across your stateful set replicas. Make sure to define the name of the bucket in the
spec.volumeClaimTemplates.metadata.annotions.ibm.io/bucket
section of your stateful set YAML.
ibm.io/auto-delete-bucket
-
In the spec volume claim templates metadata section, set an annotation to configure how buckets are deleted. Choose between the following options:
- true: Your data, the bucket, and the PV is automatically removed when you delete the PVC. Your IBM Cloud Object Storage service instance remains and is not deleted. If you choose to set this option to true, then you must
set
ibm.io/auto-create-bucket: true
andibm.io/bucket: ""
so that your bucket is automatically created with a name with the formattmp-s3fs-xxxx
. - false: When you delete the PVC, the PV is deleted automatically, but your data and the bucket in your IBM Cloud Object Storage service instance remain. To access your data, you must create a new PVC with the name of your existing bucket.
- true: Your data, the bucket, and the PV is automatically removed when you delete the PVC. Your IBM Cloud Object Storage service instance remains and is not deleted. If you choose to set this option to true, then you must
set
ibm.io/bucket
-
In the spec volume claim templates metadata section, set an annotation for the bucket details. Choose between the following options:
- If
ibm.io/auto-create-bucket
is set to true: Enter the name of the bucket that you want to create in IBM Cloud Object Storage. If in additionibm.io/auto-delete-bucket
is set totrue
, you must leave this field blank to automatically assign your bucket a name with the formattmp-s3fs-xxxx
. The name must be unique in IBM Cloud Object Storage. - If
ibm.io/auto-create-bucket
is set to false: Enter the name of the existing bucket that you want to access in the cluster.
- If
ibm.io/secret-name
-
In the spec volume claim templates metadata annotations section, enter the name of the secret that holds the IBM Cloud Object Storage credentials that you created earlier. If you add your IBM Cloud Object Storage credentials to the default storage classes, you must not list the secret in the PVC.
kubernetes.io/storage-class
-
In the spec volume claim templates metadata annotations section, enter the storage class that you want to use. Choose between the following options:
- If
ibm.io/auto-create-bucket: "true"
: Enter the storage class that you want to use for your new bucket. - If
ibm.io/auto-create-bucket: "false"
: Enter the storage class that you used to create your existing bucket.
- If
-
To list existing storage classes, run
oc get sc | grep s3
. If you don't specify a storage class, the PVC is created with the default storage class that is set in your cluster. Make sure that the default storage class uses theibm.io/ibmc-s3fs
provisioner so that your stateful set is provisioned with object storage. storageClassName
-
In the spec volume claim templates spec section, enter the same storage class that you entered in the
spec.volumeClaimTemplates.metadata.annotations.volume.beta.kubernetes.io/storage-class
section of your stateful set YAML. storage
-
In the spec volume claim templates spec resource requests section, enter a fictitious size for your IBM Cloud Object Storage bucket in gigabytes. The size is required by Kubernetes, but not respected in IBM Cloud Object Storage. You can enter any size that you want. The actual space that you use in IBM Cloud Object Storage might be different and is billed based on the pricing table.