IBM Cloud storage utilities
Classic: Installing the IBM Cloud Block Storage Attacher plug-in (beta)
Use the IBM Cloud Block Storage Attacher plug-in to attach raw, unformatted, and unmounted block storage to a classic worker node in your cluster.
The IBM Cloud Block Storage Attacher plug-in is available for classic worker nodes only. If you want to attach raw, unformatted block storage to a VPC worker node, see Adding raw Block Storage for VPC worker nodes.
For example, you want to store your data with a software-defined storage solution (SDS), such as Portworx, but you don't want to use classic bare metal worker nodes that are optimized for SDS usage and that come with extra local disks. To add local disks to your classic non-SDS worker node, you must manually create your block storage devices in your IBM Cloud infrastructure account and use the IBM Cloud Block Volume Attacher to attach the storage to your non-SDS worker node.
The IBM Cloud Block Volume Attacher plug-in creates pods on every worker node in your cluster as part of a daemon set and sets up a Kubernetes storage class that you later use to attach the block storage device to your non-SDS worker node.
Looking for instructions for how to update or remove the IBM Cloud Block Volume Attacher plug-in? See Updating the plug-in and Removing the plug-in.
-
Follow the instructions to install the Helm client version 3 on your local machine.
-
Update the Helm repo to retrieve the latest version of all Helm charts in this repo.
helm repo update
-
Install the IBM Cloud Block Volume Attacher plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.
helm install block-attacher iks-charts/ibm-block-storage-attacher --namespace kube-system
Example output
NAME: block-volume-attacher LAST DEPLOYED: Thu Sep 13 22:48:18 2018 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1beta1/ClusterRoleBinding NAME AGE ibmcloud-block-storage-attacher 1s ==> v1beta1/DaemonSet NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE ibmcloud-block-storage-attacher 0 0 0 0 0 <none> 1s ==> v1/StorageClass NAME PROVISIONER AGE ibmc-block-attacher ibm.io/ibmc-blockattacher 1s ==> v1/ServiceAccount NAME SECRETS AGE ibmcloud-block-storage-attacher 1 1s ==> v1beta1/ClusterRole NAME AGE ibmcloud-block-storage-attacher 1s NOTES: Thank you for installing: ibmcloud-block-storage-attacher. Your release is named: block-volume-attacher Please refer Chart README.md file for attaching a block storage Please refer Chart RELEASE.md to see the release details/fixes
-
Verify that the IBM Cloud Block Volume Attacher daemon set is installed successfully.
oc get pod -n kube-system -o wide | grep attacher
Example output
ibmcloud-block-storage-attacher-z7cv6 1/1 Running 0 19m
The installation is successful when you see one or more ibmcloud-block-storage-attacher pods. The number of pods equals the number of worker nodes in your cluster. All pods must be in a Running state.
-
Verify that the storage class for the IBM Cloud Block Volume Attacher is created successfully.
oc get sc | grep attacher
Example output
ibmc-block-attacher ibm.io/ibmc-blockattacher 11m
Updating the IBM Cloud Block Storage Attacher plug-in
You can upgrade the existing IBM Cloud Block Storage Attacher plug-in to the latest version.
-
Update the Helm repo to retrieve the latest version of all helm charts in this repo.
helm repo update
-
Optional: Download the latest Helm chart to your local machine. Then, extract the package and review the
release.md
file to find the latest release information.helm pull iks-charts/ibmcloud-block-storage-plugin
-
Find the name of the Helm chart for the IBM Cloud Block Storage Attacher plug-in.
helm ls -A
Example output
<helm_chart_name> 1 Wed Aug 1 14:55:15 2022 DEPLOYED ibm-block-storage-attacher-1.0.0 default
-
Upgrade the IBM Cloud Block Storage Attacher to latest.
helm upgrade --force --recreate-pods <helm_chart_name> ibm-block-storage-attacher
Removing the IBM Cloud Block Volume Attacher plug-in
If you don't want to provision and use the IBM Cloud Block Storage Attacher plug-in in your cluster, you can uninstall the Helm chart.
-
Find the name of the Helm chart for the IBM Cloud Block Storage Attacher plug-in.
helm list | grep ibm-block-storage-attacher
Example output
<helm_chart_name> 1 Wed Aug 1 14:55:15 2022 DEPLOYED ibm-block-storage-attacher-1.0.0 default
-
Delete the IBM Cloud Block Storage Attacher plug-in by removing the Helm chart.
helm uninstall <helm_chart_name> -n <namespace>
-
Verify that the IBM Cloud Block Storage Attacher plug-in pods are removed.
oc get pod -n kube-system -o wide | grep attacher
The removal of the pods is successful if no pods are displayed in your CLI output.
-
Verify that the IBM Cloud Block Storage Attacher storage class is removed.
oc get sc | grep attacher
The removal of the storage class is successful if no storage class is displayed in your CLI output.
Classic: Manually adding block storage to specific worker nodes
Use this option if you want to add different block storage configurations, add block storage to a subset of worker nodes only, or to have more control over the provisioning process.
The instructions in this topic are available for classic worker nodes only. If you want to attach raw, unformatted block storage to a VPC worker node, see Adding raw Block Storage for VPC to worker nodes.
-
List the worker nodes in your cluster and note the private IP address and the zone of the non-SDS worker nodes where you want to add a block storage device.
ibmcloud oc worker ls --cluster <cluster_name_or_ID>
-
Review step 3 and 4 in Deciding on your block storage configuration to choose the type, size, and number of IOPS for the block storage device that you want to add to your non-SDS worker node.
-
Create the block storage device in the same zone that your non-SDS worker node is in.
Example for provisioning 20 GB endurance block storage with two IOPS per GB.
ibmcloud sl block volume-order --storage-type endurance --size 20 --tier 2 --os-type LINUX --datacenter dal10
Example for provisioning 20 GB performance block storage with 100 IOPS.
ibmcloud sl block volume-order --storage-type performance --size 20 --iops 100 --os-type LINUX --datacenter dal10
-
Verify that the block storage device is created and note the
id
of the volume. Note: If you don't see your block storage device immediately, wait a few minutes. Then, run this command again.ibmcloud sl block volume-list
Example output
id username datacenter storage_type capacity_gb bytes_used ip_addr lunId active_transactions 123456789 IBM02SL1234567-8 dal10 performance_block_storage 20 - 161.12.34.123 0 0
-
Review the details for your volume and note the
Target IP
andLUN Id
.ibmcloud sl block volume-detail <volume_ID>
Example output
NAME Value ID 1234567890 User name IBM123A4567890-1 Type performance_block_storage Capacity (GB) 20 LUN Id 0 IOPS 100 Datacenter dal10 Target IP 161.12.34.123 # of Active Transactions 0 Replicant Count 0
-
Authorize the non-SDS worker node to access the block storage device. Replace
<volume_ID>
with the volume ID of your block storage device that you retrieved earlier, and<private_worker_IP>
with the private IP address of the non-SDS worker node where you want to attach the device.ibmcloud sl block access-authorize <volume_ID> -p <private_worker_IP>
Example output
The IP address 123456789 was authorized to access <volume_ID>.
-
Verify that your non-SDS worker node is successfully authorized and note the
host_iqn
,username
, andpassword
.ibmcloud sl block access-list <volume_ID>
Example output
ID name type private_ip_address source_subnet host_iqn username password allowed_host_id 123456789 <private_worker_IP> IP <private_worker_IP> - iqn.2018-09.com.ibm:ibm02su1543159-i106288771 IBM02SU1543159-I106288771 R6lqLBj9al6e2lbp 1146581
The authorization is successful when the
host_iqn
,username
, andpassword
are assigned.
Classic: Attaching raw block storage to non-SDS worker nodes
To attach the block storage device to a non-SDS worker node, you must create a persistent volume (PV) with the IBM Cloud Block Volume Attacher storage class and the details of your block storage device.
The instructions in this topic are available for classic worker nodes only. If you want to attach raw, unformatted block storage to a VPC worker node, see Adding raw Block Storage for VPC to worker nodes.
- Make sure that you manually added raw, unformatted, and unmounted block storage to your non-SDS worker nodes.
- Access your Red Hat OpenShift cluster.
- Install the block storage attacher plug-in.
-
Prepare the PV creation.
-
If you used the
mkpvyaml
container, run the following command.-
Open the
pv-<cluster_name>.yaml
file.nano pv-<cluster_name>.yaml
-
Review the configuration for your PVs.
-
-
If you manually added block storage:**
-
Create a
pv.yaml
file. The following command creates the file with thenano
editor.nano pv.yaml
-
Add the details of your block storage device to the PV.
apiVersion: v1 kind: PersistentVolume metadata: name: <pv_name> annotations: ibm.io/iqn: "<IQN_hostname>" ibm.io/username: "<username>" ibm.io/password: "<password>" ibm.io/targetip: "<targetIP>" ibm.io/lunid: "<lunID>" ibm.io/nodeip: "<private_worker_IP>" ibm.io/volID: "<volume_ID>" spec: capacity: storage: <size> accessModes: - ReadWriteOnce hostPath: path: / storageClassName: ibmc-block-attacher
metadata.name
- Enter a name for your PV.
ibm.io/iqn
- Enter the IQN hostname that you retrieved earlier.
ibm.io/username
- Enter the IBM Cloud infrastructure username that you retrieved earlier.
ibm.io/password
- Enter the IBM Cloud infrastructure password that you retrieved earlier.
ibm.io/targetip
- Enter the target IP that you retrieved earlier.
ibm.io/lunid
- Enter the LUN ID of your block storage device that you retrieved earlier.
ibm.io/nodeip
- Enter the private IP address of the worker node where you want to attach the block storage device and that you authorized earlier to access your block storage device.
ibm.io/volID
- Enter the ID of the block storage volume that you retrieved earlier.
storage
- Enter the size of the block storage device that you created earlier. For example, if your block storage device is 20 gigabytes, enter
20Gi
.
-
-
-
Create the PV to attach the block storage device to your non-SDS worker node.
-
If you used the
mkpvyaml
container, run the following command.oc apply -f pv-<cluster_name>.yaml
-
If you manually added block storage, run the following command.
oc apply -f pv.yaml
-
-
Verify that the block storage is successfully attached to your worker node.
oc describe pv <pv_name>
Example output
NAME: kube-wdc07-cr398f790bc285496dbeb8e9137bc6409a-w1-pv1 Labels: <none> Annotations: ibm.io/attachstatus=attached ibm.io/dm=/dev/dm-1 ibm.io/iqn=iqn.2018-09.com.ibm:ibm02su1543159-i106288771 ibm.io/lunid=0 ibm.io/mpath=3600a09803830445455244c4a38754c66 ibm.io/nodeip=10.176.48.67 ibm.io/password=R6lqLBj9al6e2lbp ibm.io/targetip=161.26.98.114 ibm.io/username=IBM02SU1543159-I106288771 kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{"ibm.io/iqn":"iqn.2018-09.com.ibm:ibm02su1543159-i106288771","ibm.io/lunid":"0"... Finalizers: [] StorageClass: ibmc-block-attacher Status: Available Claim: Reclaim Policy: Retain Access Modes: RWO Capacity: 20Gi Node Affinity: <none> Message: Source: Type: HostPath (bare host directory volume) Path: / HostPathType: Events: <none>
The block storage device is successfully attached when the ibm.io/dm is set to a device ID, such as
/dev/dm/1
, and you can see ibm.io/attachstatus=attached in the Annotations section of your CLI output.
If you want to detach a volume, delete the PV. Detached volumes are still authorized to be accessed by a specific worker node and are attached again when you create a new PV with the IBM Cloud Block Volume Attacher storage class to attach a
different volume to the same worker node. To avoid attaching the old detached volume again, unauthorize the worker node to access the detached volume by using the ibmcloud sl block access-revoke
command. Detaching the volume does
not remove the volume from your IBM Cloud infrastructure account. To cancel the billing for your volume, you must manually remove the storage from your IBM Cloud infrastructure account.
VPC: Adding raw Block Storage for VPC to VPC worker nodes by using the API
You can use the Kubernetes Service API to attach and detach raw, unformatted Block Storage for Classic to a worker node in your VPC cluster.
You can attach a volume to one worker node only. Make sure that the volume is in the same zone as the worker node for the attachment to succeed.
You can also attach, detach, and list the volume attachments of your worker nodes by using the CLI. For more information, see the storage CLI reference.
The instructions in this topic are available for VPC worker nodes only. If you want to attach raw, unformatted block storage to a classic worker node, you must install the IBM Cloud Block Storage attacher plug-in.
Before you begin:
Access your Red Hat OpenShift cluster.
-
Check which region and zone your VPC worker node is in.
ibmcloud oc worker ls -c <cluster_name>
-
Decide on the Block Storage for Classic profile that best meets the capacity and performance requirements that you have.
-
Provision a Block Storage for Classic volume. The volume that you provision must be in the same resource group, region, and zone as the worker node.
-
Retrieve your IAM token.
ibmcloud iam oauth-tokens
-
Retrieve the ID of the worker node that you want to attach to the Block Storage for Classic instance. Make sure to select a worker node that is located in the same zone as your Block Storage for Classic volume.
ibmcloud oc worker ls --cluster <cluster_name_or_ID>
-
Use a
POST
request to attach your Block Storage for Classic volume to the worker node.Example request
curl -X POST "https://containers.cloud.ibm.com/global/v2/storage/createAttachment" -H "accept: application/json" -H "Authorization: <IAM_token>" -H "X-Auth-Resource-Group-ID: <resource_group>" -H "Content-Type: application/json" -d "{ \"cluster\": \"<cluster_name_or_ID>\", \"volumeID\": \"<volume_ID>\", \"worker\": \"<worker_ID>\"}"
IAM_token
- The IAM OAuth token for your current session. You can retrieve this value by running
ibmcloud iam oauth-tokens
. cluster_name_or_ID
- The unique ID or the name that is assigned to your cluster. You can retrieve this ID by running
ibmcloud oc cluster ls
. worker_ID
- The unique ID that is assigned to the worker node where you want to attach your volume. You can retrieve this value by running
ibmcloud oc worker ls -c <cluster_name>
. volume_ID
- The unique ID that is assigned to your Block Storage for Classic volume. You can retrieve a list of your Block Storage for Classic volumes by running
ibmcloud is volumes
.
Example response.
{ "id": "0111-1aaa11a1-aa1a-111a-111b-1111a1dad1bc", "volume": { "name": "my-vol", "id": "r001-11aa0d59-a1aa-1a11-11ca-ba2bc11e01aa" }, "device": { "id": "" }, "name": "volume-attachment", "status": "attaching", "type": "data" }
-
Verify the attachment by reviewing existing volume attachments for a VPC worker node.
Detaching raw and unformatted Block Storage for Classic from a worker node in a VPC cluster by using the API
You can use a DELETE
request to detach storage from a VPC worker node.
Detaching storage from your VPC cluster does not remove your Block Storage for Classic volume or the data that is stored in the volume. You continue to get billed until you manually delete the volume.
-
Identify the storage volume that you want to remove and note the volume ID.
ibmcloud is volumes
-
Get details about the volume. This command returns the worker node ID and attachment ID. Note the worker node ID. In the following command this ID is returned as "Instance name".
ibmcloud is volume <volume_ID>
-
Retrieve a list of your PVs. This command returns a list of your PVs that you can then you use to determine which PVC uses the volume that you want to remove.
oc get pv
-
Describe the PV that uses the volume. If you don't know which PV uses the volume that you want to remove, you can run the
describe pv
command on each PV in your cluster. Note the PVC that uses the PV.oc describe pv <pv_name>
-
Check to see if your storage volume is in use by a pod. The following command shows the pods that mount the volume and the associated PVC. If no pod is returned, the storage is not in use.
oc get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
-
If the pod your volume is using is part of a deployment, delete the deployment. If your pod does not belong to a deployment, delete the pod.
oc delete deployment <deployment_name>
oc delete pod <pod_name>
-
Delete the PVC and PV.
oc delete pvc <pvc_name>
oc delete pv <pv_name>
-
Retrieve your IAM token.
ibmcloud iam oauth-tokens
-
Detach storage by using a
POST
request.Example request
curl -X POST "https://containers.cloud.ibm.com/global/v2/storage/deleteAttachment" -H "accept: application/json" -H "Authorization: <IAM_token>" -H "X-Auth-Resource-Group-ID: <resource_group>" -H "Content-Type: application/json" -d "{ \"cluster\": \"<cluster_name_or_ID\", \"volumeAttachmentID\": \"<volume_attachment_ID>\", \"volumeID\": \"<volume_ID>\", \"worker\": \"<worker_ID>\"}"
IAM_token
- The IAM OAuth token for your current session. You can retrieve this value by running
ibmcloud iam oauth-tokens
. cluster_name_or_ID
- The unique ID or name that is assigned to your cluster. You can retrieve this ID by running
ibmcloud oc cluster ls
. worker_ID
- The unique ID that is assigned to the worker node where you want to attach your volume. You can retrieve this value by running
ibmcloud oc worker ls -c <cluster_name>
. volume_ID
- The unique ID that is assigned to your Block Storage for Classic volume. You can retrieve a list of your Block Storage for Classic volumes by running
ibmcloud is volumes
. volume_attachment_ID
- The unique ID that is assigned to your volume attachment. You can retrieve this ID by running
ibmcloud is volume <volume_ID>
.
Reviewing volume attachment details for a VPC worker node by using the API
You can use a GET
request to retrieve volume attachment details for a VPC worker node.
-
Retrieve your IAM token.
ibmcloud iam oauth-tokens
-
Retrieve the ID of the resource group where your cluster is deployed.
ibmcloud oc cluster get <cluster_name_or_ID> | grep "Resource Group ID"
-
Retrieve the ID of the worker node for which you want to see volume attachment details. Make sure to select a worker node that is located in the same zone as your Block Storage for Classic instance.
ibmcloud oc worker ls --cluster <cluster_name_or_ID>
-
Review a list of existing volume attachments on a worker node.
curl -X GET "https://containers.cloud.ibm.com/v2/storage/getAttachments?cluster=<cluster_ID>&worker=<worker_ID>" --header "X-Auth-Resource-Group-ID: <resource_group_ID>" --header "Authorization: <IAM_token>"
-
Retrieve the details for a specific attachment.
curl -X GET "https://containers.cloud.ibm.com/v2/storage/getAttachment?cluster=<cluster_ID>&worker=<worker_ID>&volumeAttachmentID=<volume_attachment_ID>" --header "X-Auth-Resource-Group-ID: <resource_group_ID>" --header "Authorization: <IAM_token>"
IAM_token
- The IAM OAuth token for your current session. You can retrieve this value by running
ibmcloud iam oauth-tokens
. cluster_ID
- The unique ID that is assigned to your cluster. You can retrieve this ID by running
ibmcloud oc cluster ls
. worker_ID
- The unique ID that is assigned to the worker node where you want to attach your volume. You can retrieve this value by running
ibmcloud oc worker ls -c <cluster_name>
. volume_ID
- The unique ID that is assigned to your Block Storage for Classic volume. You can retrieve a list of your Block Storage for Classic volumes by running
ibmcloud is volumes
. volume_attachment_ID
- The unique ID that is assigned to your volume attachment. You can retrieve this ID by running
ibmcloud is volume <volume_ID>
.
VPC: Attaching raw Block Storage for VPC to VPC worker nodes by using the CLI
You can use the Kubernetes Service CLI to attach and detach raw, unformatted Block Storage for Classic to a worker node in your VPC cluster.
You can attach a volume to one worker node only. Make sure that the volume is in the same zone as the worker node for the attachment to succeed.
The instructions in this topic are available for VPC worker nodes only. If you want to attach raw, unformatted block storage to a classic worker node, you must install the IBM Cloud Block Storage attacher plug-in.
Before you begin:
Access your Red Hat OpenShift cluster.
-
List your storage volumes and note the ID of the volume that you want to attach.
ibmcloud is vols
-
List the worker nodes in your cluster and note the ID of the worker node where you want to attach your volume.
ibmcloud oc worker ls -c <cluster_name_or_ID>
-
Attach your Block Storage for Classic to your VPC worker node.
ibmcloud oc storage attachment create --cluster <cluster_name_or_ID> --volume <volume> --worker <worker_ID>
Removing raw Block Storage for VPC from VPC worker nodes by using the CLI
You can remove storage from your worker node by using the ibmcloud oc storage attachment rm
command.
-
List your storage volumes and note the ID of the volume that you want to remove.
ibmcloud is vols
-
Get the details of your volume such as the
worker-id
where the volume is attached. Theworker-id
is listed as the Instance name in the Volume Attachment Instance Reference section of the command output.ibmcloud is vol <volume-ID>
Example output
Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name data 0727_e18c10d7-7f18-48aa-b5ef-5ed163e54198 kube-bsaucubd07dhl66e4tgg-cluster-default-00000a19 false 0727-3bfe90b0-dc2d-498a-946b-8837a5dad7bc volume-attachment
-
List the storage attachments on a worker node in your cluster and make a note of the attachment ID that your want to remove.
ibmcloud oc storage attachment ls -c <cluster> --worker <worker-id>
Example output:
Listing volume attachments... OK ID Name Status Type Volume ID Volume Name Worker ID 0111-1a111aaa-1111-1111-111a-aaa1a1a11a11 volume-attachment attached boot a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111 dissuade-anointer-errand-handbrake kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
-
Remove the storage attachment.
ibmcloud oc storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
-
Verify that your storage was removed from the worker node.
ibmcloud oc storage attachment ls -c <cluster-ID> --worker <worker-id>
Backing up and restoring PVC data for file and block storage
The back up and restore Helm chart is not supported on Satellite version 4.13 clusters that use CoreOS workers.
With the IBM Cloud Backup Restore Helm chart, you can create a one-time or scheduled backup for data that is stored in a file storage or block storage persistent volume claim (PVC). Your data is stored in an IBM Cloud Object Storage service instance that you create and own. You can use existing backups in your IBM Cloud Object Storage service instance to restore data to a PVC in your cluster.
- What happens when I install the Helm chart?
- When you install the Helm chart, a Kubernetes pod is created in your cluster that performs a one-time or periodic backup of your PVC data, or restores data from IBM Cloud Object Storage to a PVC. You configure your backup or restore in the
values.yaml
file that is provided with the Helm chart or by setting options in thehelm install
command. - What limitations do I need to be aware of?
- If you want to back up or restore data of a block storage PVC, your PVC must not be mounted to an app. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up or restore your data, you must remove the pod that uses the storage to unmount the PVC. After the backup or restoring of data is finished, you can re-create your pod and mount the backed up or restored PVC.
- What do I need before I get started?
- To back up or restore data to IBM Cloud Object Storage, you must set up an IBM Cloud Object Storage service instance, create service credentials to access the service, and create a bucket that can hold your data.
Setting up an IBM Cloud Object Storage service instance
Create and configure an IBM Cloud Object Storage service instance to serve as the repository for the data that you want to back up.
- Create an IBM Cloud Object Storage service instance that uses HMAC credentials.
- Store your IBM Cloud Object Storage credentials in a Kubernetes secret.
- Create your first IBM Cloud Object Storage bucket.
- In the navigation on the service details page, click Buckets.
- Click Create bucket. A dialog box is displayed.
- Enter a unique name for your bucket. The name must be unique within IBM Cloud Object Storage across all regions and across all IBM Cloud accounts.
- From the Resiliency list, select the level of availability that you want for your data. For more information, see IBM Cloud Object Storage regions and endpoints.
For VPC clusters, make a note of the direct endpoint. For example:
s3.direct.us.cloud-object-storage.appdomain.cloud
. - Change the Location to the region where you want to store your data. Keep in mind that your data might not be allowed to be stored in every region due to legal reasons.
- Click Create.
- Retrieve the IBM Cloud Object Storage host name for your bucket.
- Click on your bucket name that you created in the previous step.
- In the navigation on the service details page, click Buckets > Configuration.
- Note the public URL that you can use to access the data in your bucket.
For more information about configuring your service instance, see the IBM Cloud Object Storage documentation.
Using IBM Cloud Object Storage to back up and restore PVC data
You can use the IBM Cloud Backup Restore Helm chart to back up data in a file storage or block storage PVC to IBM Cloud Object Storage, or restore data from IBM Cloud Object Storage to a PVC in your cluster.
Before you begin:
-
Make sure that you have a PVC that you can back up or restore data to. For more information about how to create a PVC, see Adding file storage to apps and Adding block storage to apps.
-
If you want to back up a block storage PVC, make sure that your PVC is not mounted to an app. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up your data, you must remove the app pod that mounts the storage. To check whether a pod is mounted to your PVC, run the following command.
oc get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{" "}{end}{end}' | grep "<pvc_name>"
-
Retrieve your IBM Cloud Object Storage service credentials, the bucket name, and the bucket hostname.
-
Follow the instructions to install the Helm client on your local machine and set up the IBM Cloud Helm chart repositories.
You can deploy the ibm-storage-backup
pod or the ibm-storage-restore
pod by either editing and applying the values.yaml
file of the Helm chart, or by running the helm install
command from
the CLI.
To back up or restore a PVC by editing the values.yaml
file:
-
Download the latest Helm chart version to your local machine.
helm fetch --untar iks-charts/ibmcloud-backup-restore
-
Open the
values.yaml
file in thenano
command line editor.nano ibmcloud-backup-restore/values.yaml
-
Configure your Helm chart to back up or restore PVC data. You can configure the backup for more than one PVC.
Example to create a backup pod by configuring the
values.yaml
file:image: repository: icr.io/iks-charts/ibmcloud-backup-restore pullPolicy: Always tag: latest ACCESS_KEY_ID: # Example: 10110abab1111bbb111aa1aaa111b1a1 SECRET_ACCESS_KEY: # Example: a1aba11aaa11b11b11aa1111a1111ba111111111a0b1b11a ENDPOINT: # Example: s3.us-east.cloud-object-storage.appdomain.cloud BUCKET_NAME: # Example: my-bucket BACKUP_NAME: # Example: my_backup PVC_NAMES: - # Example: my_pvc - # Optional example: my_pvc2 CHART_TYPE: # Example: backup or restore BACKUP_TYPE: # Example: incremental SCHEDULE_TYPE: # Example: periodic SCHEDULE_INFO: # Example: weekly
ACCESS_KEY_ID
- Enter the access key ID of the IBM Cloud Object Storage service credentials that you retrieved earlier.
SECRET_ACCESS_KEY
- Enter the secret access key of the IBM Cloud Object Storage service credentials that you retrieved earlier.
ENDPOINT
- Enter the public IBM Cloud Object Storage s3 API endpoint for your bucket that you retrieved earlier.
BUCKET_NAME
- Backup: Enter the name of the IBM Cloud Object Storage bucket that you created earlier. You use this bucket to store PVC data when you perform a backup. Restore: Enter the name of the IBM Cloud Object Storage bucket where your backup is stored.
BACKUP_NAME
- Backup: Enter the name of the backup that you want to create in IBM Cloud Object Storage. Restore: Enter the name of the backup that you created with the IBM Cloud Backup Restore Helm chart in IBM Cloud Object Storage. If you have multiple full backups in your IBM Cloud Object Storage service instance, the PVC is restored with the data of the last full backup. If you have incremental backups, the PVC is restored with the data of the last full backup, including all incremental backups up to the day where you start the restore.
PVC_NAMES
- Backup: Enter the name of the PVC that you want to back up. If you want to back up multiple PVCs, add each PVC to the list of PVCs. To list available PVCs in your cluster that you can back up, run
oc get pvc
. Restore: Enter the name of the PVC to which you want to restore data from IBM Cloud Object Storage. You can restore data to one PVC at a time only. To list available PVCs in your cluster that you can restore data to, runoc get pvc
. CHART_TYPE
- Enter the name of the chart type that you want to deploy. Enter
backup
to deploy the backup chart. Enterrestore
to deploy the restore chart. BACKUP_TYPE
- Required only for backups. Enter
full
to create a full backup, orincremental
if you want to back up only new or changed files. If you chooseincremental
, you must specify theSCHEDULING_INFO
andSCHEDULING_TYPE
option. If you don't specify theBACKUP_TYPE
option, a full backup is created by default. SCHEDULE_TYPE
- Required only for backups. Enter
periodic
to create scheduled backups, or leave this option empty to create a one-time backup. If you want to create periodic backups, you must define the backup interval in theSCHEDULE_INFO
option. SCHEDULE_INFO
- Required only for backups. If you want to create periodic backups, you must decide on the backup schedule. Choose between
hourly
,daily
, orweekly
. If you set this option, you must setSCHEDULE_TYPE
toperiodic
.
-
Save and close the
values.yaml
file. -
Install the Helm chart with your custom settings in the
values.yaml
file. When you install the Helm chart and you configure a backup or restore, anibm-storage-backup
or anibm-storage-restore
pod is deployed to your cluster. The backup pod backs up the data from your PVC to IBM Cloud Object Storage and the restore pod restores data to a PVC. Replace<release_name>
with a name for your Helm chart. Be sure to install the backup and restore pods in the same zone as the PVC that you want to back up or restore.-
Install the Helm chart by using the
helm install
command.helm install <release_name> ./ibmcloud-backup-restore -n <namespace>
Example output for backup:
NAME: <release_name> LAST DEPLOYED: Mon Jan 20 09:17:02 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing: ibmcloud-backup-restore. Your release is named: <release_name> Please refer Chart README.md file for creating a sample PVC Please refer Chart RELEASE.md to see the release details/fixes
-
Optional: Install the Helm chart by setting options in the
helm install
command. You can name your release by specifying the--name
parameter.helm install <release_name> --set ACCESS_KEY_ID=<access_key_ID> --set SECRET_ACCESS_KEY=<secret_access_key> --set ENDPOINT=<public_bucket_endpoint> --set BUCKET_NAME=<bucket_name> --set BACKUP_NAME=<backup_name> --set PVC_NAMES[0]=<pvc_name1> --set PVC_NAMES[1]=<pvc_name2> --set CHART_TYPE=backup --set BACKUP_TYPE=<backup_type> --set SCHEDULE_TYPE=<schedule_type> --set SCHEDULE_INFO=<schedule_info> ./ibmcloud-backup-restore
-
-
Verify that your data backup or restore completed successfully.
Backup:
-
Verify that the
ibm-storage-backup
pod has a status of Running.oc get pods -A | grep backup
Example output
ibm-storage-backup 1/1 Running 0 64m
-
Review the logs of the
ibm-storage-backup
pod to make sure that your backup was successful. When you see the... backup completed
message in the event logs, your backup completed successfully.oc logs ibm-storage-backup
Example output for daily backups.
[2019-04-18 16:01:51,157] [utilities : 151] [INFO] *****************Start logging to ./Backup.log [2019-04-18 16:01:51,158] [backup : 48] [INFO] Starting backup: [2019-04-18 16:01:51,158] [configureOS : 66] [INFO] Configuring duplicity with IBM CloudObjectStorage i.e s3. [2019-04-18 16:01:51,158] [backup : 62] [INFO] Configuration done!!! [2019-04-18 16:01:51,158] [backup : 78] [INFO] Got all required input from config file!! [2019-04-18 16:01:52,366] [backup : 119] [WARNING] Incremental backup was not created [2019-04-18 16:01:52,366] [backup : 120] [INFO] duplicity --no-encryption incremental /myvol s3://s3.us-south.cloud-object-storage.appdomain.cloud/mybucket/helm-backup command failed due to Fatal Error: Unable to start incremental backup. Old signatures not found and incremental specified [2019-04-18 16:01:52,367] [backup : 121] [INFO] A full backup is required before incremental backups can begin. Creating a one-time full backup and will run incremental backups for scheduled backups. [2019-04-18 16:01:54,357] [backup : 129] [INFO] Full backup completed [2019-04-18 16:01:54,357] [backup : 130] [INFO] Local and Remote metadata are synchronized, no sync needed. Last full backup date: none --------------[ Backup Statistics ]-------------- StartTime 1555603313.31 (Thu Apr 18 16:01:53 2019) EndTime 1555603313.32 (Thu Apr 18 16:01:53 2019) ElapsedTime 0.01 (0.01 seconds) SourceFiles 3 SourceFileSize 20495 (20.0 KB) NewFiles 3 NewFileSize 20495 (20.0 KB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 3 RawDeltaSize 15 (15 bytes) TotalDestinationSizeChange 183 (183 bytes) Errors 0 ------------------------------------------------- [2019-04-18 16:01:54,357] [backup : 162] [INFO] Scheduling backup as per configurations, please don't stop this program or run this in background !!! [2019-04-18 16:01:54,358] [backup : 166] [INFO] Schedule info is: ['daily'] [2019-04-18 16:01:54,358] [backup : 172] [INFO] Scheduled for daily!!!
-
-
Verify that your data is successfully backed up or restored.
-
Backup:
- Find your IBM Cloud Object Storage service instance in the IBM Cloud resource list.
- From the navigation, select Buckets and click on the bucket that you used in your backup configuration. Your backup is displayed as an object in your bucket.
- Review the compressed files. You can download the
*.gz
file, extract the file, and verify the backed-up data.
-
Restore:
-
Create a
deployment.yaml
file with a pod that mounts the PVC that contains your restored data. The following example deploys annginx
pod that mounts the PVC on the/test
mount directory.apiVersion: apps/v1 kind: Deployment metadata: name: restore labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx volumeMounts: - name: my-volume # Example: my_volume mountPath: /test # Example: /test volumes: - name: my-volume # Example: my_volume persistentVolumeClaim: claimName: my-actual-pvc-name # Example: my_pvc
spec.containers.image
- The name of the image that you want to use. To list available images in your IBM Cloud Container Registry account, run
ibmcloud cr image-list
. spec.containers.name
- The name of the container that you want to deploy to your cluster.
spec.containers.volumeMounts.mountPath
- The absolute path of the directory to where the volume is mounted inside the container. Data that is written to the mount path is stored under the root directory in your physical block storage instance. If you want to share a volume between different apps, you can specify volume sub paths for each of your apps.
spec.containers.volumeMounts.name
- The name of the volume to mount to your pod.
volumes.name
- The name of the volume to mount to your pod. Typically this name is the same as
volumeMounts/name
volumes.persistentVolumeClaim.claimName
- The name of the PVC that binds the PV that you want to use.
-
Create the deployment.
oc apply -f deployment.yaml
-
Verify that your pod has a status of Running.
If you find that your
ibm-storage-restore
pod does not reach a Completed or CrashLoopBackOff status, restoring your data might have failed. Runoc logs ibm-storage-restore
to find the root cause for the failure.oc get pods | grep restore
Example output
restore-7dfc6f4c78-wkcqp 1/1 Running 0 3m54s
-
Log in to your pod.
oc exec <pod_name> -it bash
-
Navigate to the mount directory that you specified in your deployment YAML.
cd <mount_directory>
-
List the files in your mount directory to verify that all your data is restored to the mount directory.
ls
-
Delete the Helm chart installation from your cluster. This step is required if you restored data to a block storage PVC. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. Because the
ibm-storage-restore
pod already mounts the PVC, you must remove the pod to release the PVC so that you can mount the PVC to a different pod in your cluster.helm uninstall <release_name> -n <namespace>
-
You successfully restored your backup. You can now mount the PVC that binds the PV to any other pod in your cluster to access the restored files. If the container data that was backed up included a non-root user, you must add non-root permissions to your new container. For more information, see Adding non-root user access to volumes.
-
-
Setting up IBM Cloud Monitoring for storage volumes
Set up alerts in IBM Cloud Monitoring for your workloads that are using storage volumes. For more information see, Alerts.
When a storage volume is down, your app pods that are using storage have a low file system I/O, have network errors, or crash which causes the replica count to go down. You can set up alerts in IBM Cloud Monitoring to get notified if the file
system operations for your app drop under a specific threshold, if network errors occur, or if your app pods don't reach a Ready
state.
-
From the Red Hat OpenShift clusters console, select the cluster where you want to set up alerts for your storage volumes.
-
In the Monitoring section, click Connect to connect an existing IBM Cloud Monitoring instance to your cluster. If you don't have an instance, click Create an instance to create one. For more information about how to set up an IBM Cloud Monitoring instance, see Provisioning an instance.
-
Click the Launch button to open the IBM Cloud Monitoring dashboard.
-
Create file system utilization alerts for an app that runs in your cluster.
- From the IBM Cloud Monitoring console, click Overview > Workloads.
- Select the Namespace where your app is deployed. Find your app, click the arrow icon on your app, and select Kubernetes Pod overview.
- In the File System Utilization section, review the File I/O Bandwidth by Pod tile.
- Review the file I/O bandwidth from the last day or week time window to determine the average bandwidth. You can use the average bandwidth as a threshold to set an alert for when the file I/O bandwidth is less than the average for a certain amount of time. For example, if the average file I/O bandwidth for your app is 300B/s, you can create an alert for when the network utilization remains less than 300B/s for a certain amount of time.
- In the File I/O Bandwidth by Pod tile, create an alert by clicking the Options menu and then Create alert.
- Open the Notification section of the alert menu and create or select an alert notification channel.
- Save your alert.
- Repeat these steps for all the apps that are deployed in your cluster.
- Test the alert that you created by editing the threshold that you configured to manually trigger the alert. For example, if you set a file system utilization alert to trigger when the utilization is less than 300B/s for 5 minutes, increase
the threshold to be greater than the current utilization of your app for 5 minutes and select the
at least once
option. - Verify that the alert is triggered after 5 minutes. Once you have verified the alert, reset the values to the values you configured earlier.
-
Create a network utilization alert for an app that runs in your cluster.
- From the IBM Cloud Monitoring console, click Overview > Workloads.
- Select the Namespace where your app is deployed. Find your app, click the arrow icon on your app, and select Kubernetes Pod overview.
- In the Network Utilization section, review the Network Request Count by Pod tile.
- Review the average network request count by pod from the last 1 day or 1 week time window to determine the threshold.
- In the Network Request Count by Pod tile, create an alert by clicking the Options menu and then Create alert. Set the alert parameters based on the threshold that you observed. For example, if the network utilization remains under the threshold for a certain amount of time, then the alert is triggered.
- Open the Notification section of the alert menu and create or select an alert notification channel.
- Save your alert.
- Repeat these steps for all the apps that are deployed in your cluster.
- Test the alert that you created by editing the threshold that you configured to manually trigger the alert. For example, if you set an alert for when the number of network requests by pod is less than 3/s for 5 minutes, edit your alert
threshold to be less than the threshold that you observed for 5 minutes and select the
at least once
option. - Verify that the alert is triggered after 5 minutes. Once you have verified the alert, reset the values to the values you configured earlier.
-
Create a pod availability alert for an app that runs in your cluster.
- From the IBM Cloud Monitoring console, click Overview > Workloads.
- Select the Namespace where your app is deployed. Find your app, click the arrow icon on your app, and select Kubernetes Pod overview.
- In the Pod Health section, review the Pod Availability tile.
- Review the average pod availability from the last 1 day or 1 week time window to determine the threshold.
- Review your app configuration file for the number of requested replicas. You can use this number as a threshold for sending alerts when the number of available pods is fewer than your requested number of replicas for your app. If you requested 3 replicas of your app, you can set an alert for when the number of available pods stays fewer than the requested replica count for a certain time duration, for example, a 30 minute duration. In this example, when the pod availability remains fewer than the 3 requested replicas, the alert is triggered.
- In the Pod Availability tile, create an alert by clicking the Options menu and then Create alert. Set the alert parameters based on the threshold that you observed and your requested app replicas.
- Open the Notification section of the alert menu and create or select an alert notification channel.
- Save your alert.
- Repeat these steps for all the apps that are deployed in your cluster.
- Test the alert that you created by editing the threshold that you configured to manually trigger the alert. For example, if you set a pod availability alert for when the number of available pods is less than 3 for 5 minutes, edit your
alert threshold to be less than 4 pods for 5 minutes and select the
at least once
option. - Verify that the alert is triggered after 5 minutes. Once you have verified the alert, reset the values to the values you configured earlier.
Troubleshooting persistent storage when an IBM Cloud Monitoring alert is triggered
When an alert is triggered, review the alert details in IBM Cloud Monitoring and review troubleshooting guides for persistent storage, apps, worker nodes, and clusters to find the root cause of the alert. The alerts that you set up might not be related to a storage volume issue, but to issues that occurred within your app, on the worker node, or in the cluster.