Setting up Block Storage for VPC

Block Storage for VPC provides hypervisor-mounted, high-performance data storage for your virtual server instances that you can provision within a VPC.

You can choose between predefined storage tiers with GB sizes and IOPS that meet the requirements of your workloads. To find out if Block Storage for VPC is the right storage option for you, see Choosing a storage solution. For pricing information, see Pricing for Block Storage for VPC.

The Block Storage for VPC cluster add-on is enabled by default on VPC clusters.

Quick start for IBM Cloud Block Storage for VPC

In this quick start guide, you create a 10Gi 5IOPS tier Block Storage for VPC volume in your cluster by creating a PVC to dynamically provision the volume. Then, you create an app deployment that mounts your PVC.

Your Block Storage for VPC volumes can be mounted by multiple pods as long as those pods are scheduled on the same node.

Create a file for your PVC and name it pvc.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: ibmc-vpc-block-5iops-tier
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Create the PVC in your cluster.
```
kubectl apply -f pvc.yaml
```

After your PVC is bound, create an app deployment that uses your PVC. Create a file for your deployment and name it deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
    name: my-deployment
    labels:
      app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: ngnix # Your containerized app image.
        name: my-container
        volumeMounts:
        - name: my-volume
          mountPath: /mount-path
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-pvc

Create the deployment in your cluster.
```
kubectl apply -f deployment.yaml
```

For more information, see the following links.

Adding Block Storage for VPC to apps.
Storage class reference.
Customizing the default storage settings.

Adding Block Storage for VPC to your apps

Choose your Block Storage for VPC profile and create a persistent volume claim to dynamically provision Block Storage for VPC for your cluster. Dynamic provisioning automatically creates the matching persistent volume and orders the physical storage device in your IBM Cloud account.

Decide on the Block Storage for VPC profile that best meets the capacity and performance requirements that you want.
Select the corresponding storage class for your Block Storage for VPC profile.

All IBM pre-defined storage classes set up Block Storage for VPC with an ext4 file system by default. If you want to use a different file system, such as xfs or ext3, create a customized storage class.
- 10 IOPS/GB: ibmc-vpc-block-10iops-tier or ibmc-vpc-block-retain-10iops-tier
- 5 IOPS/GB: ibmc-vpc-block-5iops-tier or ibmc-vpc-block-retain-5iops-tier
- 3 IOPS/GB: ibmc-vpc-block-general-purpose or ibmc-vpc-block-retain-general-purpose
- Custom: ibmc-vpc-block-custom or ibmc-vpc-block-retain-custom
Decide on your Block Storage for VPC configuration.
1. Choose a size for your storage. Make sure that the size is supported by the Block Storage for VPC profile that you chose.
2. Choose if you want to keep your data after the cluster or the persistent volume claim (PVC) is deleted.
  - If you want to keep your data, then choose a retain storage class. When you delete the PVC, only the PVC is deleted. The persistent volume (PV), the physical storage device in your IBM Cloud account, and your data still exist. To reclaim the storage and use it in your cluster again, you must remove the PV and follow the steps for using existing Block Storage for VPC.
  - If you want the PV, the data, and your physical Block Storage for VPC device to be deleted when you delete the PVC, choose a storage class without retain.

Create a configuration file to define your persistent volume claim and save the configuration as a YAML file.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <pvc_name> # Enter a name for your PVC.
spec:
  accessModes:
  - <access-mode> # ReadWriteOnce or ReadWriteOncePod
  resources:
    requests:
      storage: 10Gi # Enter the size. Make sure that the size is supported in the profile that you chose.
  storageClassName: <storage_class> # Enter the storage class name that you selected earlier.

Create the PVC in your cluster.
```
kubectl apply -f pvc.yaml
```

Verify that your PVC is created and bound to the PV. This process can take a few minutes.

kubectl describe pvc <pvc_name>

Example output

Name:          mypvv
Namespace:     default
StorageClass:  ibmc-vpc-block-5iops-tier
Status:        Bound
Volume:        
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"csi-block-pvc-good","namespace":"default"},"spec":{...
            volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity: 10Gi   
Access Modes:  
VolumeMode:    Filesystem
Events:
    Type       Reason                Age               From                         Message
    ----       ------                ----              ----                         -------
    Normal     ExternalProvisioning  9s (x3 over 18s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "vpc.block.csi.ibm.io" or manually created by system administrator
Mounted By:  <none>

Create a deployment configuration file for your app and mount the PVC to your app.
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <deployment_name>
  labels:
    app: <deployment_label>
spec:
  selector:
    matchLabels:
      app: <app_name>
  template:
    metadata:
      labels:
        app: <app_name>
    spec:
      containers:
      - image: <image_name>
        name: <container_name>
        volumeMounts:
        - name: <volume_name>
          mountPath: /<file_path>
      volumes:
      - name: <volume_name>
        persistentVolumeClaim:
          claimName: <pvc_name>
```
labels.app

In the metadata section, enter a label for the deployment.

matchLabels.app and labels.app

In the spec selector and template metadata sections, enter a label for your app.

image

Specify the name of the container image that you want to use. To list available images in your IBM Cloud Container Registry account, run ibmcloud cr image-list.

name

Specify the name of the container that you want to deploy in your pod.

mountPath

In the container volume mounts section, specify the absolute path of the directory to where the PVC is mounted inside the container.

name

In the container volume mounts section, enter the name of the volume to mount to your pod. You can enter any name that you want.

name

In the volumes section, enter the name of the volume to mount to your pod. Typically this name is the same as volumeMounts.name.

claimName

In the volumes persistent volume claim section, enter the name of the PVC that you created earlier.
Create the deployment in your cluster.
```
kubectl apply -f deployment.yaml
```
Verify that the PVC is successfully mounted to your app. It might take a few minutes for your pods to get into a Running state.

During the deployment of your app, you might see intermittent Unable to mount volumes errors in the Events section of your CLI output. The Block Storage for VPC cluster add-on automatically retries mounting the storage to your apps. Wait a few more minutes for the storage to mount to your app.
```
kubectl describe deployment <deployment_name>
```
Example output
```
...
Volumes:
myvol:
    Type:    PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:    mypvc
    ReadOnly:    false
```

Using an existing Block Storage for VPC instance

If you have an existing physical Block Storage for VPC device that you want to use in your cluster, you can manually create the PV and PVC to statically provision the storage.

You can attach a volume to one worker node only. Make sure that the volume is in the same zone as the worker node for the attachment to succeed.

Determine the volume that you want to attach to a worker node in your VPC cluster. Note the volume ID.
```
ibmcloud is volumes
```
List the details of your volume. Note the Size, Zone, and IOPS. These values are used to create your PV.
```
ibmcloud is volume <volume_id>
```
Retrieve a list of worker nodes in your VPC cluster. Note the Zone of the worker node that is in the same zone as your storage volume.
```
ibmcloud ks worker ls -c <cluster_name>
```
Optional: If you provisioned your physical Block Storage for VPC instance by using a retain storage class, the PV and the physical storage is not removed when you remove the PVC. To use your physical Block Storage for VPC device in your cluster, you must remove the existing PV first.
1. List the PVs in your cluster and look for the PV that belongs to your Block Storage for VPC device. The PV is in a released state.
```
kubectl get pv
```
2. Remove the PV.
```
kubectl delete pv <pv_name>
```
Create a configuration file for your PV. Include the ID, Size, Zone, and IOPS that you retrieved earlier.
```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: <pv_name> # Example: my-persistent-volume
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: <vpc_block_storage_size> # Example: 20Gi
  csi:
    driver: vpc.block.csi.ibm.io
    fsType: ext4
    volumeAttributes:
      iops: "<vpc_block_storage_iops>" # Example: "3000"
      volumeId: <vpc_block_storage_ID> # Example: a1a11a1a-a111-1111-1a11-1111a11a1a11
      zone: "<vpc_block_zone>" # Example: "eu-de-3"
      region: "<vpc_block_region>"
    volumeHandle: <vpc_block_storage_ID>
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: failure-domain.beta.kubernetes.io/zone
          operator: In
          values:
          - <worker_node_zone> # Example: eu-de-3
        - key: failure-domain.beta.kubernetes.io/region
          operator: In
          values:
          - <worker_node_region> # Example: eu-de
        - key: kubernetes.io/hostname
          operator: In
          values:
          - <worker_node_primary_IP>
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  volumeMode: Filesystem
```
name

In the metadata section, enter a name for your PV.

storage

In the spec capacity section, enter the size of your Block Storage for VPC volume in gigabytes (Gi) that you retrieved earlier. For example, if the size of your device is 100 GB, enter 100Gi.

iops

In the spec CSI volume attributes section, enter the Max IOPS of the Block Storage for VPC volume that you retrieved earlier.

zone

In the spec CSI volume attributes section, enter the VPC block zone that matches the location that you retrieved earlier. For example, if your location is Washington DC-1, then use us-east-1 as your zone. To list available zones, run ibmcloud is zones. To find an overview of available VPC zones and locations, see Creating a VPC in a different region. Please mention "region" parameter when "zone" is specified.

region

The region of the worker node where you want to attach storage.

worker_node_primary_IP

The primary IP of the worker node where you want to attach storage. You can find the primary IP of your worker node by running ibmcloud ks worker ls.

volumeId and spec.csi.volumeHandle

In the spec CSI volume attributes section, enter the ID of the Block Storage for VPC volume that you retrieved earlier.

storageClassName

For the spec storage class name, enter an empty string.

matchExpressions

In the spec node affinity section, enter the node selector terms to match the zone. For the key, enter failure-domain.beta.kubernetes.io/zone. For the value, enter the zone of your worker node where you want to attach storage.

matchExpressions

In the spec node affinity section, enter the node selector terms to match the region. For the key, enter failure-domain.beta.kubernetes.io/region. For the value, enter the region of the worker node where you want to attach storage.
Create the PV in your cluster.
```
kubectl apply -f pv.yaml
```
Verify that the PV is created in your cluster.
```
kubectl get pv
```
Create another configuration file for your PVC. In order for the PVC to match the PV that you created earlier, you must choose the same value for the storage size and access mode. In your storage class field, enter an empty string value to match your PV. If any of these fields don't match the PV, then a new PV and a Block Storage for VPC instance are created automatically via dynamic provisioning.
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <pvc_name>
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: <vpc_block_storage_size>
  storageClassName: ""
```
Create your PVC.
```
kubectl apply -f pvc.yaml
```
Verify that your PVC is created and bound to the PV that you created earlier. This process can take a few minutes.
```
kubectl describe pvc <pvc_name>
```

Create a deployment or a pod that uses your PVC.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <deployment_name>
  labels:
    app: <deployment_label>
spec:
  selector:
    matchLabels:
      app: <app_name>
  template:
    metadata:
      labels:
        app: <app_name>
    spec:
      containers:
      - image: <image_name>
        name: <container_name>
        volumeMounts:
        - name: <volume_name>
          mountPath: /<file_path>
      volumes:
      - name: <volume_name>
        persistentVolumeClaim:
          claimName: <pvc_name>
      nodeSelector:
        kubernetes.io/hostname: "<worker_node_primary_IP>"

Updating the Block Storage for VPC cluster add-on

You can update the Block Storage for VPC cluster add-on by using the addon update command.

Before updating the add-on review the change log.

Before you update to a 5.x release from a previous release, you must not have any volume snapshots in failure state. For more information, see Why can't I delete my Block Storage for VPC volume snapshot resources?.

Check to see if an update is available. If an update is available, the plug-in version is flagged with an asterisk and the latest version is shown. Note the latest version as this value is used later.
```
ibmcloud ks cluster addons --cluster <cluster_name_or_ID>
```
Example output
```
Name                   Version                 Health State   Health Status   
vpc-block-csi-driver   1.0.0* (2.0.0 latest)   normal         Addon Ready
```

Update the add-on. Note that the update commands are different depending on the version that you have installed.

5.0 and later Run the addon update command.

ibmcloud ks cluster addon update vpc-block-csi-driver --cluster CLUSTER [-f] [-q] [--version VERSION] [-y]

All versions before version 5.0 Disable and enable the add-on.

ibmcloud ks cluster addon disable vpc-block-csi-driver --cluster CLUSTER [-f] [-q]

ibmcloud ks cluster addon enable vpc-block-csi-driver --cluster CLUSTER [-f] [-q] [--version VERSION] [-y]

Verify that the add-on is in the Addon Ready state. The add-on might take a few minutes to become ready.
```
ibmcloud ks cluster addon ls --cluster <cluster_name_or_ID>
```
Example output
```
Name                   Version   Health State   Health Status   
vpc-block-csi-driver   2.0.0     normal         Addon Ready
```
If you use a default storage class other than the ibmc-vpc-block-10iops-tier storage class you must change the default storage class settings in the addon-vpc-block-csi-driver-configmap ConfigMap. For more information, see Changing the default storage class.
If you created your own storage classes based on the default Block Storage for VPC storage classes, you must re-create those storage classes to update the parameters. For more information, see Recreating your own storage classes after updating to version 4.2.

Re-creating your own storage classes after updating to version 4.2

With version 4.2, the default parameters for storage classes has changed. The sizeRange or iopsRange parameters are no longer used. If you created any of your own storage classes that use these parameters, you must edit your own storage classes to remove these parameters. To change the parameters in your own storage classes, you must delete and re-create them. Previously, sizeRange and iopsRange were provided each storage class as reference information. With version 4.2, these references have been removed. Now, for information about block storage profiles, sizes, and IOPs, see the block storage profiles reference.

To find the details of your own storage classes, run the following command.
```
kubectl describe sc STORAGECLASS
```
If the storage class uses the sizeRange or iopsRange, get the storage class YAML and save it to a file.
```
kubectl get sc STORAGECLASS -o yaml
```
In the file that you saved from the output of the previous command, remove the sizeRange or iopsRange parameters.
Delete the storage class from your cluster.
```
kubectl delete sc STORAGECLASS
```
Recreate the storage class in your cluster by using the file you created earlier.
```
kubectl apply -f custom-storage-class.yaml
```

Setting up encryption for Block Storage for VPC

Use a key management service (KMS) provider, such as IBM® Key Protect, to create a private root key that you use in your Block Storage for VPC instance to encrypt data as it is written to the storage. After you create the private root key, create your own storage class or a Kubernetes secret with your root key and then use this storage class or secret to provision your Block Storage for VPC instance.

Enabling encryption for Block Storage for VPC impacts performance by approximately 20%. However, the exact impact depends on your worker node and storage volume configuration. Consider allowing for performance impacts when enabling encryption.

Create an instance of the KMS provider that you want to use.
- Key Protect.
- Hyper Protect Crypto Services.
Create a root key in your KMS instance.
- Key Protect root key.
- Hyper Protect Crypto Services root key. By default, the root key is created without an expiration date.
Set up service to service authorization. Authorize Block Storage for VPC to access IBM® Key Protect. Make sure to give Block Storage for VPC at least Reader access to your KMS instance.
Decide if you want to store the Key Protect root key CRN in a customized storage class or in a Kubernetes secret. Then, follow the steps to create a customized storage class or a Kubernetes secret.

Example customized storage class.
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name> # Enter a name for your storage class.
provisioner: vpc.block.csi.ibm.io
parameters:
  profile: "5iops-tier"
  csi.storage.k8s.io/fstype: "ext4"
  billingType: "hourly"
  encrypted: "true"
  encryptionKey: "<encryption_key>"
  resourceGroup: ""
  zone: ""
  tags: ""
  generation: "gc"
  classVersion: "1"
reclaimPolicy: "Delete"
```
encrypted

In the parameters, enter true to create a storage class that sets up encryption for your Block Storage for VPC volumes. If you set this option to true, you must provide the root key CRN of your Key Protect service instance that you want to use in parameters.encryptionKey.

encryptionKey

In the parameters, enter the root key CRN that you retrieved earlier.

Example Kubernetes secret.
```
apiVersion: v1
kind: Secret
type: vpc.block.csi.ibm.io
metadata:
  name: <secret_name>
  namespace: <namespace_name>
stringData:
  encrypted: <true_or_false>
data
  encryptionKey: <encryption_key>
```
name

Enter a name for your secret.

namespace

Enter the namespace where you want to create your secret.

encrypted

In the parameters, enter true to set up encryption for your Block Storage for VPC volumes.

encryptionKey

In the parameters, enter the root key CRN of your Key Protect service instance that you want to use to encrypt your Block Storage for VPC volume. To use your root key CRN in a secret, you must first convert it to base64 by running echo -n "<root_key_CRN>" | base64.
Follow steps 4-9 in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage for VPC that is configured for encryption with your Key Protect root key. Then, mount this storage to an app pod.

Your app might take a few minutes to mount the storage and get into a Running state.

Verify that your data is encrypted. List your Block Storage for VPC volumes and note the ID of the instance that you created. The storage instance Name equals the name of the PV that was automatically created when you created the PVC.

ibmcloud is vols

Example output

ID                                     Name                                       Status      Capacity   IOPS   Profile           Attachment type   Created                     Zone         Resource group
a395b603-74bf-4703-8fcb-b68e0b4d6960   pvc-479d590f-ca72-4df2-a30a-0941fceeca42   available   10         3000   5iops-tier        data              2019-08-17T12:29:18-05:00   us-south-1   a8a12accd63b437bbd6d58fb6a462ca7

Using the volume ID, list the details for your Block Storage for VPC instance to ensure that your Key Protect root key is stored in the storage instance. You can find the root key in the Encryption key field of your CLI output.

ibmcloud is vol <volume_ID>

Example output

ID                                     a395b603-74bf-4703-8fcb-b68e0b4d6960   
Name                                   pvc-479d590f-ca72-4df2-a30a-0941fceeca42   
Status                                 available   
Capacity                               10   
IOPS                                   3000   
Profile                                5iops-tier   
Encryption key                         crn:v1:bluemix:public:kms:us-south:a/6ef045fd2b43266cfe8e6388dd2ec098:53369322-958b-421c-911a-c9ae8d5156d1:key:47a985d1-5f5e-4477-93fc-12ce9bae343f   
Encryption                             user_managed   
Resource group                         a8a12accd63b437bbd6d58fb6a462ca7
Created                                2019-08-17T12:29:18-05:00
Zone                                   us-south-1   
Volume Attachment Instance Reference

Customizing the default storage settings

You can change some default PVC settings by using a customized storage class or a Kubernetes secret to create Block Storage for VPC with your customized settings.

What is the benefit of using a secret and specifying my parameters in a customized storage class?: As a cluster admin, create a customized storage class when you want all the PVCs that your cluster users create to be provisioned with a specific configuration and you don't want to enable your cluster users to override the default configuration.; However, when multiple configurations are required and you don't want to create a customized storage class for every possible PVC configuration, you can create one customized storage class with the default PVC settings and a reference to a generic Kubernetes secret. If your cluster users must override the default settings of your customized storage class, they can do so by creating a Kubernetes secret that holds their custom settings.

When you want to set up encryption for your Block Storage for VPC instance, you can also use a Kubernetes secret if you want to encode the Key Protect root key CRN to base64 instead of providing the key directly in the customized storage class.

Changing the default storage class

With version 4.2 the Block Storage for VPC cluster add-on sets the default storage class to the ibmc-vpc-block-10iops-tier class. If you have a default storage class other than ibmc-vpc-block-10iops-tier and your PVCs use the default storage class, this can result in multiple default storage classes which can cause PVC creation failures. To use a default storage class other than ibmc-vpc-block-10iops-tier, you can update the addon-vpc-block-csi-driver-configmap to change the IsStorageClassDefault to false.

The default storage class for the Block Storage for VPC cluster add-on is the ibmc-vpc-block-10iops-tier storage class.

Edit the addon-vpc-block-csi-driver-configmap

kubectl edit cm addon-vpc-block-csi-driver-configmap -n kube-system

Change the IsStorageClassDefault setting to false.
Save and exit.
Wait 15 minutes and verify the change by getting the details of the ibmc-vpc-block-10iops-tier storage class.
```
kubectl get sc ibmc-vpc-block-10iops-tier -o yaml
```

Creating your own storage class

Create your own customized storage class with the preferred settings for your Block Storage for VPC instance.

You might create your own storage class if you want to:

Set a custom IOPs value.
Set up Block Storage for VPC with a file system type other than ext4.
Set up encryption.

Review the Storage class reference to determine the profile that you want to use for your storage class. You can also review the custom profiles if you want to specify custom IOPs for your Block Storage for VPC.

If you want to use a pre-installed storage class as a template, you can get the details of a storage class by using the kubectl get sc <storageclass> -o yaml command.
Create a customized storage class configuration file.
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: <storage_class_name>
provisioner: vpc.block.csi.ibm.io
parameters:
  profile: "<profile>"
  csi.storage.k8s.io/fstype: "<file_system_type>"
  billingType: "hourly"
  encrypted: "<encrypted_true_false>"
  encryptionKey: "<encryption_key>"
  resourceGroup: ""
  zone: "<zone>"
  region: "<region>"
  tags: "<tags>"
  generation: "gc"
  classVersion: "1"
  iops: "<iops>" # Only specify this parameter if you are using a "custom" profile.
allowVolumeExpansion: (true|false) # Select true or false. Only supported on version 3.0.1 and later
volumeBindingMode: <volume_binding_mode>
  # csi.storage.k8s.io/provisioner-secret-name: # Uncomment and add secret parameters to enforce encryption. 
  # csi.storage.k8s.io/provisioner-secret-namespace: 
reclaimPolicy: "<reclaim_policy>"
```
name

Enter a name for your storage class.

profile

Enter the profile that you selected in the previous step, or enter custom to use a custom IOPs value. To find supported storage sizes for a specific profile, see Tiered IOPS profile. Any PVC that uses this storage class must specify a size value that is within this range.

csi.storage.k8s.io/fstype

In the parameters, enter the file system for your Block Storage for VPC instance. Choose xfs, ext3, or ext4. If you want to modify the ownership or permissions of your volume you must specify the csi.storage.k8s.io/fstype in your own storage class and your PVC must have ReadWriteOnce as the accessMode. The Block Storage for VPC driver uses the ReadWriteOnceWithFSType fsGroupPolicy. For more information, see CSI driver documentation.

encrypted

In the parameters, enter true to create a storage class that sets up encryption for your Block Storage for VPC volume. If you set this option to true, you must provide the root key CRN of your Key Protect service instance that you want to use in parameterencryptionKey. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

encryptionKey

If you entered true for parameters.encrypted, then enter the root key CRN of your Key Protect service instance that you want to use to encrypt your Block Storage for VPC volume. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

zone

In the parameters, enter the VPC zone where you want to create the Block Storage for VPC instance. Make sure that you use a zone that your worker nodes are connected to. To list VPC zones that your worker nodes use, run ibmcloud ks cluster get --cluster <cluster_name_or_ID> and look at the Worker Zones field in your CLI output. If you don't specify a zone, one of the worker node zones is automatically selected for your Block Storage for VPC instance.

region

The region of the worker node where you want to attach storage.

tags

In the parameters, enter a space-separated list of tags to apply to your Block Storage for VPC instance. Tags can help you find instances more easily or group your instances based on common characteristics, such as the app or the environment that it is used for.

iops

If you entered custom for the profile, enter a value for the IOPs that you want your Block Storage for VPC to use. Refer to the Block Storage for VPC custom IOPs profile table for a list of supported IOPs ranges by volume size.

reclaimPolicy

Enter the reclaim policy for your storage class. If you want to keep the PV, the physical storage device and your data when you remove the PVC, enter Retain. If you want to delete the PV, the physical storage device and your data when you remove the PVC, enter Delete.

allowVolumeExpansion

Enter the volume expansion policy for your storage class. If you want to allow volume expansion, enter true. If you don't want to allow volume expansion, enter false.

volumeBindingMode

Choose if you want to delay the creation of the Block Storage for VPC instance until the first pod that uses this storage is ready to be scheduled. To delay the creation, enter WaitForFirstConsumer. To create the instance when you create the PVC, enter Immediate.
Create the customized storage class in your cluster.
```
kubectl apply -f custom-storageclass.yaml
```

Verify that your storage class is available in the cluster.

kubectl get sc

Example output

NAME                                    PROVISIONER            AGE
ibmc-vpc-block-10iops-tier              vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-5iops-tier               vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-custom                   vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-general-purpose          vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-10iops-tier       vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-5iops-tier        vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-custom            vpc.block.csi.ibm.io   4d21h
ibmc-vpc-block-retain-general-purpose   vpc.block.csi.ibm.io   4d21h
<custom-storageclass>             vpc.block.csi.ibm.io   4m26s

Follow the steps in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage for VPC. Then, mount this storage to a sample app.
Optional: Verify your Block Storage for VPC file system type.

Verifying your Block Storage for VPC file system

You can create a customized storage class to provision Block Storage for VPC with a different file system, such as xfs or ext3. By default, all Block Storage for VPC instances are provisioned with an ext4 file system.

Follow the steps to create a customized storage class with the file system that you want to use.
Follow steps 4-9 in Adding Block Storage for VPC to your apps to create a PVC with your customized storage class to provision Block Storage for VPC with a different file system. Then, mount this storage to an app pod.

Your app might take a few minutes to mount the storage and get into a Running state.
Verify that your storage is mounted with the correct file system. List the pods in your cluster and note the Name of the pod that you used to mount your storage.
```
kubectl get pods
```
Log in to your pod.
```
kubectl exec <pod_name> -it bash
```

List the mount paths inside your pod.

mount | grep /dev/xvdg

Example output for xfs.

/dev/xvdg on /test type xfs (rw,relatime,attr2,inode64,noquota)

Exit your pod.
```
exit
```

Updating the `VolumeAttachLimit`

In versions 5.2 and later of the Block Storage for VPC cluster add-on, you can edit the maximum number of volumes that can be attached to each node by editing the configmap. The default value is 12.

Your account must be approved to use this feature.

Edit the configmap. Replace VALUE with the volume attachment limit that you want to set.

kubectl patch configmap/addon-vpc-block-csi-driver-configmap \       
-n kube-system \
--type merge \
-p '{"data":{"VolumeAttachmentLimit":"VALUE"}}'

Wait for the ibm-vpc-block-csi-node pods in the kube-system namespace to restart. Verify that the pods have restarted.
```
kubectl get pods -n kube-system -w| grep block-csi
```
You can now attach volumes to your worker nodes by using dynamic provisioning or by manually creating the attachments. For more information, see Adding Block Storage for VPC to your apps or Using an existing Block Storage for VPC instance.

Storing your custom PVC settings in a Kubernetes secret

Specify your PVC settings in a Kubernetes secret and reference this secret in a customized storage class. Then, use the customized storage class to create a PVC with the custom parameters that you set in your secret.

What options do I have to use the Kubernetes secret?: As a cluster admin, you can choose if you want to allow each cluster user to override the default settings of a storage class, or if you want to create one secret that everyone in your cluster must use and that enforces base64 encoding for your Key Protect root key CRN.
Every user can customize the default settings: In this scenario, the cluster admin creates one customized storage class with the default PVC settings and a reference to a generic Kubernetes secret. Cluster users can override the default settings of the storage class by creating a Kubernetes secret with the PVC settings that they want. In order for the customized settings in the secret to get applied to your Block Storage for VPC instance, you must create a PVC with the same name as your Kubernetes secret.
Enforce base64 encoding for the Key Protect root key: In this scenario, you create one customized storage class with the default PVC settings and a reference to a static Kubernetes secret that overrides or enhances the default settings of the customized storage class. Your cluster users can't override the default settings by creating their own Kubernetes secret. Instead, cluster users must provision Block Storage for VPC with the configuration that you chose in your customized storage class and secret. The benefit of using this method over creating a customized storage class only is that you can enforce base64 encoding for the root key CRN of your Key Protect service instance when you want to encrypt the data in your Block Storage for VPC instance.
What do I need to be aware of before I start using the Kubernetes secret for my PVC settings?: Some PVC settings, such as the reclaimPolicy, fstype, or the volumeBindingMode can't be set in the Kubernetes secret and must be set in the storage class. As the cluster admin, if you want to enable your cluster users to override your default settings, you must ensure that you set up enough customized storage classes that reference a generic Kubernetes secret so that your users can provision Block Storage for VPC with different reclaimPolicy, fstype, and volumeBindingMode settings.

Enabling every user to customize the default PVC settings

As the cluster admin, follow the steps to create a customized storage class. In the customized storage class YAML file, reference the Kubernetes secret in the metadata.parameters section as follows. Make sure to add the code as-is and not to change variables names.
```
csi.storage.k8s.io/provisioner-secret-name: ${pvc.name}
csi.storage.k8s.io/provisioner-secret-namespace: ${pvc.namespace}
```
As the cluster user, create a Kubernetes secret that customizes the default settings of the storage class.
```
apiVersion: v1
kind: Secret
type: vpc.block.csi.ibm.io
metadata:
  name: <secret_name>
  namespace: <namespace_name>
stringData:
  iops: "<IOPS_value>"
  zone: "<zone>"
  tags: "<tags>"
  encrypted: <true_or_false>
  resourceGroup: "<resource_group>"
data
  encryptionKey: <encryption_key>
```
name

Enter a name for your Kubernetes secret.

namespace

Enter the namespace where you want to create your secret. To reference the secret in your PVC, the PVC must be created in the same namespace.

iops

In the string data section, enter the range of IOPS that you want to allow for your Block Storage for VPC instance. The range that you enter must match the Block Storage for VPC tier that you plan to use.

zone

In the string data section, enter the VPC zone where you want to create the Block Storage for VPC instance. Make sure that you use a zone that your worker nodes are connected to. To list VPC zones that your worker nodes use, run ibmcloud ks cluster get --cluster <cluster_name_or_ID> and look at the Worker Zones field in your CLI output. If you don't specify a zone, one of the worker node zones is automatically selected for your Block Storage for VPC instance.

tags

In the string data section, enter a comma-separated list of tags to use when the PVC is created. Tags can help you find your storage instance after it is created.

resourceGroup

In the string data section, enter the resource group ID that you want your Block Storage for VPC instance to get access to. If you don't enter a resource group, the instance is automatically authorized to access resources of the resource group that your cluster belongs to.

encrypted

In the string data section, enter true to create a secret that sets up encryption for Block Storage for VPC volumes. If you set this option to true, you must provide the root key CRN of your Key Protect service instance that you want to use in parameters.encryptionKey. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

encryptionKey

In the data section, if you entered true for parameters.encrypted, then enter the root key CRN of your Key Protect service instance that you want to use to encrypt your Block Storage for VPC volumes. To use your root key CRN in a secret, you must first convert it to base64 by running echo -n "<root_key_CRN>" | base64. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.
Create your Kubernetes secret.
```
kubectl apply -f secret.yaml
```
Follow the steps in Adding Block Storage for VPC to your apps to create a PVC with your custom settings. Make sure to create the PVC with the customized storage class that the cluster admin created and use the same name for your PVC that you used for your secret. Using the same name for the secret and the PVC triggers the storage provider to apply the settings of the secret in your PVC.

Enforcing base64 encoding for the Key Protect root key CRN

As the cluster admin, create a Kubernetes secret that includes the base64 encoded value for your Key Protect root key CRN. To retrieve the root key CRN, see Setting up encryption for your Block Storage for VPC.
```
apiVersion: v1
kind: Secret
type: vpc.block.csi.ibm.io
metadata:
  name: <secret_name>
  namespace: <namespace_name>
stringData:
  encrypted: <true_or_false>
  resourceGroup: "<resource_group>"
data:
  encryptionKey: <encryption_key>
```
name

Enter a name for your Kubernetes secret.

namespace

Enter the namespace where you want to create your secret. To reference the secret in your PVC, the PVC must be created in the same namespace.

encrypted

In the string data section, enter true to create a secret that sets up encryption for Block Storage for VPC volumes. If you set this option to true, you must provide the root key CRN of your Key Protect service instance that you want to use in parameters.encryptionKey. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.

encryptionKey

In the data section, if you entered truefor parameters.encrypted, then enter the root key CRN of your Key Protect service instance that you want to use to encrypt your Block Storage for VPC volume. To use your root key CRN in a secret, you must first convert it to base 64 by running echo -n "<root_key_CRN>" | base64. For more information about encrypting your data, see Setting up encryption for your Block Storage for VPC.
Create the Kubernetes secret.
```
kubectl apply -f secret.yaml
```
Follow the steps to create a customized storage class. In the customized storage class YAML file, reference the Kubernetes secret in the metadata.parameters section as follows. Make sure to enter the name of the Kubernetes secret that you created earlier and the namespace where you created the secret.
```
csi.storage.k8s.io/provisioner-secret-name: <secret_name>
csi.storage.k8s.io/provisioner-secret-namespace: <secret_namespace>
```
As the cluster user, follow the steps in Adding Block Storage for VPC to your apps to create a PVC from your customized storage class.

Setting up volume expansion

To provision volumes that support expansion, you must use storage class that has allowVolumeExpansion set to true.

You can only expand volumes that are mounted by an app pod.

If you are not using version 4.2 or later of the add-on, update the Block Storage for VPC cluster add-on in your cluster.
Create a PVC that uses a storage class that supports volume expansion.
Deploy an app that uses your PVC. When you create your app, make a note of the mountPath that you specify.
After your PVC is mounted by an app pod, you can expand your volume by editing the value of the spec.resources.requests.storage field in your PVC. To expand your volume, edit your PVC and increase the value in the spec.resources.requests.storage field.
```
kubectl edit pvc <pvc-name>
```
Example
```
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
```
Save and close the PVC.
Optional: Verify that your volume is expanded. Get the details of your PVC and make a note of the PV name.
```
kubectl get pvc <pvc-name>
```
Describe your PV and make a note of the volume ID.
```
kubectl describe PV
```
Get the details of your Block Storage for VPC volume and verify the capacity.
```
ibmcloud is vol <volume-ID>
```

Manually expanding volumes before add-on version 4.2

Complete the following steps to manually expand your existing Block Storage for VPC volumes that were created before version 4.2 of the add-on.

You can only expand volumes that are mounted by an app pod.

Get the details of your app and make a note of the PVC name and mountPath.
```
kubectl get pod <pod-name> -n <pod-namespace> -o yaml
```
Get the details of your PVC and make a note of the PV name.
```
kubectl get pvc
```
Describe your PV and get the volumeId.
```
kubectl describe pv `pv-name` | grep volumeId 
```
Example output for a volume ID of r011-a1aaa1f1-3aaa-4a73-84aa-0aa32e11a1a1.
```
volumeId=r011-a1aaa1f1-3aaa-4a73-84aa-0aa32e11a1a1
```
Resize the volume by using a PATCH request. The following example resizes a volume to 250 GiB.
```
curl -sS -X PATCH -H "Authorization: <iam_token>" "https://<region>.iaas.cloud.ibm.com/v1/volumes/<volumeId>?generation=2&version=2020-06-16" -d '{"capacity":250}'
```
<iam_token>

Your IAM token. To retrieve your IAM token, run ibmcloud iam oauth-tokens.

<region>

The region your cluster is in, for example us-south.

<volumeId>

The volume ID that you retrieved earlier. For example r011-a1aaa1f1-3aaa-4a73-84aa-0aa32e11a1a1.

<capacity>

The increased capacity in GiB, for example 250.
Log in to your app pod.
```
kubectl exec <pod-name> -it -- bash
```
Run the following command to use host binaries.
```
chroot /host
```

Get the file system details and make a note of the Filesystem path that you want to update. You can also grep for the mount path as specified in your application pod. df -h | grep <mount-path>.

df -h

Example output

Filesystem      Size  Used Avail Use% Mounted on
overlay          98G   64G   29G  70% /
tmpfs            64M     0   64M   0% /dev
tmpfs            32G     0   32G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/vda2        98G   64G   29G  70% /etc/hosts
/dev/vdg        9.8G   37M  9.8G   1% /mount-path # Note the Filesystem path that corresponds to the mountPath that you specified in your app.
tmpfs            32G   40K   32G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware

Resize the file system.

sudo resize2fs <filesystem-path>

Example command

sudo resize2fs /dev/vdg

Verify the file system is resized.
```
df -h
```

Backing up and restoring data

Data on Block Storage for VPC is secured across redundant fault zones in your region. To manually back up your data, use the Kubernetes kubectl cp command.

You can use the kubectl cp command to copy files and directories to and from pods or specific containers in your cluster

Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To back up or restore data, choose between the following options:

Copy data from your local machine to a pod in your cluster.

kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath>

Copy data from a pod in your cluster to your local machine.

kubectl cp <namespace>/<pod>:<pod_filepath>/<filename> <local_filepath>/<filename>

Copy data from your local machine to a specific container that runs in a pod in your cluster.

kubectl cp <local_filepath>/<filename> <namespace>/<pod>:<pod_filepath> -c CONTAINER

Understanding volume request capacity

The VPC Block CSI Driver calculates the volume capacity by using the following formula.

If value is provided in Gi: rBytes(requestedBytes) = X * 1024^3
If value is provided in G: rBytes(requestedBytes) = X * 10^9

The requested value is equal to (((rBytes+ GiB - 1) / GiB) * GiB) / GiB.

For example:

If a value of 20Gi is provided, applying the above formula, the volume that is created is 20GB.
If a value of 20G is provided, the volume that is created is 19GB.

Sometimes, the capacity of the volume that is created is less than the requested value. Note that billing applies for the volume that gets created, not the requested volume.

Setting up Block Storage for VPC

Quick start for IBM Cloud Block Storage for VPC

Adding Block Storage for VPC to your apps

Using an existing Block Storage for VPC instance

Updating the Block Storage for VPC cluster add-on

Re-creating your own storage classes after updating to version 4.2

Setting up encryption for Block Storage for VPC

Customizing the default storage settings

Changing the default storage class

Creating your own storage class

Verifying your Block Storage for VPC file system

Updating the VolumeAttachLimit

Storing your custom PVC settings in a Kubernetes secret

Enabling every user to customize the default PVC settings

Enforcing base64 encoding for the Key Protect root key CRN

Setting up volume expansion

Manually expanding volumes before add-on version 4.2

Backing up and restoring data

Understanding volume request capacity

Updating the `VolumeAttachLimit`