OpenShift Data Foundation Regional Disaster Recovery on Red Hat OpenShift on IBM Cloud clusters

Virtual Private Cloud 4.17 and later

Regional Disaster Recovery ensures business continuity during the unavailability of a geographical region. You can use Red Hat Advanced Cluster Management (ACM) to set up the Regional Disaster Recovery solutions for ODF clusters.

The following steps for setting ODF Disaster Recovery are available as a Technical Preview only and not for production use.

Here are the high-level steps of this solution:

Create the 3 VPC clusters (each in different regions/VPCs) and set the --disable-outbound-traffic-protection parameter for each.
Install ODF on 2 of them.
Install ACM on 1 of them.
Configure ACM to manage the 2 ODF clusters.

With this set up, the ACM cluster manages the ODF clusters. So if one ODF cluster goes down, then the ACM cluster rolls over the apps and data from that cluster to the other cluster.

Creating the clusters

Create 3 VPC clusters in different regions. Make sure to include the --disable-outbound-traffic-protection parameter.

Create managed cluster 1 in us-east, which will be the primary ODF cluster.

ibmcloud ks cluster create vpc-gen2 --flavor bx2.16x64 --name managed-cluster-1-dr-odf --subnet-id 0767-5c97dd71-95e7-4f26-a31b-e2c7a8ad16f7 --vpc-id r014-8ac09b7e-99d0-460d-96bf-2dfe3eac1041 --zone us-east-2  --version 4.17.10_openshift --workers 3 --cos-instance crn:v1:bluemix:public:cloud-object-storage:global:a/c468d8824937fecd8a0860fe0f379bf9:3887cefc-edcc-48df-a05e-ac3f02df10d5:: --disable-outbound-traffic-protection

Create managed cluster 2 in jp-tok, which will be the secondary ODF cluster. Ensure that managed cluster 2 does not have overlapping network with managed cluster 1.

ibmcloud ks cluster create vpc-gen2 --flavor bx2.16x64 --name managed-cluster-2-dr-odf --subnet-id 02e7-4f2da612-6327-4968-88db-41d92c3e9c1b --vpc-id r022-fd23415e-fab0-42b1-b6da-567cc6f3df0c --zone jp-tok-1 --version 4.17.10_openshift  --workers 3 --cos-instance crn:v1:bluemix:public:cloud-object-storage:global:a/c468d8824937fecd8a0860fe0f379bf9:3887cefc-edcc-48df-a05e-ac3f02df10d5:: --disable-outbound-traffic-protection

Create a hub cluster in us-east to install ACM on.

ibmcloud ks cluster create vpc-gen2 --flavor bx2.8x32 --name acm-hub-cluster-dr-odf --subnet-id 0767-5c97dd71-95e7-4f26-a31b-e2c7a8ad16f7 --vpc-id r014-8ac09b7e-99d0-460d-96bf-2dfe3eac1041 --zone us-east-2 --version 4.17.10_openshift --workers 3 --cos-instance crn:v1:bluemix:public :cloud-object-storage:global:a/c468d8824937fecd8a0860fe0f379bf9:3887cefc-edcc-48df-a05e-ac3f02df10d5:: --disable-outbound-traffic-protection

Enabling the Red Hat OperatorHub catalog

By default, the Red Hat OperatorHub catalog is disabled. You need to enable it manually.

Run the following command.

kubectl edit operatorhub cluster -n openshift-marketplace

Update disableAllDefaultSources value to false.

Verify that all marketplace pods are up.

spec:
disableAllDefaultSources: false
status:
  sources:
  - disabled: false
    name: redhat-operators
    status: Success
  - disabled: false
    name: certified-operators
    status: Success
  - disabled: false
    name: community-operators
    status: Success
  - disabled: false
    name: redhat-marketplace
    status: Success

Using ACM to set up Regional Diaster Recovery

Follow these steps to install ACM on the hub cluster and then set up the disaster recovery policy.

Follow the steps to install the ACM add-on. Be sure to import any ODF clusters you want to apply a disaster recovery policy to.
Configure the Submariner add-on from the ACM Console.
1. Create a cluster set and add the managed clusters to the cluster set.
2. Select to install Submariner add-ons for the cluster set.
3. Select the managed ODF clusters as target clusters for add-on installation.
4. When reviewing the configuration for both clusters, change the following settings as shown and leave the rest as default. Then click Install.
```
  globalnetEnabled: true (checked)
  gateways: 2
  NATTEnable: false (unchecked)
  cableDriver: vxlan.
```
5. Wait for the Submariner add-on status to show healthy (green). This can take up to 20 minutes. For more information about Submariner setup, see Deploying Submariner manually.
Install the OpenShift Data Foundation add-on version 4.17 or later on the primary and secondary managed clusters. Make sure you select the Enable DR option and enable NooBaa installation during the ODF installation. For more information, see Installing the OpenShift Data Foundation add-on.
In the storagecluster resource’s multiClusterService section, update the ACM Managed Cluster Name to allow ODF to use GlobalNet. For more information, see Creating an OpenShift Data Foundation cluster on managed clusters.
```
kubectl patch storagecluster -n openshift-storage ocs-storagecluster --type merge -p'{"spec":{"network":{"multiClusterService":{"clusterID":"managed-cluster-1-dr-odf","enabled":true}}}}’
```

Verify the service exports. This might take a few minutes to show in the output.

oc get serviceexport -n openshift-storage

Example output:

NAME              AGE
rook-ceph-mon-d   4d14h
rook-ceph-mon-e   4d14h
rook-ceph-mon-f   4d14h
rook-ceph-osd-0   4d14h
rook-ceph-osd-1   4d14h
rook-ceph-osd-2   4d14h

If you enabled GlobalNet when you installed Submariner, connect the StorageCluster with the ocs-provider-server ServiceExport.
1. Copy the following yaml to create a ServiceExport called ocs-provider-server. Create the resource on both the primary managed cluster and the secondary managed cluster.
```
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: ocs-provider-server
  namespace: openshift-storage
```
2. Run the command to add an annotation to the StorageCluster. Complete this step on both the primary managed cluster and the secondary managed cluster.
```
oc annotate storagecluster ocs-storagecluster -n openshift-storage ocs.openshift.io/api-server-exported-address=<managedcluser_name>.ocs-provider-server.openshift-storage.svc.clusterset.local:500511. 
```
Install ODF Multicluster Orchestrator to the ACM hub cluster. For more information, see Installing ODF Multicluster Orchestrator on Hub cluster.
Create a DR Policy for both managed ODF clusters with a sync interval of 5 minutes. For more information, see Creating Disaster Recovery Policy on Hub cluster.
Optional: Review the operators you can install to enhance ODF Regional Disaster Recovery features.

Optional operators for ODF Regional Disaster Recovery

Review the optional operators you can install on your ACM hub or managed clusters to enhance ODF Regional Disaster Recovery features. Note that IBM is not responsible for managing these operators.

You are responsible for managing these operators, including but not limited to updating, monitoring, recovery, and re-installation.

Optional operators for ODF Regional Disaster Recovery
Operator	Description	Additional information
OpenShift API for Data Protection (OADP) Operator	Use to create backup and restore APIs for OpenShift clusters. Install on managed clusters.	Introduction to OpenShift API for data protection
Submariner	Provides direct networking between two or more Kubernetes clusters in your environment. Install on managed clusters.	Submariner

Testing your disaster recovery configuration

Create a sample application to test your disaster recovery solution. For more information, see Create sample application for testing disaster recovery application.

Deploy a subscription-based application from the ACM Console. The application's topology tab shows green when all application resources are deployed successfully.
On the application page, go to Actions > Manage Data Policy.
Assign the DR policy created earlier to this application.
Verify that the application pods are running on the primary cluster.
On the application page, go to Actions > Failover application. Select your secondary ODF cluster as the target cluster. Click Initiate.
Verify that the application pods are moved to the secondary cluster.
On the application page, go to Actions > Relocate application. Select your primary ODF cluster as the target cluster. Click Initiate.
Verify that the application pods are moved back to the primary cluster.