I deleted a portable subnet and now my Classic cluster my Load Balancers are failing. How do I recover?

Classic

your cluster is experiencing network errors.

A portable subnet was deleted.

Complete the following steps.

1. For the cluster that is experiencing issues, run the following command and make a note of the cluster ID.

   ibmcloud oc cluster ls
   

2. List the subnets your cluster is using. In the output, make a note of the subnets that your cluster is using.

   ibmcloud oc subnets --provider classic | grep CLUSTER-ID
   

3. List the subnets in your account. Check whether any subnets that you found in step 1 are missing from the list of subnets in your account. This means your cluster is using a subnet that was deleted, which is causing your Load Balancer to fail.

   ibmcloud sl subnet list
   

   ibmcloud sl subnet detail SUBNET-ID
   

4. For each subnet that was deleted, re-create it. If you need public IP addresses for your ALBs or LoadBalancers, specify the public VLAN ID of your worker nodes; for private IPs specify the private VLAN ID of your worker nodes.

   ibmcloud oc cluster subnet create -c CLUSTER --size 8 --vlan VLAN-ID
   

5. Find any LoadBalancers that use IP addresses in the deleted portable subnets, save them, and then delete them. Do not delete the LoadBalancers in the openshift-ingress namespace, as those are deleted later in these steps.

   1. Find your LoadBalancer details and make a note of the IP addresses they use.

      kubectl get svc -A -o wide | grep LoadBalancer
      

   2. For each LoadBalancer not in the openshift-ingress namespace that uses a portable IP address, save the settings to a YAML file.

   kubectl get svc -o yaml -n LB-NAMESPACE LB-NAME > LB-NAMESPACE.LB-NAME.yaml
   

   1. Delete the LoadBalancer.

      kubectl delete svc -n LB-NAMESPACE LB-NAME
      

6. Complete the following steps quickly, because the portable subnet must be detached before the openshift-ingress-operator pod restarts and re-creates the LoadBalancers.

   1. Scale down the Ingress operator.

      kubectl scale deploy -n openshift-ingress-operator ingress-operator --replicas 0
      

   2. Delete the LoadBalancers in the openshift-ingress namespace.

      kubectl delete svc -n openshift-ingress LB-NAME
      

   3. Detach the portable subnets from your cluster.

      ibmcloud oc cluster subnet detach -c CLUSTER-ID --subnet-id SUBNET-ID
      

7. Wait a few minutes for the detach command to finish, then check that they have been detached by reviewing the ibm-cloud-provider-vlan-ip-config ConfigMap that stores the subnets and IP addresses.

   kubectl get cm -n kube-system ibm-cloud-provider-vlan-ip-config -o yaml
   

   It might take 5-10 minutes for the detach to complete. While you wait, you can ensure that the new portable subnets you created for this cluster also appear in the ConfigMap.

8. Check whether the LoadBalancers in the openshift-ingress namespace re-created, and if so, check whether the IP addresses used are from the new subnets. If they are from the old subnets, then delete them again by using the kubectl delete svc -n openshift-ingress LOADBALANCER_NAME command.

9. After the openshift-ingress LoadBalancers re-create with IP addresses from the new subnets, add the new IP addresses to the DNS record.

   1. List the DNS entries.

      ibmcloud oc nlb-dns ls -c CLUSTER-ID
      

   2. In the output, find the openshift-ingress entries, and add the new IP address(es) to them by using the following command.

      ibmcloud ks nlb-dns add -c CLUSTER-ID --ip NEW-LOADBALANCE-IP --nlb-host INGRESS-DOMAIN-NAME
      

   3. Remove the old IPs.

      ibmcloud oc nlb-dns rm classic -c CLUSTER-ID --ip OLD-LOADBALANCER-IP --nlb-host INGRESS-DOMAIN-NAME
      

10. Re-create any LoadBalancers not in the openshift-ingress namespace that you deleted in step 3 by by using the YAML file that you saved earlier.

    kubectl apply -f LB-NAMESPACE.LB-NAME.yaml