Debugging Block Storage for Classic failures

Review the options to debug Block Storage for Classic and find the root causes of any failures.

Checking whether the pod that mounts your storage instance is successfully deployed

Follow the steps to review any error messages related to pod deployment.

List the pods in your cluster. A pod is successfully deployed if the pod shows a status of Running.
```
oc get pods
```
Get the details of your pod and review any error messages that are displayed in the Events section of your CLI output.
```
oc describe pod <pod_name>
```
Retrieve the logs for your pod and review any error messages.
```
oc logs <pod_name>
```
Review the Block Storage for Classic troubleshooting documentation for steps to resolve common errors.

Restarting your app pod

Some issues can be resolved by restarting and redeploying your pods. Follow the steps to redeploy a specific pod.

If your pod is part of a deployment, delete the pod and let the deployment rebuild it. If your pod is not part of a deployment, delete the pod and reapply your pod configuration file.
1. Delete the pod.
```
oc delete pod <pod_name>
```
  Example output
```
pod "nginx" deleted
```
2. Reapply the configuration file to redeploy the pod.
```
oc apply -f <app.yaml>
```
  Example output
```
pod/nginx created
```
If restarting your pod does not resolve the issue, reload your worker nodes.
Verify that you use the latest IBM Cloud and IBM Cloud Kubernetes Service plug-in version.
```
ibmcloud update
```
```
ibmcloud plugin repo-plugins
```
```
ibmcloud plugin update
```

Verifying that the storage driver and plug-in pods show a status of Running

Follow the steps to check the status of your storage driver and plug-in pods and review any error messages.

List the pods in the kube-system project.
```
oc get pods -n kube-system
```
If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, the following commands might fail.
1. Get the names of the containers that run in the driver pod.
```
kubectl describe pod <pod_name> -n kube-system 
```
2. Export the logs from the driver pod to a logs.txt file on your local machine.
```
oc logs <pod_name> -n kube-system > logs.txt
```
3. Review the log file.
```
cat logs.txt
```
If the storage driver and plug-in pods don't show a Running status, get more details of the pod to find the root cause. Depending on the status of your pod, you might not be able to execute all the following commands.
1. Get the names of the containers that run in the driver pod.
```
kubectl get pod ibm-vpc-block-csi-controller-0 -n kube-system -o jsonpath="{.spec['containers','initContainers'][*].name}" | tr -s '[[:space:]]' '\n'
```
  Example output for Block Storage for VPC
```
csi-provisioner 
csi-attacher
liveness-probe
iks-vpc-block-driver
```
2. Export the container logs from the driver pod to a logs.txt file on your local machine.
```
oc logs <pod_name> -n kube-system -c <container_name> > logs.txt
```
Check the latest logs for any error messages. Review the Block Storage for Classic troubleshooting documentation for steps to resolve common errors.

Checking whether your PVC is successfully provisioned.

Follow the steps to check the status of your PVC and review any error messages.

Check the status of your PVC. A PVC is successfully provisioned if the PVC shows a status of Bound.

oc get pvc

If the PVC shows a status of Bound, the PVC is successfully provisioned.

Example output

NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
silver-pvc   Bound     pvc-4b881a6b-ada8-4a44-b568-fe909107d756   24Gi       RWX            ibmc-file-silver            7m29s

If the status of the PVC shows Pending, describe the PVC and review the Events section of the output for any warnings or error messages. Note that PVCs that reference storage classes with the volume binding mode set to WaitForFirstConsumer remain Pending until an app pod is deployed that uses the PVC.

oc describe pvc <pvc_name>

Example output

Name:          local-pvc
Namespace:     default
StorageClass:  sat-local-file-gold
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
Type     Reason              Age                 From                         Message
----     ------              ----                ----                         -------
Warning  ProvisioningFailed  60s (x42 over 11m)  persistentvolume-controller  storageclass.storage.k8s.io "sat-local-file-gold" not found

Review the Block Storage for Classic troubleshooting documentation for steps to resolve common errors.

Checking and updating the oc CLI version

If you use a oc CLI version that does not match at least the major.minor version of your cluster, you might experience unexpected results. For example, Kubernetes does not support oc client versions that are 2 or more versions apart from the server version (n +/- 2).

Verify that the oc CLI version that you run on your local machine matches the Kubernetes version that is installed in your cluster. Show the oc CLI version that is installed in your cluster and your local machine.

oc version

Example output:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.31", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.31+IKS", GitCommit:"e15454c2216a73b59e9a059fd2def4e6712a7cf0", GitTreeState:"clean", BuildDate:"2019-04-01T10:08:07Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

The CLI versions match if you can see the same version in GitVersion for the client and the server. You can ignore the +IKS part of the version for the server.

If the oc CLI versions on your local machine and your cluster don't match, either update your cluster or install a different CLI version on your local machine.

Checking and updating the Block Storage for Classic driver

For Block Storage for VPC, verify that you have the latest version of the Block Storage for VPC cluster add-on.
For Block Storage for Classic on classic clusters, make sure that you installed the latest Helm chart version for the plug-in.
1. Update your Helm chart repositories.
```
helm repo update
```
2. List the Helm charts in the repository.
```
helm search repo iks-charts | grep block-storage-plugin
```
  Example output
```
iks-charts-stage/ibmcloud-block-storage-plugin    1.5.0                                                        A Helm chart for installing ibmcloud block storage plugin   
iks-charts/ibmcloud-block-storage-plugin          1.5.0                                                        A Helm chart for installing ibmcloud block storage plugin   
```
3. List the installed Helm charts in your cluster and compare the version that you installed with the version that is available.
```
helm list --all-namespaces
```
4. If a more recent version is available, install this version. For instructions, see Updating the IBM Cloud Block Storage plug-in.