Deploying apps in Red Hat OpenShift clusters

Prerequisites

Before you begin

• Create a cluster or worker pool that uses a GPU flavor. Keep in mind that setting up a bare metal machine can take more than one business day to complete. For a list of available flavors, see the following links.

  • Classic flavors
  • VPC flavors

• Make sure that you are assigned a service access role that grants the appropriate Kubernetes RBAC role so that you can work with Kubernetes resources in the cluster.

Private only cluster limitations: If you cluster doesn't have public network connectivity, you must allow public network connectivity or mirror images from external registries and image streams to icr.io. In the following example, the GPU app uses the OLM marketplace with image streams. This example does not work if your cluster can't access the NVIDIA registry (nvcr.io).

• Install the NVIDIA GPU operator for your cluster version.
• Install the Node Feature Discovery and NVIDIA GPU operators for your cluster version.

You must use NVIDIA GPU operator version 1.3.1 or later. When you install the Node Feature Discovery operator, select the update channel that matches your Red Hat OpenShift cluster version. Don't install the operators through another method, such as a Helm chart.

If you experience issues installing the Node Feature Discovery Operator or the NVIDIA GPU Operator, contact NVIDIA support for help or open an issue in the NVIDIA GPU Operator repo

Deploying a workload

1. Create a YAML file. In this example, a Job YAML manages batch-like workloads by making a short-lived pod that runs until the command completes and successfully terminates.

   For GPU workloads, you must specify the resources: limits: nvidia.com/gpu field in the job YAML.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: nvidia-devicequery
     labels:
       name: nvidia-devicequery
   spec:
     template:
       metadata:
         labels:
           name: nvidia-devicequery
       spec:
         containers:
         - name: nvidia-devicequery
           image: nvcr.io/nvidia/k8s/cuda-sample:devicequery-cuda11.7.1-ubuntu20.04
           imagePullPolicy: IfNotPresent
           resources:
             limits:
               nvidia.com/gpu: 2
         restartPolicy: Never
   

   Understanding your YAML components

   Component	Description
   Metadata and label names	Enter a name and a label for the job, and use the same name in both the file's metadata and the spec template metadata. For example, nvidia-devicequery.
   containers.image	Provide the image that the container is a running instance of. In this example, the value is set to use the DockerHub CUDA device query image:nvcr.io/nvidia/k8s/cuda-sample:devicequery-cuda11.7.1-ubuntu20.04.
   containers.imagePullPolicy	To pull a new image only if the image is not currently on the worker node, specify IfNotPresent.
   resources.limits	For GPU machines, you must specify the resource limit. The Kubernetes Device Plug-in sets the default resource request to match the limit. You must specify the key as nvidia.com/gpu. Enter the whole number of GPUs that you request, such as 2. Note that container pods don't share GPUs and GPUs can't be overcommitted. For example, if you have only 1 mg1c.16x128 machine, then you have only 2 GPUs in that machine and can specify a maximum of 2.

2. Apply the YAML file. For example:

   oc apply -f nvidia-devicequery.yaml
   

3. Check the job pod by filtering your pods by the nvidia-devicequery label. Verify that the STATUS is Completed.

   oc get pod -A -l 'name in (nvidia-devicequery)'
   

   Example output

   NAME                  READY     STATUS      RESTARTS   AGE
   nvidia-devicequery-ppkd4      0/1       Completed   0          36s
   

4. Describe the pod to see how the GPU device plug-in scheduled the pod.

   • In the Limits and Requests fields, see that the resource limit that you specified matches the request that the device plug-in automatically set.

   • In the events, verify that the pod is assigned to your GPU worker node.

     oc describe pod nvidia-devicequery-ppkd4
     

     Example output

     NAME:           nvidia-devicequery-ppkd4
     Namespace:      default
     ...
     Limits:
         nvidia.com/gpu:  1
     Requests:
         nvidia.com/gpu:  1
     ...
     Events:
     Type    Reason                 Age   From                     Message
     ----    ------                 ----  ----                     -------
     Normal  Scheduled              1m    default-scheduler        Successfully assigned nvidia-devicequery-ppkd4 to 10.xxx.xx.xxx
     ...
     

5. To verify that the job used the GPU to compute its workload, you can check the logs.

   oc logs nvidia-devicequery-ppkd4
   

   Example output

   /cuda-samples/sample Starting...
   
   CUDA Device Query (Runtime API) version (CUDART static linking)
   
   Detected 1 CUDA Capable device(s)
   
   Device 0: "Tesla P100-PCIE-16GB"
   CUDA Driver Version / Runtime Version          11.4 / 11.7
   CUDA Capability Major/Minor version number:    6.0
   Total amount of global memory:                 16281 MBytes (17071734784 bytes)
   (056) Multiprocessors, (064) CUDA Cores/MP:    3584 CUDA Cores
   GPU Max Clock rate:                            1329 MHz (1.33 GHz)
   Memory Clock rate:                             715 Mhz
   Memory Bus Width:                              4096-bit
   L2 Cache Size:                                 4194304 bytes
   Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
   Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
   Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
   Total amount of constant memory:               65536 bytes
   Total amount of shared memory per block:       49152 bytes
   Total shared memory per multiprocessor:        65536 bytes
   Total number of registers available per block: 65536
   Warp size:                                     32
   Maximum number of threads per multiprocessor:  2048
   Maximum number of threads per block:           1024
   Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
   Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
   Maximum memory pitch:                          2147483647 bytes
   Texture alignment:                             512 bytes
   Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
   Run time limit on kernels:                     No
   Integrated GPU sharing Host Memory:            No
   Support host page-locked memory mapping:       Yes
   Alignment requirement for Surfaces:            Yes
   Device has ECC support:                        Enabled
   Device supports Unified Addressing (UVA):      Yes
   Device supports Managed Memory:                Yes
   Device supports Compute Preemption:            Yes
   Supports Cooperative Kernel Launch:            Yes
   Supports MultiDevice Co-op Kernel Launch:      Yes
   Device PCI Domain ID / Bus ID / location ID:   0 / 175 / 0
   Compute Mode:
   < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
   
   deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.7, NumDevs = 1
   Result = PASS
   

   In this example, you see a GPU was used to execute the job because the GPU was scheduled in the worker node. If the limit is set to 2, only 2 GPUs are shown.

Now that you deployed a test GPU workload, you might want to set up your cluster to run a tool that relies on GPU processing, such as IBM Maximo Visual Inspection.