Monitoring a Red Hat OpenShift cluster

Use this tutorial to learn how to configure a Red Hat® OpenShift® cluster to forward metrics to the IBM Cloud® Monitoring service. You can monitor clusters in IBM Cloud, on-prem, and in other clouds.

To configure a cluster to forward metrics, you must install a monitoring agent onto each worker node in your Red Hat OpenShift cluster by using a DaemonSet. The monitoring agent uses an access key (token) to authenticate with the IBM Cloud Monitoring instance. The monitoring agent acts as a data collector. It automatically collects metrics such as worker node CPU and worker node memory usage, HTTP traffic into and out of your containers, and data about several infrastructure components. In addition, the agent can collect custom application metrics by using either a Prometheus-compatible scraper or a StatsD facade.

Components overview on the IBM Cloud — Components overview

For example, to configure your Red Hat OpenShift cluster to forward metrics to your IBM Cloud Monitoring instance, you can deploy the agent by using Helm or a script:

Helm
Script

The Monitoring agent automatically collects the following types of system metrics per host:

System hosts metrics provide information about CPU, memory, and storage usage metrics, that you can use to analyze the performance and resource utilization of all your processes.
File and File System metrics provide information about files and file system that you can use to analyze file interactions that occur in your system. For example, you can find information about your open files, bytes going in and out, or the percentage of usage of a given file system.
Process metrics provide information about the processes that run in your servers. For example, you can use these metrics to explore the number of processes, or get client or server information.
Network metrics provide information about the network. They offer insight to the connections that are established between your applications, containers, and servers. For example, you can find information about the bytes that are being sent or received, or the number of HTTP requests, connections, and latency. In addition, for SQL or MongoDB, the agent collects additional information when it is configured in troubleshooting mode.

The Monitoring agent automatically collects the following types of metrics per Red Hat OpenShift cluster:

State metrics: Kube state metrics report on the health and state of the various objects that run inside Red Hat OpenShift components, such as deployments, nodes and pods. To see the list of metrics that are collected by default, see Red Hat OpenShift State.
Resource usage metrics: Resource usage metrics reports on the health and state of CPU and memory for workers (nodes) and pods that are running in the cluster. The data can be analyzed by namespace, by worker, by pod, by workload object such as deployments, daemonSets, and more.

For a list of collected metrics, see Metrics Available for orchestrated environments.

Through the Monitoring UI, you can analyze data in the Advisor tab, the Explore tab, and in the Dashboard tab. You monitor the data through metric views and dashboards.

Consider the following information when monitoring your data:

In the Explorer tab, you can monitor individual metrics.
In the Advisor tab, you can monitor Red Hat OpenShift or host level metrics.

This tab is only available for users that belong to a team that has access to monitor Red Hat OpenShift or host level metrics.
In the Dashboard tab, you can monitor through panels predefined dashboards or custom ones and get a specialized insight into network data, application data, topology, services, hosts, and containers. A panel displays a metric or group of metrics in a dashboard.

For each metric view and dashboard, you can define the scope of the data, how to aggregate data, and what time and group filters to apply to the data. For more information, see Managing panels.

You can configure a dashboard as the default entry point for a team, unifying a team's experience, and allowing users to focus their immediate attention on the most relevant information for them.

For more information, see Viewing metrics.

Objectives

In this tutorial, you configure metrics for your Red Hat OpenShift on IBM Cloud cluster. In particular, you:

Provision an IBM Cloud Monitoring instance.
Configure the monitoring agent in your cluster to send metrics.
Use the monitoring UI to analyze your cluster metrics.

Before you begin

Read about Monitoring.
Have a user ID that is a member or an owner of an IBM Cloud account. To get an IBM Cloud user ID, go to: Registration.
Get information about monitoring agent images.
Check resource requirements.
Install the IBM Cloud CLI and plug-ins:
- IBM Cloud CLI (ibmcloud)
- IBM Cloud Kubernetes Service plug-in (ibmcloud ks)
- IBM Cloud Container Registry plug-in (ibmcloud cr)
- IBM Cloud Kubernetes Service observability plug-in (ibmcloud ob)
Create a cluster or use an existing Red Hat OpenShift on IBM Cloud cluster.
Make sure that your user ID is assigned the following IBM Cloud® Identity and Access Management policies:

List of IAM policies required to complete the tutorial
Resource	Scope of the access policy	Role	Region	Information
Resource group default	Resource group	Viewer	us-south	This policy is required to allow the user to see service instances in the Default resource group.
IBM Cloud Monitoring service	Resource group	Editor	us-south	This policy is required to allow the user to provision and administer the IBM Cloud Monitoring service in the default resource group.
Red Hat OpenShift cluster instance	Resource	Editor	us-south	This policy is required to configure the secret and the monitoring agent in the Red Hat OpenShift on IBM Cloud cluster.

For more information about the IBM Cloud® Kubernetes Service IAM roles, see User access permissions.

Step 1. Provision an IBM Cloud Monitoring instance

In this getting started tutorial, instructions are provided to provision an instance of the IBM Cloud Monitoring in the us-south region. For more information about supported regions, see Regions.

To provision an instance of IBM Cloud Monitoring, complete the following steps:

Log in to your IBM Cloud account.

After you log in with your user ID and password, the IBM Cloud UI opens.
Click Catalog. The list of the services that are available in IBM Cloud opens.
To filter the list of services that is displayed, select the Logging and monitoring category.
Click the IBM Cloud Monitoring tile.
Select a location and a service plan.

By default, the Lite plan is set.

For more information about other service plans, see Pricing plans.
Configure the resource.

Enter a name for the service instance.

Select a resource group.

Optionally add tags.
Click Create.

After you provision an instance, the Observability dashboard opens and shows details for your Monitoring instances.

To provision an instance through the CLI, see Provisioning an instance through the IBM Cloud CLI.

Step 2. Configure your Red Hat OpenShift on IBM Cloud cluster to send metrics to your instance

To configure your Red Hat OpenShift on IBM Cloud cluster to send metrics to your IBM Cloud Monitoring instance, you must install a monitoring agent pod on each node of your cluster. The monitoring agent is installed using a DaemonSet which ensures an instance of the agent is running on every worker node. The monitoring agent collects metrics from the pod where it is installed, and forwards the data to your instance.

In order to provide the full suite of system metrics, the monitoring agent needs to have a privileged status.

To configure your Red Hat OpenShift on IBM Cloud cluster to forward metrics to your IBM Cloud Monitoring instance, complete the following steps from the command-line.

Set the cluster context and log in to the cluster

Complete the following steps:

Open a terminal to log in to IBM Cloud.
```
ibmcloud login -a cloud.ibm.com
```
Select the account where you provisioned the IBM Cloud Monitoring instance.
List the clusters to find out in which region and resource group the cluster is available.
```
ibmcloud oc clusters
```
Set the resource group and region.
```
ibmcloud target -g RESOURCE_GROUP -r REGION
```
Where

RESOURCE_GROUP is the name of the resource group where the cluster is available, for example, default.

REGION is the region where the cluster is available, for example, us-south.

Set the cluster context in your session.

ibmcloud oc cluster config --cluster <cluster_name_or_ID>

Install the IBM Cloud Monitoring agent in your cluster

Run the following command for your public or private endpoint.

Private endpoints
```
curl -sL https://ibm.biz/install-sysdig-k8s-agent | bash -s -- -a <MONITORING_ACCESS_KEY> -c ingest.private.us-south.monitoring.cloud.ibm.com -ac 'sysdig_capture_enabled: false' --openshift
```
Public endpoints
```
curl -sL https://ibm.biz/install-sysdig-k8s-agent | bash -s -- -a <MONITORING_ACCESS_KEY> -c ingest.us-south.monitoring.cloud.ibm.com -ac 'sysdig_capture_enabled: false' --openshift
```
Where MONITORING_ACCESS_KEY is the ingestion key for the instance.

By default the slim agent is installed. The slim agent reduces the possible are of attack for potential vulnerabilities and, as a result, is more secure. If installing the full agent is desired, add the -af option to the curl command.
Verify that the monitoring agent is created successfully and its status. Run the following command:
```
oc get pods -n ibm-observe
```
The deployment is successful when you see one or more sysdig-agent pods. The number of sysdig-agent pods equals the number of worker nodes in your cluster. All pods must be in a Running state.

Step 3. Launch the monitoring UI

To launch the monitoring UI through the IBM Cloud console, complete the following steps.

Log in to your IBM Cloud account.

After you log in with your user ID and password, the IBM Cloud Dashboard opens.
From the menu , select Observability.
Select Monitoring. The list of instances that are available on IBM Cloud is displayed.
Find your instance and click Open dashboard. The web UI opens.

It may take some time before you see the cluster entry while the information is initally collected and processed by the monitoring agent.

You only can monitor one instance per browser. You could have multiple tabs for the same instance.

Step 4. Monitor your cluster

In the Advisor tab, you can monitor and troubleshoot the health, risk, and capacity of hosts, and Kubernetes and Red Hat OpenShift clusters.

Data is refreshed every 10 minutes.
Metrics are prioritized by event count and severity.
For more information, see Advisor.

In the Advisor section, you can choose to monitor your Red Hat OpenShift clusters by cluster, by node, by namespace, or by workload. Each option offers a set of predefined dashboards that you can use to monitor the health of your resources. You can also select to monitor by host.

Monitoring Red Hat OpenShift clusters by cluster

When you choose to monitor your clusters by cluster, you can select more filters to display data by node or by namespace, or you can choose any of the following dashboards:

Workload Status & Performance
Node Status & Performance
Pod Rightsizing & Workload Capacity Optimization
Cluster Capacity Planning
Cluster / Namespace Available Resources
Cluster Overview
CPU Allocation Optimization
Memory Allocation Optimization

Advisor predefined dashboards by cluster

For more information on how to interpret this view, see About Clusters Overview.

Monitoring clusters by node

When you choose to monitor your clusters by node, you can choose any of the following dashboards:

Node Status & Performance
Pod Scheduling Troubleshooting
Node Overview
CPU Allocation Optimization
Memory Allocation Optimization

For more information on how to interpret this view, see About Nodes Overview.

Monitoring clusters by namespace

When you choose to monitor your clusters by namespace, you can select more filters to display data by workload, or you can choose any of the following dashboards:

Workload Status & Performance
Pod Status & Performance
Pod Rightsizing & Workload Capacity Optimization
Namespace Overview
Workloads CPU Usage and Allocation
Workloads Memory Usage and Allocation

For more information on how to interpret this view, see About Namespaces Overview.

Monitoring clusters by workloads

When you choose to monitor your clusters by workloads, you can choose any of the following dashboards:

Container Resource Usage & Troubleshooting
Pod Status & Performance
Pod Rightsizing & Workload Capacity Optimization
Workload Status & Performance
Deployment Overview
Pod Overview
Workloads CPU Usage and Allocation
Workloads Memory Usage and Allocation

For more information on how to interpret this view, see About Workloads Overview.

Next steps

Create a custom dashboard. For more information, see Working with dashboards.
Learn about alerts. For more information, see Working with alerts.
Learn how to manage logs from your cluster. See Logging with Red Hat OpenShift clusters.
Learn about the IBM Cloud Monitoring Workload Protection functionality to find and prioritize software vulnerabilities, detect and respond to threats, and manage configurations, permissions and compliance from source to run. See Getting started with IBM Cloud® Security and Compliance Center Workload Protection.