Monitoring cluster health

Choosing a monitoring solution

Metrics help you monitor the health and performance of your clusters. You can use the standard Kubernetes and container runtime features to monitor the health of your clusters and apps.

Every Kubernetes master is continuously monitored by IBM. IBM Cloud Kubernetes Service automatically scans every node where the Kubernetes master is deployed for vulnerabilities that are found in Kubernetes and OS-specific security fixes. If vulnerabilities are found, IBM Cloud Kubernetes Service automatically applies fixes and resolves vulnerabilities on behalf of the user to ensure master node protection. You are responsible for monitoring and analyzing the logs for the rest of your cluster components.

To avoid conflicts when using metrics services, be sure that clusters across resource groups and regions have unique names.

IBM Cloud® Monitoring

Gain operational visibility into the performance and health of your apps and your cluster by deploying a Monitoring agent to your worker nodes. The agent collects pod and cluster metrics, and sends these metrics to IBM Cloud Monitoring. For more information about IBM Cloud Monitoring, see the service documentation. To set up the Monitoring agent in your cluster, see Viewing cluster and app metrics with IBM Cloud Monitoring.

Kubernetes dashboard

The Kubernetes dashboard is an administrative web interface where you can review the health of your worker nodes, find Kubernetes resources, deploy containerized apps, and troubleshoot apps with logging and monitoring information. For more information about how to access your Kubernetes dashboard, see Launching the Kubernetes dashboard for IBM Cloud Kubernetes Service.

Forwarding cluster and app metrics to IBM Cloud Monitoring

The following steps are deprecated. For the latest steps, see Setting up the Monitoring agent.

Use the IBM Cloud Kubernetes Service observability plug-in to create a monitoring configuration for IBM Cloud Monitoring in your cluster, and use this monitoring configuration to automatically collect and forward metrics to IBM Cloud Monitoring.

With IBM Cloud Monitoring, you can collect cluster and pod metrics, such as the CPU and memory usage of your worker nodes, incoming and outgoing HTTP traffic for your pods, and data about several infrastructure components. In addition, the agent can collect custom application metrics by using either a Prometheus-compatible scraper or a statsd facade.

Considerations for using the IBM Cloud Kubernetes Service observability plug-in:

• You can have only one monitoring configuration for IBM Cloud Monitoring in your cluster at a time. If you want to use a different IBM Cloud Monitoring service instance to send metrics to, use the ibmcloud ob monitoring config replace command.
• If you created a Monitoring configuration in your cluster without using the IBM Cloud Kubernetes Service observability plug-in, you can use the ibmcloud ob monitoring agent discover command to make the configuration visible to the plug-in. Then, you can use the observability plug-in commands and functionality in the IBM Cloud console to manage the configuration.

Before you begin:

• Verify that you are assigned the Editor platform access role and Manager server access role for IBM Cloud Monitoring.
• Verify that you are assigned the Administrator platform access role and the Manager service access role for all Kubernetes namespaces in IBM Cloud Kubernetes Service to create the monitoring configuration. To view a monitoring configuration or launch the Monitoring dashboard after the monitoring configuration is created, users must be assigned the Viewer platform access role and Reader service access role for the ibm-observe Kubernetes namespace in IBM Cloud Kubernetes Service.
• If you want to use the CLI to set up the monitoring configuration:
  • Install the IBM Cloud Kubernetes Service observability CLI plug-in (ibmcloud ob).
  • Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To set up a monitoring configuration for your cluster:

1. Create an IBM Cloud Monitoring service instance and note the name of the instance. The service instance must belong to the same IBM Cloud account where you created your cluster, but can be in a different resource group and IBM Cloud region than your cluster.

2. Set up a monitoring configuration for your cluster. When you create the monitoring configuration, a Kubernetes namespace ibm-observe is created and a Monitoring agent is deployed as a Kubernetes daemon set to all worker nodes in your cluster. This agent collects cluster and pod metrics, such as the worker node CPU and memory usage, or the amount incoming and outgoing network traffic to your pods.

   In the console.

   1. From the Kubernetes clusters console, select the cluster for which you want to create a Monitoring configuration.
   2. On the cluster Overview page, click Connect.
   3. Select the region and the IBM Cloud Monitoring service instance that you created earlier, and click Connect.

   In the CLI.

   1. Create the Monitoring configuration. When you create the Monitoring configuration, the access key that was last added is retrieved automatically. If you want to use a different access key, add the --sysdig-access-key <access_key> option to the command.

      To use a different service access key after you created the monitoring configuration, use the ibmcloud ob monitoring config replace command.

      Version 1.30 and later: If your cluster has outbound traffic protection enabled, you must set up monitoring by using the private endpoint. To do this, specify the --private-endpoint option.

      ibmcloud ob monitoring config create --cluster <cluster_name_or_ID> --instance <Monitoring_instance_name_or_ID> [--private-endpoint]
      

      Example output

      Creating configuration...
      OK
      

   2. Verify that the monitoring configuration was added to your cluster.

      ibmcloud ob monitoring config list --cluster <cluster_name_or_ID>
      

      Example output

      Listing configurations...
      
      OK
      Instance Name                Instance ID                            CRN   
      IBM Cloud Monitoring-aaa     1a111a1a-1111-11a1-a1aa-aaa11111a11a   crn:v1:prod:public:sysdig:us-south:a/a11111a1aaaaa11a111aa11a1aa1111a:1a111a1a-1111-11a1-a1aa-aaa11111a11a::  
      

3. Optional: Verify that the Monitoring agent was set up successfully.

   1. If you used the console to create the Monitoring configuration, log in to your cluster.

   2. Verify that the daemon set for the Monitoring agent was created and all instances are listed as AVAILABLE.

      kubectl get daemonsets -n ibm-observe
      

      Example output

      NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
      sysdig-agent   9         9         9       9            9           <none>          14m
      

      The number of daemon set instances that are deployed equals the number of worker nodes in your cluster.

   3. Review the ConfigMap that was created for your Monitoring agent.

      kubectl describe configmap -n ibm-observe
      

4. Access the metrics for your pods and cluster from the Monitoring dashboard.

   1. From the Kubernetes clusters console, select the cluster that you configured.
   2. On the cluster Overview page, click Launch. The Monitoring dashboard opens.
   3. Review the pod and cluster metrics that the Monitoring agent collected from your cluster. It might take a few minutes for your first metrics to show.

5. Review how you can work with the Monitoring dashboard to further analyze your metrics.

Viewing cluster states

Review the state of a Kubernetes cluster to get information about the availability and capacity of the cluster, and potential problems that might occur.

To view information about a specific cluster, such as its zones, service endpoint URLs, Ingress subdomain, version, and owner, use the ibmcloud ks cluster get --cluster <cluster_name_or_ID> command. Include the --show-resources option to view more cluster resources such as add-ons for storage pods or subnet VLANs for public and private IPs.

You can review information about the overall cluster, the IBM-managed master, and your worker nodes. To troubleshoot your cluster and worker nodes, see Troubleshooting clusters.

• Cluster states.
• Worker node states.

Setting up IBM Cloud® Monitoring alerts

When you set up alerts, make sure to allow your cluster enough time to self-heal. Because Kubernetes has self healing capabilities, configure your alerts only for the issues that arise over time. By observing your cluster over time, you can learn which issues Kubernetes can resolve itself and which issues require alerts to avoid downtime.

On 15 June 2022, the naming convention for IBM Cloud® Monitoring alerts is change to a Prometheus compatible format. For more information, see the Sysdig release notes, Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics, and Mapping Classic Metrics with PromQL.

Depending on the size of your cluster, consider setting up alerts on the following levels:

• Apps
• Worker nodes
• Cluster
• Zone
• Account
• Block Storage for VPC

Set up autorecovery on your worker nodes to enable your cluster to automatically resolve issues.

Monitoring worker node health in with Autorecovery

The Autorecovery system uses various checks to query worker node health status. If Autorecovery detects an unhealthy worker node based on the configured checks, Autorecovery triggers a corrective action like rebooting a VPC worker node or reloading the operating system in a classic worker node. Only one worker node undergoes a corrective action at a time. The worker node must complete the corrective action before any other worker node undergoes a corrective action.

Autorecovery requires at least one healthy worker node to function properly. Configure Autorecovery with active checks only in clusters with two or more worker nodes.

Before you begin:

• Ensure that you have the following IBM Cloud IAM roles:
  • Administrator platform access role for the cluster
  • Writer or Manager service access role for the kube-system namespace
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To configure Autorecovery:

1. Follow the instructions to install the Helm version 3 client on your local machine.

2. Create a configuration map file that defines your checks in JSON format. For example, the following YAML file defines three checks: an HTTP check and two Kubernetes API server checks. Refer to the component descriptions and the health check component table for information about the three kinds of checks and information about the individual components of the checks.

   Define each check as a unique key in the data section of the configuration map.

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: ibm-worker-recovery-checks
     namespace: kube-system # The `kube-system` namespace is a constant and can't be changed.
   data:
     checknode.json: |
       {
         "Check":"KUBEAPI",
         "Resource":"NODE",
         "FailureThreshold":3,
         "CorrectiveAction":"RELOAD",
         "CooloffSeconds":1800,
         "IntervalSeconds":180,
         "TimeoutSeconds":10,
         "Enabled":true
       }
     checkpod.json: |
       {
         "Check":"KUBEAPI",
         "Resource":"POD",
         "PodFailureThresholdPercent":50,
         "FailureThreshold":3,
         "CorrectiveAction":"RELOAD",
         "CooloffSeconds":1800,
         "IntervalSeconds":180,
         "TimeoutSeconds":10,
         "Enabled":true
       }
     checkhttp.json: |
       {
         "Check":"HTTP",
         "FailureThreshold":3,
         "CorrectiveAction":"REBOOT",
         "CooloffSeconds":1800,
         "IntervalSeconds":180,
         "TimeoutSeconds":10,
         "Port":80,
         "ExpectedStatus":200,
         "Route":"/myhealth",
         "Enabled":false
       }
   

3. Create the configuration map in your cluster.

   kubectl apply -f ibm-worker-recovery-checks.yaml
   

4. Verify that you created the configuration map with the name ibm-worker-recovery-checks in the kube-system namespace with the proper checks.

   kubectl -n kube-system get cm ibm-worker-recovery-checks -o yaml
   

5. Deploy Autorecovery into your cluster by installing the ibm-worker-recovery Helm chart.

   helm install ibm-worker-recovery iks-charts/ibm-worker-recovery --namespace kube-system
   

6. After a few minutes, you can check the Events section in the output of the following command to see activity on the Autorecovery deployment.

   kubectl -n kube-system describe deployment ibm-worker-recovery
   

7. If you don't see activity on the Autorecovery deployment, you can check the Helm deployment by running the tests that are in the Autorecovery chart definition.

   helm test ibm-worker-recovery -n kube-system