Streaming metrics to a Kafka service

You can use IBM Cloud® Monitoring to push a set of selected metrics to a Kafka service such as Event Streams.

You will need to configure:

The metrics to export within a defined scope.

The metric names and scope are specified in PromQL format.
The granularity of the data.

For example, 10 seconds (10s) or 1 minute (1m). The recommended aggregation is 1m.
The Kafka topic to be used by IBM Cloud Monitoring to export the data.

Prereqs

Check the following prereqs before managing streaming:

Create an Event Streams instance..
Check the limitations of the service plans. For more information, see Limits and quotas.
Make sure you have the appropriate permissions.
- You must have the manager role to configure streaming.
- When you configure streaming, the IBM Cloud Monitoring instance and the Event Streams instance must be provisioned in the same account.
- To connect the IBM Cloud Monitoring instance to the Event Streams instance, you need the following information:
  - You must have the manager role to configure streaming in the IBM Cloud Monitoring instance. This role includes the sysdig-monitor.launch.admin IAM action role that allows a user to perform admin tasks such as configure streaming.
  - When you configure streaming, the Monitoring instance and the Event Streams instance must be provisioned in the same account.
  - To connect the Monitoring instance to the Event Streams instance, you need the following information:
    - Endpoint URLs to call the APIs
    - Credentials for authentication
- To create a topic in Event Streams, you must have the manager role. This role includes the messagehub.topic.manage IAM action role that allows an app or user to create or delete topic.
- The credential that Monitoring uses to publish data in Event Streams must have the writer role. This role includes the messagehub.topic.write IAM action role that allows an app or service to write data to 1 or more topics.
If you configure the account to restrict access to configured IP addresses via IAM settings, or if the account limits the network locations that connections are accepted from via context based restrictions rules (CBR) for the Event Streams service, you must allowlist the Monitoring IPs in the account. For more information, see Monitoring endpoints and Event Streams - Restricting network access.

The instructions on this topic are based on using an Enterprise plan.

Manage streaming through the UI

Configure streaming

Complete the following steps to configure streaming through the UI:

Step 1. Create a topic in Event Streams

Complete the following steps to create an Event Streams topic:

Log in to your IBM Cloud account.

After you log in with your user ID and password, the IBM Cloud dashboard opens.
Click the Menu icon > Resource list.
Look for the Event Streams instance that you plan to use, and select it.
In the Event Streams instance console, click Manage > Launch Dashboard.
Click Create a topic.

Create a topic
Enter a topic name and click Next.

Enter topic name
Enter the number of partitions and click Next.

One or more partitions make up a topic. A partition is an ordered list of messages. Partitions are distributed across the brokers in order to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.

Enter number of partitions
Select a Message retention and click Create Topic.

Message retention defines how long messages are retained before they are deleted. If your messages are not read by a consumer within this time, they will be missed.

Step 2. Create credentials to authenticate the monitoring instance with Event Streams

You need the following information to connect the Monitoring instance to the Event Streams instance:

Endpoint URLs to call the APIs
Credentials for authentication

Complete the following steps to create service credentials that the Monitoring instance needs to communicate with the Event Streams instance:

In the IBM Cloud, click the Menu icon > Resource list.
Look for the Event Streams instance that you plan to use, and select it.
In the Event Streams console, click Service credentials.
Select New credential.
Enter a name and select the writer role.

Create a credential
Click Add.

To restrict access to 1 topic, complete the following steps:

From the menu bar, click Manage > Access (IAM), and select Service IDs.

Select Service IDs
Select the service ID.
Select Access policies.
Select the policy and modify it to specify the topic.

Edit policy

Modify policy
Get credentials using the IBM Cloud CLI and make note of the api key and broker URL values.

Step 3. Configure the connection in Monitoring to Event Streams

To configure IBM Cloud Monitoring metric data streaming, do the following:

Launch the web UI as a user with administrator authority. For more information on how to launch the Web UI, see Navigating to the Web UI.
Click the user icon. This is the icon with the initials of the logged on user. Then click Settings.
Select Metric Data Streaming.
Click Add Integration. The New Metric Streaming Integration panel is displayed.

Metrics Streaming integration panel
Specify the following information:

Integration Name

The name of the streaming configuration.

Topic

The name of the Kafka topic to receive the metrics.

Brokers

A list of Kafka brokers separated by commas, for example kafka-2.mh-svc.eu-de.containers.appdomain.cloud:9093,kafka-1.mh-svc.eu-de.containers.appdomain.cloud:9093,kafka-0.mh-svc.eu-de.containers.appdomain.cloud:9093. Notice that you must not enclose within " the URLs.

TLS

The Transport Layer Security (TLS) method. Enable TLS to encrypt and protect the transfer of data.

Credentials

The credentials used by Monitoring to to export the data from the Monitoring instance to the Kafka server. You can choose user and password, or a base64 encoded certificate.

Credential options

Compression

Choose the type of compression. Valid options are: LZ4, Snappy, Gzip, and Zstandard. For more information, see Message compression in Apache Kafka.
Click Test Connection to verify the connection between IBM Cloud Monitoring and Kafka has been successfully configured.

You must test the connection before saving and enable the configuration.
Select the Scope and define the Metrics to be exported.

For Scope, select one or more labels to filter the defined metrics.

Metrics must specify the complete metric name using PromQL.
Click Save.

After you save the confuguration, you must wait at least 30 minutes before the streaming is active.

Enable or stop streaming through the UI

You can start and stop streaming of metrics from the Metric Data Streaming page:

Launch the web UI as a user with administrator authority. For more information on how to launch the Web UI, see Navigating to the Web UI.
Click the user icon. This is the icon with the initials of the logged on user. Then click Settings.
Select Metric Data Streaming. The list of streaming configurations is displayed.

Streaming configurations
Enable or disable a streaming configuration through the toggle.

Manage streaming by using the API

Complete the following steps to manage streaming through the UI:

List the streaming configurations

Use this method to get the details for all the existing streaming configurations.

curl --request GET https://<ENDPOINT>/api/v1/dataStream/configs
 -H "content-type: application/json"
 -H "Authorization: <TOKEN>"

Where

<ENDPOINT>indicates the endpoint targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
<TOKEN> API token that you use to authenticate with the Monitoring service. You must include Bearer with the token. For more information, see Working with Monitoring API tokens.

The response will be similar to the following:

{"dataStreamConfigs":[{"uuid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","enabled":true,"name":"Streaming metrics - demo","type":"kafka"}]}

Get details for a streaming configuration

Use this method to get the details for 1 streaming configuration.

curl --request GET https://<ENDPOINT>/api/v1/dataStream/configs/<STREAMING_CONFIG_ID>
 -H "content-type: application/json"
 -H "Authorization: <TOKEN>"

Where

<ENDPOINT> represents the endpoint that is targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
<TOKEN> API token that you use to authenticate with the Monitoring service. You must include Bearer with the token. For more information, see Working with Monitoring API tokens.
<STREAMING_CONFIG_ID> represents the UUID of the streaming configuration for which you want to get details.

The response will be similar to the following:

{
    "dataStreamConfig": {
        "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "enabled": true,
        "name": "Streaming metrics - demo",
        "emitterType": "kafka",
        "emitterProperties": {
            "bootstrap.servers": "kafka-0.mh-svc1.eu-de.containers.appdomain.cloud:9093, kafka-2.mh-svc2-0000.eu-de.containers.appdomain.cloud:9093,kafka-1.mh-svc3.eu-de.containers.appdomain.cloud:9093",
            "topic.name": "my-monitoring-topic",
            "compression.type": "lz4",
            "security.protocol": "SASL_SSL",
            "sasl.mechanism": "PLAIN",
            "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"token\" password=\"xxxxxxxxxxxxx\";","ssl.truststore.base64.content":""}},"dataStreamRules":[{"uuid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","enabled":true,"metrics":["kube_daemonset_labels","kube_node_status_condition"],"aggregation":"avg","aggregationSec":10,"scope":"kubernetes.cluster.name = \"mycluster\"",
            "dataStreamConfigUuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        }
    ]
}

Create a streaming configuration

Use this method to configure 1 streaming configuration that forwards a set of metrics to Event Streams.

curl --request POST https://<ENDPOINT>/api/v1/dataStream/configs
 -H "content-type: application/json"
 -H "Authorization: <TOKEN>"
 -d @streaming-config.json

Where

<ENDPOINT> represents the endpoint that is targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
<TOKEN> API token that you use to authenticate with the Monitoring service. You must include Bearer with the token. For more information, see Working with Monitoring API tokens.

A sample streaming-config.json file looks as follows:

{
    "dataStreamConfig": {
        "enabled": true,
        "name": "Streaming metrics config",
        "emitterType": "kafka",
        "emitterProperties": {
            "bootstrap.servers": "kafka-0.mh-svc1.eu-de.containers.appdomain.cloud:9093,kafka-2.mh-svc2.eu-de.containers.appdomain.cloud:9093,kafka-1.mh-svc3.eu-de.containers.appdomain.cloud:9093",
            "topic.name": "my-monitoring-topic",
            "compression.type": "lz4",
            "security.protocol": "SASL_SSL",
            "sasl.mechanism": "PLAIN",
            "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"token\" password=\"xxxxxxxxxx\";",
            "ssl.truststore.base64.content": ""
        }
    },
    "dataStreamRules": [
        {
            "enabled": true,
            "metrics": [
                "kube_daemonset_labels",
                "kube_node_status_condition"
            ],
            "aggregation": "avg",
            "aggregationSec": 10,
            "scope": "kubernetes.cluster.name = \"mycluster\""
        }
    ]
}

Update a streaming configuration

Update a streaming configuration in any of the following siuations:

The API key to authenticate with Event Streams is rotated, compromissed or needs changing.
When you want to change the metrics that you stream, by either adding new metrics or removing metrics.
When you must change the scope.
To enable or disable data streaming for the configuration.

Use this method to update 1 streaming configuration that forwards a set of metrics to Event Streams.

curl --request PUT https://<ENDPOINT>/api/v1/dataStream/configs/<STREAMING_CONFIG_ID>
 -H "content-type: application/json"
 -H "Authorization: <TOKEN>"
 -d @streaming-config.json

Where

<ENDPOINT> represents the endpoint that is targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
<TOKEN> API token that you use to authenticate with the Monitoring service. You must include Bearer with the token. For more information, see Working with Monitoring API tokens.
<STREAMING_CONFIG_ID> represents the UUID of the streaming configuration.

A sample streaming-config.json file looks as follows:

{
    "dataStreamConfig": {
        "uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "enabled": true,
        "name": "Streaming metrics - demo",
        "emitterType": "kafka",
        "emitterProperties": {
            "bootstrap.servers": "kafka-0.mh-svc1.eu-de.containers.appdomain.cloud:9093, kafka-2.mh-svc2-0000.eu-de.containers.appdomain.cloud:9093,kafka-1.mh-svc3.eu-de.containers.appdomain.cloud:9093",
            "topic.name": "my-monitoring-topic",
            "compression.type": "lz4",
            "security.protocol": "SASL_SSL",
            "sasl.mechanism": "PLAIN",
            "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"token\" password=\"xxxxxxxxxxxxx\";","ssl.truststore.base64.content":""}},"dataStreamRules":[{"uuid":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","enabled":true,"metrics":["kube_daemonset_labels","kube_node_status_condition"],"aggregation":"avg","aggregationSec":10,"scope":"kubernetes.cluster.name = \"mycluster\"",
            "dataStreamConfigUuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
        }
    ]
}

Delete a streaming configuration

Use this method to delete 1 streaming configuration that forwards a set of metrics to Event Streams.

curl --request DELETE https://<ENDPOINT>/api/v1/dataStream/configs/<STREAMING_CONFIG_ID>
 -H "content-type: application/json"
 -H "Authorization: <TOKEN>"

Where

<ENDPOINT> represents the endpoint that is targetted by the REST API call. For more information, see Monitoring REST API endpoints. For example, the public endpoint for an instance that is available in us-south is the following: https://us-south.monitoring.cloud.ibm.com/api
<TOKEN> API token that you use to authenticate with the Monitoring service. You must include Bearer with the token. For more information, see Working with Monitoring API tokens.
<STREAMING_CONFIG_ID> represents the UUID of the streaming configuration.

Monitoring streaming by using IBM Cloud Monitoring

Event Streams is integrated with the Monitoring service. Monitoring provides a default template that you can customize to monitor the Event Streams instance, how data is streamed out of Monitoring and consumed by any application or service that is subscribed to Event Streams.

Complete the following steps to monitor the Event Streams instance:

Check that you have an instance of the Monitoring in the same region as your Event Streams instance. This instance must be configured to collect platform metrics. For more information, see Enabling platform metrics.
Launch the Monitoring UI.
In the Dashboards section, go to Dashboard templates and select the template IBM Event Streams (Enterprise).
Create a copy of the temnplate by clicking Create custom dashboard.

You can use the metric Topic bytes in per second to see how data is sent by Monitoring to Event Streams.

You can use the metric Topic bytes out per second to see how data is consumed by any application or service that is subscribed to Event Streams.
(Optional) Edit the panel Topic bytes in per second.

Edit panel

Then, customize the metric to see data per topic.

Customize metric

Check that the resolution is set to 10M.