Quick setup guide for Event Streams for IBM Cloud

Step 2: Provision an Event Streams instance by using the console

1. Log in to the IBM Cloud console.

2. Click the Event Streams service in the Catalog.

3. Select the Lite plan, Standard plan, or Enterprise plan from the Select a pricing plan section.

4. Enter a name for your service. You can use the default value.

5. Click Create. The Event Streams Resource list page opens.

6. When your instance has been created, click on the instance name to view more information.

7. Optional. You can complete the steps in the Getting started tutorial to run a sample starter app.

Step 2: Provision an Event Streams instance by using the CLI

If it's the first time you've used the CLI, see Getting started with the CLI.

To provision an instance of Event Streams Standard Plan with the IBM Cloud CLI, complete the following steps:

1. Install the IBM Cloud CLI by completing the steps in Getting started with the IBM Cloud CLI.

2. Log in to IBM Cloud by running the following command:

   ibmcloud login -a cloud.ibm.com
   

3. Create an Event Streams instance on IBM Cloud using the Lite, Standard, or Enterprise plans.

   Select one of the following methods:

   • To create an instance from the CLI on the Enterprise plan, run the following command:

     ibmcloud resource service-instance-create <INSTANCE_NAME> messagehub enterprise-3nodes-2tb <REGION>
     

     Because the Enterprise plan has its own dedicated resources for each cluster, it requires more time for provisioning, so a new Enterprise instance might take up to 3 hours.

   • To create an instance from the CLI on the Standard plan, run the following command:

     ibmcloud resource service-instance-create <INSTANCE_NAME> messagehub standard <REGION>
     

     Provisioning a new Standard plan instance is instantaneous because the underlying resources are already set up.

Step 2: Provision an Event Streams instance by using the resource controller API

The preferred method to provision an instance is to use the CLI.

Alternatively, you can use the resource controller API. First, retrieve an access token then run a resource controller API command with the access token to create the instance.

curl -X POST https://resource-controller.cloud.ibm.com/v2/resource_instances -H "Authorization: ${token}" -H "Content-Type: application/json" \
-d '{ "name": "JG-test-curl", "target": "us-south", "resource_group":"9eba3cff1b0540b9ab7fb93829911da0", "resource_plan_id": "ibm.message.hub.enterprise.3nodes.2tb", "parameters":{"service-endpoints":"public","throughput":"150"}}'

Step 3: Create a topic and select number of partitions by using the console

For guidance about the settings that you can modify when creating topics, see topic configuration.

1. From your newly provisioned instance, navigate to Topics using the menu on the left.

2. Click the Create topic button and an enter a topic name. Click Next. Topic names are restricted to a maximum of 200 characters.

3. Select the number of partitions.

   One or more partitions make up a topic. A partition is an ordered list of messages. 1 partition is sufficient for getting started, but production systems often have more.

   Partitions are distributed across the brokers to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.

   Click Next.

4. Set the message retention period. This is how long messages are retained before they are deleted. If your messages are not read by a consumer within this time, they will be missed. The default retention period for messages is 24 hours. The minimum is 1 hour and the maximum is 30 days. Specify this value as multiples of hours.

   Click Create topic.

Step 3: Create a topic and select number of partitions by using the CLI

For guidance about the settings that you can modify when creating topics, see topic configuration.

Use the following ibmcloud es topic-create command to create a new topic with your chosen number of partitions:

ibmcloud es topic-create [--name] TOPIC_NAME [--partitions PARTITIONS] [--config KEY=VALUE[;KEY=VALUE]* ]*

Prerequisites: None

Command options:

--name value, -n value

Topic name. Topic names are restricted to a maximum of 200 characters.

--partitions value, -p value

Set the number of partitions for the topic.

One or more partitions make up a topic. A partition is an ordered list of messages. 1 partition is sufficient for getting started, but production systems often have more.

Partitions are distributed across the brokers to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.

--config KEY=VALUE, -c KEY=VALUE(optional)

Set a configuration option for the topic as a KEY=VALUE pair.

You can specify multiple --config options. Each '--config' option can specify a semicolon-delimited list of assignments. The following list shows the valid configuration keys:

• cleanup.policy
• retention.ms
• retention.bytes
• segment.bytes
• segment.ms
• segment.index.bytes

The default retention period for messages as specified by the retention.ms key is 24 hours. The minimum is 1 hour and the maximum is 30 days. Specify this value as multiples of hours.

ibmcloud es topics [--filter FILTER] [--json]

ibmcloud es cluster [--json]

Step 3: Create a topic and select number of partitions by using the Admin REST API

Event Streams provides a REST API for administration that you can use to create and list topics.

You can create a Kafka topic by issuing a POST request to the /admin/topics path. The body of the request must contain a JSON document. For example:

{
    "name": "topicname",
    "partitions": 1,
    "configs": {
        "retentionMs": 86400000,
        "cleanupPolicy": "delete"
    }
}

The JSON document must contain a name attribute, specifying the name of the Kafka topic to create. Topic names are restricted to a maximum of 200 characters.The JSON can also specify the number of partitions to assign to the topic (using the partitions property). If the number of partitions is not specified, the topic is created with a single partition.

One or more partitions make up a topic. A partition is an ordered list of messages. 1 partition is sufficient for getting started, but production systems often have more.

Partitions are distributed across the brokers to increase the scalability of your topic. You can also use them to distribute messages across the members of a consumer group.

You can also specify an optional configs object within the request. This allows you to specify the retentionMs property, which controls how long (in milliseconds) Kafka retains messages published to the topic. After this time elapses the messages are automatically deleted to free space. You must specify the value of the retentionMs property in a whole number of hours (for example, multiples of 3600000). The default retention period for messages is 24 hours. The minimum is 1 hour and the maximum is 30 days.

For guidance about the settings that you can modify when creating topics, see topic configuration.

The expected HTTP status codes are as follows:

• 202: Topic creation request was accepted.
• 400: Invalid request JSON.
• 403: Not authorized to create topic.
• 422: Semantically invalid request.

If the request to create a Kafka topic succeeds, HTTP status code 202 (Accepted) is returned. If the operation fails, an HTTP status code of 422 (Unprocessable Entity) is returned, and a JSON object containing additional information about the failure is returned as the body of the response.

curl -i -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'Authorization: Bearer ${TOKEN}' --data '{ "name": "newtopic", "partitions": 1}' ${ADMIN_URL}/admin/topics

[
  {
    "name": "topic1",
    "partitions": 1,
    "retentionMs": 86400000,
    "cleanupPolicy": "delete"
  },
  { "name": "topic2",
    "partitions": 2,
    "retentionMs": 86400000,
    "cleanupPolicy": "delete"
  }
]

curl -i -X GET -H 'Accept: application/json' -H 'Authorization: Bearer ${TOKEN}' ${ADMIN_URL}/admin/topics

Step 4: Create a service credential by using the console

To allow you to connect to your Event Streams instance, create a service key by using the IBM Cloud console:

1. Locate your Event Streams service in the Resource list.
2. Click your service tile.
3. Click Service credentials.
4. Click New credential.
5. Complete the details for your new credential like a name and role and click Add. A new credential appears in the credentials list.
6. Expand the new credential's section to reveal the details in JSON format.

Step 4: Create a service credential by using the CLI

Create a service key by using the IBM Cloud CLI, so that you can connect to your Event Streams instance:

1. Locate your service:

   ibmcloud resource service-instances
   

2. Create a service key:

   ibmcloud resource service-key-create <key_name> <key_role> --instance-name <your_service_name>
   

3. Print the service key:

   ibmcloud resource service-key <key_name>
   

   A single set of endpoint details are contained in each service key. For service instances configured to be connected to a single network type, either the IBM Cloud Public network (the default) or the IBM Cloud Private network, the service key contains the details relevant to that network type. For instances configured to support both the private and public networks, details for the public network are returned. If you want details for the private network, you must add the --service-endpoint private parameter to the previous service-key-create CLI command. For example:

   ibmcloud resource service-key-create <private-key-name> <role> --instance-name <instance-name> --service-endpoint private
   

Step 4: Create a service credential by using the CLI and the REST producer API

To connect to your Event Streams instance, the supported authentication mechanism is using a bearer token. To obtain your token by using the IBM Cloud CLI, first log in to IBM Cloud and then run the following command:

ibmcloud iam oauth-tokens

Place this token in the Authorization header of the HTTP request in the form Bearer<token>. Both API key or JWT tokens are supported.

Step 5: Produce data using the console

You cannot produce data by using the console. You can produce data using the command line, the REST Producer API, or the Kafka API.

However, you can complete the steps for the console in the Getting started tutorial to run a sample starter app and see messages flowing through a topic.

Step 5: Produce data using the command line

You can use the Event Streams Kafka console producer tool to produce data. The console tools are in the bin directory of your Kafka client download, which you can download from Apache Kafka downloads. We recommend that you download the latest available stable binary version. Kafka client versions are backwards compatible with the version of Kafka on the server.

You must provide a list of brokers (using the BOOTSTRAP_ENDPOINTS property) and SASL credentials.

To provide the SASL credentials to this tool, create a properties file based on the following example:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<user>" password="<api_key>";
    security.protocol=SASL_SSL
    sasl.mechanism=PLAIN
    ssl.protocol=TLSv1.2
    ssl.enabled.protocols=TLSv1.2
    ssl.endpoint.identification.algorithm=HTTPS

Use the <user> field from the service key as the username and the <api_key> field from the service key as the password. You can find these values in the Event Streams Service Credentials tab in the IBM Cloud console.

Event Streams provides example producer.properties and consumer.properties files for the Java client.

After you create the properties file, you can run the console producer in a terminal as follows:

   kafka-console-producer.sh --broker-list BOOTSTRAP_ENDPOINTS --producer.config CONFIG_FILE --topic TOPIC_NAME

Replace the following variables in the example with your own values:

• BOOTSTRAP_ENDPOINTS with the value from your Event Streams Service Credentials tab in the IBM Cloud console.
• CONFIG_FILE with the path of the configuration file.

• Use the <bootstrap_endpoints> field from the service key as the bootstrap.servers property of your Kafka application.
• Use the <user> field from the service key as the username and the <api_key> field from the service key as the password. Ensure that your application parses the details.

You can use many of the other options of this tool, except for those that require access to ZooKeeper. For more information, see Using Kafka console tools with Event Streams.

Step 5: Produce data using the REST Producer API

Use the v2 endpoint of the REST Producer API to send messages of type text, binary, JSON, or avro to topics. With the v2 endpoint you can use the Event Streams schema registry by specifying the schema for the avro data type.

• What are headers? Are they optional? You see headers in the message body. That’s the chunk of data that includes everything in the request or response. The headers usually come after the request line or response line. Value of colour is base64 - how to do that?

The following code shows an example of sending a message of text type by using curl:

curl -v -X POST \
-H "Authorization: Bearer $token" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{
  "headers": [
    {
      "name": "colour",
      "value": "YmxhY2s="
    }
  ],
  "key": {
    "type": "text",
    "data": "Test Key"
  },
  "value": {
    "type": "text",
    "data": "Test Value"
  }
}' \
"$kafka_http_url/v2/topics/$topic_name/records"

For more information, see the Event Streams REST Producer v2 endpoint API reference.

Step 6: Consume data using the console

You cannot consume data by using the console. You can consume data using the command line or the Kafka API.

However, you can complete the steps in the Getting started tutorial to run a sample starter app and see messages flowing through a topic.

Step 6: Consume data using the command line

You can use the Event Streams Kafka console consumer tool to consume data.

The console tools are in the bin directory of your Kafka client download.

You must provide a list of brokers and SASL credentials. After you create the properties file as described in produce data using the command line, run the console consumer in a terminal as follows:

   kafka-console-consumer.sh --bootstrap-server BOOTSTRAP_ENDPOINTS --consumer.config CONFIG_FILE --topic TOPIC_NAME 

Replace the following variables in the example with your own values:

• BOOTSTRAP_ENDPOINTS with the value from your Event Streams Service Credentials tab in the IBM Cloud console.
• CONFIG_FILE with the path of the configuration file.

You can use many of the other options of this tool, except for those that require access to ZooKeeper. For more information, see Using Kafka console tools with Event Streams.

Step 6: Consume data using an API

You cannot consume data using an Event Streams API although consumption of data from Kafka is possible using the native Kafka libraries. For more information, see Kafka consumer API.

As an alternative, use the command line.

Step 7: Connect IBM Cloud Monitoring for operational visibility by using the console

You can use IBM Cloud Monitoring to get operational visibility into the performance and health of your applications, services, and platforms. IBM Cloud Monitoring provides administrators, DevOps teams, and developers full stack telemetry with advanced features to monitor and troubleshoot, define alerts, and design custom dashboards.

For more information about how to use Monitoring with Event Streams, see:

• Opting in to metrics
• Enabling default metrics
• Enabling enhanced metrics
• Viewing details of available metrics
• Understanding metrics cost information

Step 7: Connect IBM Cloud Monitoring for operational visibility by using the CLI or command line

You cannot connect IBM Cloud Monitoring by using the CLI or command line. Use the console to complete this task.

Step 7: Connect IBM Cloud Monitoring for operational visibility by using an API

You cannot connect IBM Cloud Monitoring by using an API. Use the console to complete this task.

Step 8: Connect IBM Cloud® Activity Tracker to audit service activity

IBM Cloud Activity Tracker allows you to view, manage, and audit service activity to comply with corporate policies and industry regulations. Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. Use Activity Tracker to track how users and applications interact with the Event Streams service on the Standard and Enterprise plans.

To get up and running with Activity Tracker, see Getting Started with Activity Tracker.

Activity Tracker can have only one instance per location. To view events, you must access the web UI of the Activity Tracker service in the same location where your service instance is available. For more information, see Launch the web UI.

For more information about events specific to Event Streams, see:

• Where to view events
• Topic events
• Message audit events
• Other events

Events are formatted according to the Cloud Auditing Data Federation (CADF) standard. For further details of the information they include, see CADF standard.

Step 8: Connect IBM Cloud® Activity Tracker using the CLI or command line to audit service activity

You cannot connect Activity Tracker using the CLI or command line. Use the console to complete this task.

Step 8: Connect IBM Cloud® Activity Tracker using an API to audit service activity

You cannot connect Activity Tracker using an API. Use the console to complete this task.