Integrating IBM Cloud Logs alerts with Event Notifications and raising PagerDuty alerts

This tutorial shows you how to integrate IBM Cloud Logs alerts with Event Notifications and send PagerDuty Alerts.

IBM Cloud Logs is used to visualize and alert on events that are generated in your account. When an event of interest takes place in your IBM Cloud Logs instance, it communicates with a connected Event Notifications instance to raise an alert to the PagerDuty destination.

This tutorial shows you how to configure the following flow:

Creating a IBM Cloud Logs and an Event Notifications instance.
Connecting an instance of IBM Cloud Logs with an Event Notifications instance.
Configuring IBM Cloud Logs to send out notifications to PagerDuty as a destination.
Configuring an alert in IBM Cloud Logs.
Raising a PagerDuty alert when an alert is open in IBM Cloud Logs and automatically resolving it in PagerDuty when the alert is closed IBM Cloud Logs.

Before you begin

You need an IBM Cloud® account. If you don't have an account, then Create an IBM Cloud account.

Create an Event Notifications service instance

Login to your IBM Cloud® account.
In the IBM Cloud catalog, search Event notifications > Event notifications.
Select a Location from the list of supported locations and select a pricing plan. To know more about the pricing plans, refer here.
Enter a service name.
Select a resource group.
Accept the license agreement terms and click Create.

Create an IBM Cloud Logs service instance

In the IBM Cloud catalog, search Cloud Logs > Cloud Logs.
Select a Location from the list of supported locations and select a pricing plan.
Enter a service name.
Select a resource group.
Accept the license agreement terms and click Create.

Authorize the connection between the IBM Cloud Logs service instance and the Event Notifications service instance

Select Manage > Access(IAM) from the upper right on your Dashboard.
Under Authorizations, click Create.
Select the source account, Cloud Logs as the source service and the specific instance of Cloud Logs that needs access.
Next, select Event notifications as the target service, the service instance of Event Notifications that needs source access and assign the viewer, reader, and event source manager roles to the source that it can use to interact with the target.
Click Review. Review all the assignments.
Select Authorize.

Enabling access in the instances and testing the connection

Navigate to the cloud logs instance that was selected as the source in the connection that was previously established by clicking > Resource List > Logging and Monitoring > Your Cloud Logs instance.
In your IBM Cloud Logs instance, select Dashboard. Now, in your dashboard, click and select Integrations > Outbound Integrations.
Select Add under Event Notifications.
Click Add New. Since the authorization policy has been created, select next.
Name your integration, select the Event Notifications instance that was used for authorizing and set your endpoint type to Public. Click Save.
Before moving to the Test step, we need to verify whether a connection has been established between the Event Notifications instance and the IBM Cloud Logs instance.
Navigate to your Event Notifications instance by clicking > Resource List > Developer Tools > Your Event notifications instance.
Under Sources in your Event Notifications instance, you should see your IBM Cloud Logs instance as a source of the form IBM Cloud Logs - GUID of your Cloud Logs Instance.
Go to topics and click Create. Name your topic, and select your Cloud Logs instance as the Source. To learn more about creating a topic and providing filters that can be used in your instance, see Creating an Event notifications topic. Click Create.
Navigate to Destinations and click Create. Name your destination and select destination type as PagerDuty. Provide the and Routing key, then click Add. See here for process to generate the Routing key.

Create a template for your PagerDuty alert by navigating to Templates > Create. Name your template, select the template type as PagerDuty notification, and provide your template.

Example Template:

{
"payload": {
    "summary": "{{ data.alert_definition.name}}",
    "timestamp": "{{time}}",
    {{#equal data.alert_definition.severity "Critical"}}
    "severity": "critical",
    {{/equal}}
    {{#equal data.alert_definition.severity "High"}}
    "severity": "error",
    {{/equal}}
    {{#equal data.alert_definition.severity "Info"}}
    "severity": "info",
    {{/equal}}
    {{#equal data.alert_definition.severity "Warning"}}
    "severity": "warning",
    {{/equal}}
    "source": "{{ source }}"
},
"dedup_key": "{{ data.alert_definition.id }}",
{{#equal data.status "triggered"}}
"event_action": "trigger"
{{/equal}}

{{#equal data.status "resolved"}}
"event_action": "resolve"
{{/equal}}

{{#equal data.status "acknowledged"}}
"event_action": "acknowledge"
{{/equal}}
}

To learn more about the fields that can be used to construct the template block, see here.

To create a Subscription, navigate to Subscriptions > Create. Name your subscription, select the topic, the PagerDuty destination, and the template that were previously created. Click Create.

Creating an IBM Cloud Logs alert

Navigate to your IBM Cloud Logs instance and click Dashboard.
Click > Alerts > Alert Management.
Create a New Alert.
Provide the Alert Name and Severity. Select the wanted alert type and provide the query to filter the logs, along with the severity and conditions for the alert to be raised.
Set the time period and schedule for the alert notifications and specify the notification content.
Verify the alert and click create.

Verifying that the connection was successfully established

Navigate to your IBM Cloud Logs Dashboard and click Test. If the connection was successfully established, you receive a PagerDuty alert.
You can set the time period and frequency of the Alert notifications in the next step and select alerts to attach to the webhook.
Select Done.

The process of integration is completed. You should start receiving PagerDuty alerts according to the criteria that were set when unusual activity is observed.