Streaming auditing data from selected services

Stream auditing data that is generated by selected IBM Cloud® services from an IBM Cloud Logs instance to other corporate tools such as Security Information and Event Management (SIEM) tools by integrating IBM® Cloud Logs and Event Streams.

Prereqs

An IBM® Event Streams for IBM Cloud® instance is provisioned in the same account as the IBM Cloud Logs instance from where you plan to stream data.

Check the limitations of the service plans. For more information, see Limits and quotas.
You have permissions to create a topic in the IBM® Event Streams for IBM Cloud® instance.

To create a topic in Event Streams, you must have the manager role for the Event Streams instance. This role includes the messagehub.topic.manage IAM action role that allows an app or user to create or delete topics.
You route the auditing events that you want to stream to the IBM® Cloud Logs instance.
You have permissions to configure streaming in your IBM® Cloud Logs instance.

You must have the manager service role.
You might need a parsing rule that extracts specific values such as the service name from the logSourceCRN if you want to filter by service name.

Define a streaming rule

You can define rules that filter the auditing events that you stream to IBM® Event Streams for IBM Cloud®.

The following table shows some rules that you can configure:

Example rules
Rule description	Rule
Stream all auditing events	`<v1> $l.applicationname =='ibm-audit-event'`
Stream auditing events from 1 service	`<v1> $l.applicationname =='ibm-audit-event' && $d.serviceName =='REPLACE_WITH_CRN_SERVICE_NAME'`
Stream auditing events from 2 services	`<v1> $l.applicationname =='ibm-audit-event' && ( $d.serviceName =='REPLACE_WITH_CRN_SERVICE_NAME_1' \|\| $d.serviceName =='REPLACE_WITH_CRN_SERVICE_NAME_2' )`
Stream auditing events that have a critical severity	`<v1> $l.applicationname =='ibm-audit-event' && $d.severity =='REPLACE_WITH_SEVERITY_VALUE'` Valid severity values are: `normal`, `warning`, and `critical`.
Stream auditing events with different severity values from selected services	`<v1> ( ( $l.applicationname =='ibm-audit-event' && ( $d.serviceName =='REPLACE_WITH_CRN_SERVICE_NAME_1' \|\| $d.serviceName =='REPLACE_WITH_CRN_SERVICE_NAME_2' ) && $d.severity =='REPLACE_WITH_SEVERITY_VALUE') ) \|\| ( $l.applicationname =='ibm-audit-event' && $d.serviceName =='iam-identity' && $d.severity =='REPLACE_WITH_SEVERITY_VALUE')` Valid severity values are: `normal`, `warning`, and `critical`.
Stream auditing events based on data in the `requestData` or `responseData` sections	`$d.requestData.REPLACE_WITH_FIELD_NAME` For example, for a field `requestId`, you can configure `<v1> $d.requestData.requestId`

For more information, see Configuring streaming data rules.

Note that these examples include the <v1> prefix. This prefix is automatically included when using the UI and does not need to be specified. However, a DPXL expression used in an API must start with the <v1> prefix.

For example, to filter auditing events with different severity values from selected services, you can define a query as follows:

<v1> ( ( $l.applicationname =='ibm-audit-event' && ( $d.serviceName =='iam-am' ||  $d.serviceName =='cloud-object-storage' ) && $d.severity =='critical') )
|| ( $l.applicationname =='ibm-audit-event' && $d.serviceName =='iam-identity' && $d.severity =='normal')

Validate a streaming rule

To verify the streaming rule, in the left-hand navigation, click Explore logs > Logs. Select DataPrime query, and change the query as follows:

filter 'PASTE_YOUR_CONDITION'

The output of the query should give you the list of log records that would be streamed once you enable the streaming configuration.

For example, change <v1> $l.applicationname =='ibm-audit-event' to:

filter $l.applicationname =='ibm-audit-event'

Configure streaming for auditing event

To configure streaming, see Integrating IBM Cloud Logs with Event Streams.