Configuring streaming data rules

You can define rules by using the DataPrime eXpression Language (DPXL) to filter what data is streamed to IBM® Event Streams for IBM Cloud® from an IBM Cloud Logs instance.

For more information about the DataPrime eXpression Language, see the DataPrime Expression Language (DPXL) reference.

You can only define rules that define conditions based on the value of 1 or more fields.

You might need a parsing rule that extracts specific values such as the service name from the logSourceCRN if you want to filter by service name.

The following sections show samples of different types of rules that you can configure.

Note that some examples include the <v1> prefix. This prefix is automatically included when using the UI and does not need to be specified. However, a DPXL expression used in an API must start with the <v1> prefix.

Streaming all data

Do not define a DataPrime rule when you configure streaming.

Streaming data that contains free text

To stream data that includes free text, you must set up the rule as follows:

$d.msg.contains('ENTER_FREE_TEXT')

For example, to search for Error 1234, you can include the following in your rule:

$d.msg.contains('Error1234')

Stream data for 1 or more applications

In an IBM Cloud Logs instance, you can collect data from multiple applications.

When you configure streaming, you can define custom rules to stream selected data.

Streaming auditing events

To stream all auditing events, define the following DataPrime rule:

<v1> $l.applicationname =='ibm-audit-event'

Streaming platform logs

To stream all platform logs, define the following DataPrime rule:

<v1> $l.applicationname =='ibm-platform-logs'

Streaming logs by applicationName

To stream logs by applicationName for selected applications, define a DataPrime rule. For example, to stream data from 3 applications, you can use:

<v1> $l.applicationname =='app1' || $l.applicationname =='app2' || $l.applicationname =='app3'

Use || to indicate OR.

Stream data for 1 or more subsystems

To stream logs by subsystemName, define a DataPrime rule. For example, to stream data from 3 subsystems, you can use:

<v1> $l.subsystemname =='subsystem1' || $l.subsystemname =='subsystem2' || $l.subsystemname =='subsystem3'

Use || to indicate OR.

Stream data based on multiple values for a field that is included in the log record

If you are interested in streaming some data based on the value of a field in your log record, you can define a rule as follows:

This sample shows how to define a rule for 3 different values of a field in the log record:

<v1> $d.<FIELD_NAME> =='value1' || $d.<FIELD_NAME> =='value2' || $d.<FIELD_NAME> =='value3'

Stream data based on the values of 1 or more fields that are included in the log record

If you are interested in streaming some data based on the value of 1 or more fields in your log record, you can define a rule as follows:

<v1> $d.<FIELD_NAME_1> =='value1' && $d.<FIELD_NAME_2> =='value2'

Stream data based on 2 conditions

To stream data based on 2 conditions where the value of 1 field is the same but the value of the other field can have different values, for example, you can define the rule as follows:

<v1> ( $d.<FIELD_NAME_1> =='value1' && $d.<FIELD_NAME_2> =='value2' ) || ( $d.<FIELD_NAME_1> =='value1' && $d.<FIELD_NAME_2> =='value3' )

For example, to filter auditing events with different severity values from selected services, you can define a query as follows:

<v1> ( ( $l.applicationname =='ibm-audit-event' && ( $d.serviceName =='iam-am' ||  $d.serviceName =='cloud-object-storage' ) && $d.severity =='critical') )
|| ( $l.applicationname =='ibm-audit-event' && $d.serviceName =='iam-identity' && $d.severity =='normal')

Validate a rule

To verify the streaming rule, in the left-hand navigation, click Explore logs > Logs. Select DataPrime query, and change the query as follows:

filter 'PASTE_YOUR_CONDITION'

The output of the query should give you the list of log records that would be streamed once you enable the streaming configuration.

For example, change <v1> $l.applicationname =='ibm-audit-event' to:

filter $l.applicationname =='ibm-audit-event'