Querying data

Considerations when querying Priority insights data

There are considerations when querying log data in the IBM Cloud Logs Priority insights pipeline:

• Filtering can be used in conjunction with searching using Lucene or DataPrime.

• Logs in the Priority insights pipeline are indexed. If your instance reaches its maximum amount of indexed fields, additional fields are unavailable to query. For more information on indexing and data mapping, see Understanding indexing and field mapping.

• You can get a mapping exception when data that is ingested through the Priority insights data pipeline detects the same field sent by different log records with different types. Mapping exceptions make fields unavailable to query. For more information, see Mapping exceptions.

• Logs ingested through the Analyze and alert and Store and search data pipelines can only be directly queried from the archive..

• You can query logs that are ingested and processed through the Priority insights data pipeline by using a Lucene query or a DataPrime query.

  For example, when you define a Lucene query, you can run queries such as free-text searches, regular RegEX expressions, or using field searches.

  If you are not seeing expected data:

  • Consider broadening your query or removing filters.

  • Note that some of your logs might not yet be indexed and will not be found by the filter or query.

Considerations when querying data from a bucket

You can query data from the Explorer logs page or by running an archive query.

There are considerations when querying log data from the data bucket:

• Data stored in the data bucket include data ingested through the Priority insights, Analyze and alert, and Store and search data pipelines. It also includes logs that are matched through a Parsing Block rule that has the View blocked logs in Live tail and archive to IBM Cloud Object Storage option enabled.
• You maintain the data in the bucket. You can keep the data for as long as you need and query it via the Logs page, selecting the All logs option.
• Filtering can be used in conjunction with searching using Lucene or DataPrime.
• You can query data with unlimited time frames. There are no restrictions on how far back in time your data can go. You maintain the data and you ahve access to the data for as long as you keep it.
• You can query logs regardless of log priority and daily quota. Only blocked logs are not sent to the archive.
• Archive Query lets you to directly query your logs from your archive using any text or a wide range of syntax queries. You can query logs regardless of log priority, daily quota, or the time frame of your data. For more information, see Querying archived data.

Query limits

Archive query limitations

The following are the limitations placed on queries.

You also need to be aware of the following when querying archived data:

• You can use the same query syntax (Lucene or DataPrime) on the Archive Queries page.

• You might see slight delays when querying archived data when compared with other Explore queries.

• Once a limit is reached, a warning message is displayed. Refine your query results to avoid reaching a limit.

Refining archive query results

You can refine your query results using the following methods:

• Apply more selective filters to your queries (for example, application or subsystem).

• If using the DataPrime extract operator and subsequently filtering its results, create a parsing rule and filter on the parsed field instead.

• Avoid regular expressions or wildcards in filters.

• In DataPrime, switch from using the contains operator on strings to the free text search operator (~).