Archiving events to IBM Cloud Object Storage

You can archive events from an IBM Cloud Activity Tracker instance into a bucket in an IBM Cloud Object Storage (COS) instance.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025. For information about IBM Cloud Logs, see the IBM Cloud Logs documentation.

In Activity Tracker, by default archiving is not enabled. Data is available for search and analysis for the number of days that your service instance plan indicates. However, you might need to access the data longer for troubleshooting. You might also have to keep the data for longer for compliance, and for corporate or industry regulations. When you need access to data for longer than the number of search days, you must configure archiving.

You can only have 1 Activity Tracker instance per region. Each IBM Cloud Activity Tracker instance has its own archiving configuration.

The following figure shows a high-level view of the different components that are involved when archiving events:

High-level view archiving events

The IBM Cloud Object Storage instance is provisioned within the context of a resource group. The IBM Cloud Activity Tracker instance is also provisioned within the context of a resource group. Both instances can be grouped under the same resource group or in different ones.

IBM Cloud Activity Tracker uses a service ID to communicate with the IBM Cloud Object Storage service.

The service ID that you create for an IBM Cloud Object Storage instance is used by IBM Cloud Activity Tracker to authenticate and access the IBM Cloud Object Storage instance.
You can assign specific access policies to the service ID that restrict permissions on the IBM Cloud Object Storage instance. Restrict the service ID to only have writing permissions on the bucket where you plan to archive the events.
You can also restrict the IP addresses that are allowed to manage the bucket.

You are responsible for configuring and managing the bucket and the data stored in it.

If you configure archiving in an EU-managed location, you must configure a bucket that complies with the EU-managed and GDPR regulations.

After you configure archiving,consider the following information:

Events are automatically archived in a compressed format (.json.gz). Each event preserves its metadata.
Events are archived within 24-48 hours after you save the configuration.
Events are archived hourly.
The first archive file is created when the archiving process runs and there is data.
Automatic archiving is disabled for an instance when an the credentials that are used to archive data are invalid for over 24 hours.

You have a service plan of 30 days. You configured the instance 10 days ago. You enable archiving today. There will be no archive data for the first 10 days the instance was running.

The archiving process generates multiple files. Each file includes events for the period of time indicated as part of its name. If there is no data, the archive file for that period is empty.

Archived file format

The archive directory format looks like this:

year=<YYYY>/month=<MM>/day=<DD>/<accountID>.<YYYY>-<MM>-<DD>.<HHHH>.json.gz

Where

YYYY represents the year; MM represents the month; and DD represents the day.

<accountID> represents the auditing account ID, that is, the ID that is shown in the web UI URL.

HHHH represents hours in 24 format.

The timestamp that is used to determine whether the event is included in an archive is the UTC timestamp.

Depending on your location, there might be events that you see in local time in your views on a specific day. However, you cannot find them in the archive file. You are most likely viewing events in local time and the archive process uses the UTC timestamp.

Configure archiving

For information on how to configure archiving, see:

In addition, consider the following information:

You must have the manager role to configure archiving in the IBM Cloud Activity Tracker instance. This role includes the logdna.dashboard.manage IAM action role that allows a user to perform admin tasks such as configure archiving.
When you configure archiving, the Activity Tracker instance and the IBM Cloud Object Storage (COS) instance must be provisioned in the same account.
The credential that Activity Tracker uses to write data into a COS bucket must have writer role.

Monitor archiving

To monitor archiving, you can use the following services:

IBM Cloud Monitoring service:

IBM Cloud Object Storage is integrated with the Monitoring service. Monitoring provides a default template that you can customize to monitor the bucket that you configure to store data for long term.

For more information, see Monitoring archiving by using IBM Cloud Monitoring.
IBM Cloud Activity Tracker:

Archiving generates Activity Tracker events with the action logdnaat.archiving.send to notify of failures sending data to Event Streams. There are different reasons for failure such as invalid credentials and topic deleted.

For more information, see Configuring an alert to monitor archiving.

Viewing archived events by using the SQL Query service

Data Engine provides a serverless, no-ETL solution to easily query data stored in COS. Learn more.

You can use this service to analyze data from archived files in COS.

Once you have SQL Query running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library.

When you query events:

You must provision an instance of the Data Engine service.
You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using a custom bucket, create one. You can specify your custom bucket as the place to to store results.

IAM permissions to configure archiving

To configure archiving, you need the following permissions:

IBM Cloud Activity Tracker service

The following table lists the minimum roles that a user must have to be able to launch the IBM Cloud Activity Tracker web UI, and configure archiving through the UI or by using the API:

IAM roles
Role	Permission granted
Platform role: `Viewer`	Allows the user to view the list of service instances in the Observability dashboard.
Service role: `Manager`	Allows the user to launch the web UI and configure archiving through the web UI or by using the API.

For more information on how to configure policies for a user, see Granting user permissions to a user or service ID.

IBM Cloud Object Storage service

The following table lists the roles that a user can have to complete the actions required to configure the IBM Cloud Object Storage service:

Roles and actions
Service	Roles	Action
`Cloud Object Storage`	Platform role: `Administrator`	Allows the user to assign policies to users in the account to work with the IBM Cloud Object Storage service.
`Cloud Object Storage`	Platform role: `Administrator` or Platform role:`Editor`	Allows the user to provision an instance of the IBM Cloud Object Storage service.
`Cloud Object Storage`	Platform role: `Administrator` or Platform role:`Editor` or Platform role: `Operator`	Allows the user to create a service ID.
`Cloud Object Storage`	Service role: `writer`	Grants permissions to create, modify, and delete buckets. In addition, grants permissions to upload and download the objects in the bucket.

For more information on how to configure policies for a user, see Grant IAM policies to a user to work with IBM Cloud Object StorageD.

Service ID

The service ID that you must create for an IBM Cloud Object Storage instance is used by IBM Cloud Activity Tracker to authenticate and access the IBM Cloud Object Storage instance. This service ID must have the writer role. This role grants permissions to upload archive files in the bucket.

When the service credential is rotated, make sure the API Key is updated with the new API Key. Archiving will stop if the API Key is not updated.

Activity Tracker events

The following Activity Tracker events are generated when you configure archiving:

Archiving Activity Tracker events
Action	Description
`logdnaat.account-archive-setting.configure`	This event is generated when an administrator configures archiving for an auditing instance.