Archiving logs to IBM Cloud Object Storage

You can archive logs from an IBM® Log Analysis instance into a bucket in an IBM Cloud Object Storage (COS) instance.

As of 28 March 2024 the IBM Log Analysis and IBM Cloud Activity Tracker services are deprecated and will no longer be supported as of 30 March 2025. Customers will need to migrate to IBM Cloud Logs, which replaces these two services, prior to 30 March 2025. For information about IBM Cloud Logs, see the IBM Cloud Logs documentation.

Data is available for search and analysis for the number of days that your service instance plan indicates. However, you might need to access the data longer for troubleshooting. You might also have to keep the data for longer for compliance, and for corporate or industry regulations. When you need access to data for longer than the number of search days, you must configure archiving.

Archiving is not enabled by default.

You can have 1 or more Log Analysis instances per region. Each IBM Log Analysis instance has its own archiving configuration.

The following figure shows a high-level view of the different components that are involved when archiving logs:

The IBM Cloud Object Storage instance is provisioned within the context of a resource group. The IBM Log Analysis instance is also provisioned within the context of a resource group. Both instances can be grouped under the same resource group or in different ones.

IBM Log Analysis uses a service ID to communicate with the IBM Cloud Object Storage service.

• The service ID that you create for an IBM Cloud Object Storage instance is used by IBM Log Analysis to authenticate and access the IBM Cloud Object Storage instance.
• You can assign specific access policies to the service ID that restrict permissions on the IBM Cloud Object Storage instance. Restrict the service ID to only have writing permissions on the bucket where you plan to archive the logs.
• You can also restrict the IP addresses that are allowed to manage the bucket.

You are responsible for configuring and managing the bucket and the data stored in it.

• If you configure archiving in an EU-managed location, you must configure a bucket that complies with the EU-managed and GDPR regulations.

When you configure archiving, consider the following information:

• Logs are automatically archived in a compressed format (.json.gz). Each log preserves its metadata.

• After you enable and configure archiving, it can take between 24-48 hours before the first archive file is created. Data included in the first archive contains data for only the first hour.

  To create an archive of older data to have available for search, you can export the data and uploaded the data to the archiving bucket.

• Archives can take 24 hours to be available, but can take as long as 72 hours for larger archive files.

• There is a minimum 6 hour delay from when data is archived.

• Logs are archived hourly. If a log line is received for an archive that was created 6 hours or more in the past, an archive file is created with the file name of the associated archive file appended with a sequence number (for example, 1).

• If there are no log entries for an hour, an archive file will not be written for that hour.

• In rare cases log lines can be duplicated in archive files. Duplicates can be determined by the log line ID.

• Automatic archiving is disabled for an instance when an the credentials that are used to archive data are invalid for over 24 hours.

The first time the archive process runs the number of days archived is dependent on the plan:

• The maximum number of days that data is archived includes logs for the past 30 days when the instance has a 30 day search plan.
• The maximum number of days that data is archived includes logs for the past 14 days when the instance has a 14 day search plan.
• The maximum number of days that data is archived includes logs for the past 7 days when the instance has a 7 day search plan.

For example, you have a service plan of 30 days. You configured the instance 10 days ago. You enable archiving on the 10th day. The archiving process generates multiple files. Each file includes logs for the period of time indicated as part of its name. If there is no data, the archive file for that period is empty.

year=<YYYY>/month=<MM>/day=<DD>/<accountID>.<YYYY>-<MM>-<DD>.<HHHH>.json.gz