IBM Cloud Docs
Using IBM Cloud Object Storage buckets as an evidence locker

Using IBM Cloud Object Storage buckets as an evidence locker

You can configure IBM Cloud Object Storage (COS) buckets to store the evidence that is generated by the compliance checks that are integrated into the DevSecOps pipelines. Compliance evidence creates the audit trail that auditors look for during a compliance audit. One of the goals of DevSecOps is automated evidence generation and storage in auditable evidence lockers. For more information, see evidence locker.

The compliance automation pipeline stores the following information in the COS bucket:

Task artifacts
Test results, scan results, or any saved output by tasks.
Task logs
After the pipeline runs, the logs for that run are sent to the evidence locker.
Evidence
Information about tasks and their result output, which can be either a failure or a success. For more information about the format of the evidence that is sent, see Evidence summary.

Configuring your bucket

A dedicated Cloud Object Storage instance must be created before setting-up a continuous integration or continuous deployment toolchain. This COS bucket is used for compliance-related storage as evidence lockers must be created in-boundary to your applications. This helps to improve the resiliency of your pipeline. For more information, see Resiliency.

To configure your Cloud Object Storage bucket to act as a compliance evidence locker as part of a continuous integration or continuous deployment pipeline, you can use the following information as a guide. The pipeline or toolchain template scripts do not set up the locker in Cloud Object Storage.

Retention policy

You can set Cloud Object Storage buckets to enforce a retention policy or period for uploaded objects, otherwise known as Immutable Object Storage. Immutable Object Storage preserves electronic records and maintains data integrity. Retention policies ensure that data is stored in a Write-One-Read-Many (WORM), nonerasable, and nonrewritable manner. You cannot change or delete objects in protected buckets within the retention period, or delete protected buckets with objects themselves until the retention period is over. The policy is enforced until the end of a retention period and the removal of any legal holds.

It is recommended that teams set a retention policy for the buckets that are used as their evidence locker that stores every object for a minimum of 365 days.

Bucket access permissions

When using Cloud Pipelines within the DevSecOps environment, objects such as evidence, evidence summaries, and artifacts are either piped into or read from Buckets in IBM Cloud Object Storage (COS). The tools do not create, update, delete, or alter any objects or buckets.

To ensure secure access to your cloud Object Storage buckets while facilitating the necessary pipeline operations, follow these access policies:

  • Reader.

    1. This permission ensures that Continuous Delivery (CD) pipelines can verify the bucket's retention settings without modifying any data.
    2. Required to read the evidences generated by CI pipeline
  • Object Writer.

    1. This permission allows Continuous Integration (CI), CD, and Configuration Control (CC) pipelines to upload or write new objects into the buckets.

Steps to Create Service Credentials

To gain access to your COS bucket using Cloud Pipelines:

  1. Navigate to Service Credentials:
  2. Create a New Credential:
    • Click on "Create" and follow the prompts to create a new service credential for your COS bucket.

Steps to Assign Access to COS Buckets

To assign appropriate access permissions to your COS buckets:

  1. Navigate to IAM Bucket Permissions:
  2. Assign Roles and Policies:
    • Assign the Reader role to CD pipelines for checking retention policies.
    • Assign the Object Writer role to CI, CD, and CC pipelines for writing evidence into buckets.

When you use the Cloud Object Storage bucket as the evidence storage, the recommended permissions are Reader and Object writer. Permissions with higher privileges (e.g., admin-level access) should be avoided to prevent accidental or malicious modification of your objects.

Storage classes

Costs vary for teams with different setups and deployment frequency. It is not recommended that you use the free tier as Cloud Object Storage buckets because the free tier cannot be configured to be immutable.

Sample estimation

If you are working with a reference continuous integration or continuous deployment pipeline with six evidence in each, a single continuous integration and continuous deployment run pair makes 37 class A requests and six class B requests.

  • Continuous integration writes six logs, six artifacts, and six evidence, which equals 18 PUT - Class A.
  • Continuous deployment reads six evidence (six GET - Class B), writes six evidence, six logs, six artifacts, and a summary, which equals 19 PUT - Class A.

With an average of five microservices (five x continuous integration) and four deployment regions (four x continuous deployment), one full deployment equals 166 Class A and 24 Class B requests.

With one full deployment per week (four per every month), you can calculate 664 Class A and 96 Class B requests per month.

The data amount that is collected varies by use-case. With average sizes for evidence (1 kB), test artifacts (100 kB), and logs (15 kB), you can calculate 0.01 GByte of data that is created and transferred per month.

Resiliency

It is recommended that you use Cross-Region or Regional resiliency, if it needs to be kept in-boundary. For more information about these regions, see Endpoints and storage locations.

Bucket name

Cloud Object Storage bucket names must be globally unique and DNS-compliant. Names must be 3 - 63 characters in length and must contain lowercase letters, numbers, and dashes. Bucket names must begin and end with a lowercase letter or number. Names that resemble IP addresses are not allowed. Bucket names be unique across the entire IBM Cloud Object Storage system and they cannot contain any personal information, such as any part of a name or address, or financial, security accounts, or SSN.

Endpoint

Use private endpoints for most requests that originate from within IBM Cloud® and use public endpoints for most requests that originate from outside IBM Cloud®. For more information, see Endpoint Types.

For pipelines that are running in the London region, use direct endpoints due to the pipeline-managed worker infrastructure there.

Configuring Toolchains with the COS bucket

To store evidences, assets and attachment, configure COS bucket in your pipelines. Since ths bucket is used to reterive the existing information it should have Reader and Object Writer access. To confgure this bucket in the pipeline.

Environment Properties for COS Bucket Configuration

Name Type Description Required or Optional Locked or Unlocked
cos-api-key SECRET The Cloud Object Storage API key. Required Locked
cos-access-key-id SECRET The Cloud Object Storage Access Key ID from HMAC credentials. (Provided along with cos-secret-access-key instead of cos-api-key) Required Unlocked
cos-secret-access-key SECRET The Cloud Object Storage Secret Access Key from HMAC credentials. (Provided along with cos-access-key-id instead of cos-api-key) Required Unlocked
cos-bucket-name text The name of the bucket in your Cloud Object Storage instance that is used as an evidence locker. Required Unlocked
cos-endpoint text The endpoint that reads the evidence from the Cloud Object Storage instance that is used as an evidence locker. For more information, see Endpoint Types. Required Unlocked

Configure the same bucket in all your pipelines CI/CD/CC.

Migrating from one COS bucket to Another COS bucket

Migrating from One COS Bucket to Another COS Bucket If you are an existing user of the COS evidence locker and need to migrate from one COS bucket to another, it's important to ensure a smooth transition without disrupting your workflows. Below are the steps and considerations for migrating between COS buckets.

Reasons for Migration:

  • Organisational Restructuring: You may want to stop using one COS bucket and start using another.
  • Bucket Relocation: The bucket needs to be moved from one account to another, possibly due to organizational changes or compliance requirements.

Steps to Migrate:

Configure the Backup-COS Bucket: If you are migrating from an old COS bucket to a new one, ensure that your pipeline is configured to use both the old and new buckets. This allows for smooth migration without disrupting your existing workflows. - Create the New COS Bucket as defined in above steps. - Configure IAM Policies: Ensure that the new COS bucket has the necessary IAM policies for Reader and Object Writer access as required by your pipelines. - Update Environment Variables

In IBM Toolchains, update the environment variables to include both the old and new COS buckets. To configure the old bucket use backup- prefix in all COS env properties and use the normal properties to configure new COS bucket.

Name Type Description Required or Optional Locked or Unlocked
backup-cos-api-key SECRET The Backup Cloud Object Storage API key. Required Locked
backup-cos-access-key-id SECRET The Backup Cloud Object Storage Access Key ID from HMAC credentials. (Provided along with backup-cos-secret-access-key instead of backup-cos-api-key) Required Unlocked
backup-cos-secret-access-key SECRET The Backup Cloud Object Storage Secret Access Key from HMAC credentials. (Provided along with backup-cos-access-key-id instead of backup-cos-api-key) Required Unlocked
backup-cos-bucket-name text The name of the backup bucket in your Cloud Object Storage instance that is used as an evidence locker. Required Unlocked
backup-cos-endpoint text The endpoint that reads the evidence from the backup Cloud Object Storage instance that is used as an evidence locker. For more information, see Endpoint Types. Required Unlocked

Do not delete the this old bucket for 365 days, as it would be required for audit purposes.