Using the AWS CLI

The official command-line interface for AWS is compatible with the IBM Storage Ceph as a Service S3 API.

Written in Python, it can be installed from the Python Package Index by using pip install awscli. By default, access keys are sourced from ~/.aws/credentials, but can also be set as environment variables.

These examples were generated by using version 1.14.2 of the CLI. To check the version installed, run aws --version.

Configure the CLI to connect to Object Storage

To configure the AWS CLI, type aws configure and provide your HMAC credentials.

aws configure
AWS Access Key ID [None]: {Access Key ID}
AWS Secret Access Key [None]: {Secret Access Key}
Default region name [None]: {Region name}
Default output format [None]: json

This action creates two files:

~/.aws/credentials:

[default]
aws_access_key_id = {Access Key ID}
aws_secret_access_key = {Secret Access Key}

~/.aws/config:

[default]
region = {Region name}
output = json
endpoint_url = {endpoint}
ca_bundle = {CA certificate location}

The IBM CephaaS endpoint must be sourced by using the --endpoint-url option, and can be set in the aws config file.

Add your endpoint and CA Certificate location in this ~/.aws/config config file instead of using it in all of your AWS requests.

You can also use environment variables to set HMAC credentials:

export AWS_ACCESS_KEY_ID="{Access Key ID}"
export AWS_SECRET_ACCESS_KEY="{Secret Access Key}"

High-level syntax commands

Simple use cases can be accomplished by using aws s3 <command>. For more information about endpoints, see Endpoints and storage locations. Objects are managed by using familiar shell commands, such as ls, mv, cp, and rm. Buckets can be created by using mb and deleted by using rb.

List all buckets within a service instance

aws s3 ls
2016-09-09 12:48  s3://bucket-1
2016-09-16 21:29  s3://bucket-2

List objects within a bucket

aws s3 ls s3://bucket-1
2016-09-28 15:36       837   s3://bucket-1/c1ca2-filename-00001
2016-09-09 12:49       533   s3://bucket-1/c9872-filename-00002
2016-09-28 15:36     14476   s3://bucket-1/98837-filename-00003
2016-09-29 16:24     20950   s3://bucket-1/abfc4-filename-00004

Make a new bucket

Personally Identifiable Information (PII): When you are naming buckets or objects, do not use any information that can identify any user (natural person) by name, location, or any other means.

aws s3 mb s3://bucket-1
make_bucket: s3://bucket-1/

Add an object to a bucket

aws s3 cp large-dataset.tar.gz s3://bucket-1
upload: ./large-dataset.tar.gz to s3://bucket-1/large-dataset.tar.gz

You can also set a new object key that is different from the file name:

aws s3 cp large-dataset.tar.gz s3://bucket-1/large-dataset-for-project-x
upload: ./large-dataset.tar.gz to s3://bucket-1/large-dataset-for-project-x

Copying an object from one bucket to another

$ aws s3 cp s3://bucket-1/new-file s3://bucket-2/
copy: s3://bucket-1/new-file to s3://bucket-2/new-file

Delete an object from a bucket

aws s3 rm s3://mybucket/argparse-1.2.1.tar.gz
delete: s3://mybucket/argparse-1.2.1.tar.gz

Remove a bucket

aws s3 rb s3://bucket-1
remove_bucket: s3://bucket-1/

Low-level syntax commands

The AWS CLI also allows direct API calls that provide the same responses as direct HTTP requests by using the s3api command.

Listing buckets

aws s3api list-buckets
{
    "Owner": {
        "DisplayName": "{storage-account-uuid}",
        "ID": "{storage-account-uuid}"
    },
    "Buckets": [
        {
            "CreationDate": "2016-09-09T12:48:52.442Z",
            "Name": "bucket-1"
        },
        {
            "CreationDate": "2016-09-16T21:29:00.912Z",
            "Name": "bucket-2"
        }
    ]
}

Listing objects within a bucket

aws s3api list-objects --bucket bucket-1

{
    "Contents": [
        {
            "LastModified": "2016-09-28T15:36:56.807Z",
            "ETag": "\"13d567d518c650414c50a81805fff7f2\"",
            "StorageClass": "STANDARD",
            "Key": "c1ca2-filename-00001",
            "Owner": {
                "DisplayName": "{storage-account-uuid}",
                "ID": "{storage-account-uuid}"
            },
            "Size": 837
        },
        {
            "LastModified": "2016-09-09T12:49:58.018Z",
            "ETag": "\"3ca744fa96cb95e92081708887f63de5\"",
            "StorageClass": "STANDARD",
            "Key": "c9872-filename-00002",
            "Owner": {
                "DisplayName": "{storage-account-uuid}",
                "ID": "{storage-account-uuid}"
            },
            "Size": 533
        },
        {
            "LastModified": "2016-09-28T15:36:17.573Z",
            "ETag": "\"a54ed08bcb07c28f89f4b14ff54ce5b7\"",
            "StorageClass": "STANDARD",
            "Key": "98837-filename-00003",
            "Owner": {
                "DisplayName": "{storage-account-uuid}",
                "ID": "{storage-account-uuid}"
            },
            "Size": 14476
        },
        {
            "LastModified": "2016-10-06T14:46:26.923Z",
            "ETag": "\"2bcc8ee6bc1e4b8cd2f9a1d61d817ed2\"",
            "StorageClass": "STANDARD",
            "Key": "abfc4-filename-00004",
            "Owner": {
                "DisplayName": "{storage-account-uuid}",
                "ID": "{storage-account-uuid}"
            },
            "Size": 20950
        }
    ]
}

Next Steps

The detailed description of the RESTful API for IBM Storage Ceph as a Service can be found in the API Documentation.