Working with the IBM Cloud Object Storage event producer

The IBM Cloud Object Storage subscription listens for changes to an Object Storage bucket. When you create a subscription to a bucket, your app or job receives a separate event for each successful change to that bucket. You can subscribe to different events such as write events, delete events, or all events. You can create at most 100 Object Storage subscriptions per project.

Setting up the IBM Cloud Object Storage event producer

To get started, you must create an IBM Cloud Object Storage service instance and create a regional bucket in one of the supported regions for Code Engine.

Your Object Storage bucket must be a regional bucket that is located in the same region as your Code Engine project. Cross-region and single-site buckets are not supported. For more information about setting up the IBM Cloud Object Storage event producer, see Getting started with Object Storage.

If your application or job that is receiving events wants to talk to a service by using private networking and the service has both private and direct endpoints (such as IBM Cloud Object Storage), then the direct endpoints must be used.

Assigning the Notifications Manager role to Code Engine

Before you can create an Object Storage subscription, you must assign the Notifications Manager role to Code Engine. As a Notifications Manager, Code Engine can view, modify, and delete notifications for an Object Storage bucket.

Only account administrators can assign the Notifications Manager role.

When you assign the Notifications Manager role to your project, you can then create event subscriptions for any regional buckets in your Object Storage instance that are in the same region as your project.

Navigate to the Grant a Service Authorization page in the IAM dashboard.
From Source service, select Code engine.
Select Resources based on selected attributes and Source service instance. Then, select a Code Engine project.
In Target service, select Cloud Object Storage.
Select Resources based on selected attributes and Service instance. Then, select your IBM Cloud Object Storage instance.
Assign the Notifications Manager role and click Authorize.

You can also assign the Notifications Manager role to your project by using the ibmcloud iam authorization-policy-create command.

Subscribing to IBM Cloud Object Storage events for an application

You can work with IBM Cloud Object Storage subscriptions from the console or with the CLI so events are sent to a Code Engine application.

By default, events are routed to the root URL of the destination application. You can send events to a different destination within the app by using the --path option. For example, if your subscription specifies --path /event, the event is sent to https://<base application URL>/events.

Events are sent to applications as HTTP POST requests. For more information, see HTTP headers and body information for events.

Subscribing to IBM Cloud Object Storage events for an application from the console

You can create and update IBM Cloud Object Storage event subscriptions for an application from the console.

Before you begin

Create a project.
Create an application. For example, create an application that is called myapp that uses the icr.io/codeengine/cos-event image. This image is built from cos-listen.go, available from the Samples for IBM Cloud Code Engine GitHub repo.

Complete the following steps to create and update an IBM Cloud Object Storage event subscription for an application from the console.

From the Code Engine Projects page, go to your project.
From the Overview page, click Event subscriptions.
From the Event subscriptions page, click Create to create your subscription.
From the Create an event subscription page, complete the following steps.
1. For Event type, select the Cloud Object Storage tile. Click Next.
2. For General, provide a name for the Object Storage subscription, for example, mycos. You can optionally provide event attributes. Note that if the Object Storage event consumer is an application, event attributes are available as HTTP headers. If the event consumer is a job, event attributes are available as environment variables. Click Next to proceed.
3. For Bucket event details, select or type the name of an existing Object Storage bucket. Specify the types of changes for your object and optionally provide an object name prefix or suffix to filter objects in the bucket that to trigger events for the subscription. Click Next to proceed.
4. For Event consumer, specify the application to receive events. Notice that you can choose from a list of defined applications and jobs. For this example, use the myapp application that references the icr.io/codeengine/cos-listen image. If your app does not exist, you can provide the name of your application and create your application after you create the Object Storage subscription. For applications only, you can optionally specify a path. By default, events are routed to the root URL of the destination application. You can send events to a different destination within the app by specifying a path. For example, if your subscription path specifies /events, the events are sent to https://<base application URL>/events. Click Next to proceed.
5. For Summary, review the settings for your Object Storage event subscription and make changes if needed. When ready, click Create to create the Object Storage subscription.
Now that your Object Storage subscription is created, go to the Event subscriptions page to view a listing of defined subscriptions.
To update a subscription, navigate to your Object Storage subscription page. From the Event subscriptions page, click the name of the subscription that you want to update.
From your Object Storage subscription page, let's change the type of object change to only delete object changes. From the Bucket event details tab, select only the delete type of object change. Click Save to save your changes.
Because the myapp application references the sample cos-listen application, which prints information to log files, you can view the logs. After an object is deleted from the bucket, you can see an event for the delete in the app logs. See Viewing application logs from the console.

After you define an IBM Cloud Object Storage event subscription that references a specific bucket, you cannot update this subscription to use a different bucket. You must create a new subscription to reference the bucket that you want.

Subscribing to IBM Cloud Object Storage events for an application with the CLI

Before you begin

Set up your Code Engine CLI environment.
Create and work with a project.
Create an application. For example, create an application that is called myapp that uses the icr.io/codeengine/cos-listen image. This image is built from cos-listen.go, available from the Samples for IBM Cloud Code Engine GitHub repo. For example,
```
ibmcloud ce application create --name myapp --image icr.io/codeengine/cos-listen
```

Connect your application to the IBM Cloud Object Storage event producer by using the ibmcloud ce subscription cos create command. For example,
```
ibmcloud ce subscription cos create --name mycosevent --destination-type app --destination myapp --bucket mybucket
```

After your subscription creates, run the subscription cos get command for details about your subscription.

ibmcloud ce subscription cos get --name mycosevent

Example output

Getting COS source 'mycosevent'...
OK

Name:          mycosevent
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           59s
Created:       2021-03-01T20:08:36-06:00

Destination:  App:myapp
Bucket:       mybucket
Event Type:   all
Ready:        true

Conditions:
    Type            OK    Age  Reason
    CosConfigured   true  55s
    Ready           true  55s
    ReadyForEvents  true  55s
    SinkProvided    true  55s

Events:
    Type     Reason           Age      Source                Messages
    Normal   FinalizerUpdate  61s      cossource-controller  Updated "mycosevent" finalizers

Now every time that you change your bucket, your app receives notification.

Want to try a tutorial? See Subscribing to Object Storage events. Looking for more code examples? Check out the Samples for IBM Cloud Code Engine GitHub repo.

Viewing event information for an application from the console

To view information about your event subscriptions,

From the Code Engine Projects page, go to your project.
From the Overview page, click Event subscriptions to view a listing of defined subscriptions.

If your application prints information to log files, as the sample cos-listen application does, then view the log files for your event consumer application. See Viewing application logs from the console.

Viewing event information for an application with the CLI

If your application prints information to log files, as the cos-listen application does, then use the ibmcloud ce app logs CLI command to view the information that was sent.

Before you can view event information for your application, you must first create an Object Storage event. Make a change to your bucket.

To view the logs for the application that you created in the previous example,

ibmcloud ce application logs --application myapp

Example output

Getting logs for all instances of application 'myapp'...
OK

myapp-a2mvv-1-deployment-d97dcd6cf-zc9lg/user-container:
Listening on port 8080
2021-04-13 19:43:45 - Received:

Body: {"bucket":"mybucket","endpoint":"","key":"Notes.rtf","notification":{"bucket_name":"mybucket","content_type":"text/rtf","event_type":"Object:Write","format":"2.0","object_etag":"2944035e54ee1bdc423848c8eaf05e86","object_length":"4642","object_name":"NOtes.rtf","request_id":"6abc7123-382d-4115-98e8-7568b2cc03f8","request_time":"2021-04-13T19:43:36.610Z"},"operation":"Object:Write"}

Note that log information for apps lasts for only one hour. For more information about logging, see Viewing logs.

Looking for more code examples? Check out the Samples for IBM Cloud Code Engine GitHub repo.

IBM Cloud Object Storage header and body information for events delivered to applications

All events that are delivered to applications are received as HTTP messages. Events contain certain HTTP headers that help you to quickly determine key bits of information about the events without looking at the body (business logic) of the event. For more information, see the CloudEvents specification.

Headers

The following table describes the headers for Object Storage events.

Header files for events
Header	Description
`ce-id`	A unique identifier for the event, unless an event is replayed, in which case, it is assigned the same ID.
`ce-source`	A URI-reference that indicates where this event originated from within the event producer. For Object Storage events, this value is `https://cloud.ibm.com/catalog/services/cloud-object-storage/[BUCKET_NAME]` where `[BUCKET_NAME]` is the name of the bucket that contains the object.
`ce-specversion`	The version of the `CloudEvents` spec. This value is always `1.0`.
`ce-subject`	Indicates the resource about which the event is related. For Object Storage events, this is the name of the object (or key) that was acted upon.
`ce-time`	The time that the event was generated.
`ce-type`	The type of the event. For Object Storage events, this is `com.ibm.cloud.cos.document.[ACTION]` where `[ACTION]` is either `write` or `delete`. When a create or update for an event occurs, a `write` action is used for `ce-type`.

Example

ce-id:  3fb2c04e-a660-4640-8899-b82efb8169b6
ce-source: https://cloud.ibm.com/catalog/services/cloud-object-storage/mybucket
ce-specversion: 1.0
ce-subject: object-69-144
ce-time: 2021-08-17T20:22:02.917Z
ce-type: com.ibm.cloud.cos.document.delete

HTTP body

The HTTP body for an IBM Cloud Object Storage event is in the following format,

{
  "bucket": "mybucket",
  "endpoint": "",
  "key": "object-69-144",
  "notification": {
    "bucket_name": "mybucket",
    "content_type": "image/svg+xml",
    "event_type": "Object:Delete",
    "format": "2.0",
    "object_etag": "f3a9dbde30fdf48abc23e5f8b485d6e5",
    "object_length": "1064391",
    "object_name": "object-69-144",
    "request_id": "67a2048a-abcd-abcd-9e0c-968744094b85",
    "request_time": "2021-08-17T20:22:02.917Z"
  },
  "operation": "Object:Delete"
}

The following table describes the body field.

Body fields for IBM Cloud Object Storage
Body field	Description
`bucket`	The bucket name for the object that is related to the event.
`endpoint`	This value is always an empty string.
`key`	The name of the object in the bucket.
`operation`	The event type or operation, either type `Object:Write` or `Object:Delete`. Create or upload events are tagged as `Object:Write` operations.
`Notification.bucket_name`	The bucket name for the object that is related to the event.
`Notification.content_type`	The MIME type of the object, for example, `text/html`.
`Notification.event_type`	The event type or operation, either type `Object:Write` or `Object:Delete`. Create or upload events are tagged as `Object:Write` operations.
`Notification.format`	This value is always `2.0`.
`Notification.object_etag`	A unique value that changes each time that the object is modified. This value does not display for an `Object:Delete` operation.
`Notification.object_length`	The size of the object, in bytes.
`Notification.request_id`	The unique ID that is related to the object change.
`Notification.request_time`	The time that the object change occurred.

Subscribing to IBM Cloud Object Storage events for a job

You can work with IBM Cloud Object Storage subscriptions from the console or with the CLI so events are sent to a Code Engine job.

When you create an event subscription for a job, a job run is created for each event that is triggered, and this job run has the environment variables that are related to the job. For more information about the environment variables that are sent by IBM Cloud Object Storage, see Environment variables for events.

Subscribing to IBM Cloud Object Storage events for a job from the console

You can create and update IBM Cloud Object Storage event subscriptions for a job from the console.

Before you begin

Create a project.
Create a job. For example, create a job that is called myjob that uses the icr.io/codeengine/codeengine image. This image is built from codeengine.go, available from the Samples for IBM Cloud Code Engine GitHub repo.

Complete the following steps to create and update an IBM Cloud Object Storage event subscription for a job from the console.

From the Code Engine Projects page, go to your project.
From the Overview page, click Event subscriptions.
From the Event subscriptions page, click Create to create your subscription.
From the Create an event subscription page, complete the following steps.
1. For Event type, select the Cloud Object Storage tile. Click Next.
2. For General, provide a name for the Object Storage subscription, for example, mycos-job. You can optionally provide event attributes. When the event consumer is a job, event attributes are available as environment variables. Click Next to proceed.
3. For Bucket event details, select or type the name of an existing Object Storage bucket. Specify the types of changes for your object and optionally provide an object name prefix or suffix to filter objects in the bucket to trigger events for the subscription. Click Next to proceed.
4. For Event consumer, specify the job to receive events. Notice that you can choose from a list of defined jobs. For this example, use the myjob job that references the icr.io/codeengine/codeengine image. If your job does not exist, you can specify the name of your job and create your job after you create the Object Storage subscription. Click Next to proceed.
5. For Summary, review the settings for your Object Storage event subscription and make changes if needed. When ready, click Create to create the Object Storage subscription.
Now that your Object Storage subscription is created, go to the Event subscriptions page to view a listing of defined subscriptions.
To update a subscription, navigate to your Object Storage subscription page. From the Event subscriptions page, click the name of the subscription that you want to update.
From your Object Storage subscription page, let's change the type of object change to only delete object changes. From the Bucket event details tab, only select the delete type of object change. Click Save to save your changes.
Because the myjob job references the sample icr.io/codeengine/codeengine job, which prints information to log files, you can view the logs. After an object is deleted from the bucket, you can see an event for the delete in the logs for the job run. See Viewing job logs from the console.

Job runs that are created by subscriptions are deleted after 10 minutes.

Subscribing to IBM Cloud Object Storage events for a job with the CLI

Before you begin

Set up your Code Engine CLI environment.
Create and work with a project.
Create a job. For example, create a job that is called myjob that uses the icr.io/codeengine/codeengine image. This image is built from codeengine.go, available from the Samples for IBM Cloud Code Engine GitHub repo.
```
ibmcloud ce job create --name myjob --image icr.io/codeengine/codeengine
```

Connect your job to the IBM Cloud Object Storage event producer by using the ibmcloud ce subscription cos create command.
```
ibmcloud ce subscription cos create --name mycosevent --destination-type job --destination myjob --bucket mybucket
```

After your subscription creates, run the subscription cos get command for details about your subscription.

ibmcloud ce subscription cos get --name mycosevent

Example output

Getting COS source 'mycosevent'...
OK

Name:          mycosevent
ID:            abcdefgh-abcd-abcd-abcd-1a2b3c4d5e6f
Project Name:  myproject
Project ID:    01234567-abcd-abcd-abcd-abcdabcd1111
Age:           59s
Created:       2021-03-01T20:08:36-06:00

Destination Type:  job
Destination:       myjob
Bucket:       mybucket
Event Type:   all
Ready:        true

Conditions:
    Type            OK    Age  Reason
    CosConfigured   true  55s
    Ready           true  55s
    ReadyForEvents  true  55s
    SinkProvided    true  55s

Events:
    Type     Reason           Age      Source                Messages
    Normal   FinalizerUpdate  61s      cossource-controller  Updated "mycosevent" finalizers

Now every time that you change your bucket, your job receives notification.

Job runs that are created by subscriptions are deleted after 10 minutes.

Want to try a tutorial? See Subscribing to Object Storage events. Looking for more code examples? Check out the Samples for IBM Cloud Code Engine GitHub repo.

Viewing event information for a job from the console

To view information about your event subscriptions,

From the Code Engine Projects page, go to your project.
From the Overview page, click Event subscriptions to view a listing of defined subscriptions.

If your job prints information to log files, as the sample codeengine job does, then view the log files for your event consumer application. See Viewing job logs from the console.

Viewing event information for a job with the CLI

If your job prints information to log files, as the codeengine job does, then use the ibmcloud ce jobrun logs CLI command to view the information that was sent.

Before you can view event information for your job, you must first create an Object Storage event. Make a change to your bucket.

To find the job run for the job in the previous example,

ibmcloud ce jobrun list

Example output

Listing job runs...
OK

Name         Failed  Pending  Requested  Running  Succeeded  Unknown  Age
myjob-pnz6m  0       0        0          0        1          0        39s

View the logs for the job run by specifying the job run name.

ibmcloud ce jobrun logs --jobrun myjob-pnz6m

Example output

Getting logs for all instances of job run 'myjob-pnz6m'...
Getting jobrun 'myjob-pnz6m'...
Getting instances of jobrun 'myjob-pnz6m'...
OK

myjob-pnz6m-0-0/myjob:
Hello from helloworld! I'm a batch job! Index: 0

Hello World from:
. ___  __  ____  ____
./ __)/  \(    \(  __)
( (__(  O )) D ( ) _)
.\___)\__/(____/(____)
.____  __ _   ___  __  __ _  ____
(  __)(  ( \ / __)(  )(  ( \(  __)
.) _) /    /( (_ \ )( /    / ) _)
(____)\_)__) \___/(__)\_)__)(____)

Some Env Vars:
--------------
CE_DATA={"bucket":"mybucket","endpoint":"","key":"Notes.rtf","notification":{"bucket_name":"mybucket","content_type":"text/rtf","event_type":"Object:Delete","format":"2.0","object_length":"4642","object_name":"Notes.rtf","request_id":"b59727ee-9c4e-446a-9261-5616f6d1283b","request_time":"2021-04-13T20:10:37.631Z"},"operation":"Object:Delete"}
CE_ID=b59727ee-9c4e-446a-9261-5616f6d1283b
CE_SOURCE=https://cloud.ibm.com/catalog/services/cloud-object-storage/mybucket
CE_SPECVERSION=1.0
CE_TIME=2021-04-13T20:10:37.631Z
CE_TYPE=com.ibm.cloud.cos.document.delete
CONTENT_TYPE=application/json
HOME=/root
HOSTNAME=myjob-pnz6m-0-0
JOB_INDEX=0
KUBERNETES_PORT=tcp://172.21.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.21.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.21.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.21.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1

Note that log information for job runs lasts for only one hour. For more information about logging, see Viewing logs.

Looking for more code examples? Check out the Samples for IBM Cloud Code Engine GitHub repo.

Environment variables for events delivered to jobs

All events that are delivered to a job are received as environment variables. These environment variables include a prefix of CE_ and are based on the CloudEvents spec.

Each event contains some common environment variables that appear every time that the event is delivered to a job. The actual set of variables in each event can include more options. For more information and more environment variable options, see the CloudEvent attributes.

The following table describes the environment variables that are specific to IBM Cloud Object Storage events.

Environment variables for events
Variable	Description
`CE_DATA`	The data (body) for the event. See `CE_DATA` for Object Storage events.
`CE_ID`	A unique identifier for the event, unless an event is replayed, in which case, it is assigned the same ID.
`CE_SOURCE`	A URI-reference that indicates where this event originated from within the event producer. For Object Storage events, this value is `https://cloud.ibm.com/catalog/services/cloud-object-storage/[BUCKET_NAME]` where `[BUCKET_NAME]` is the name of the bucket that contains the object.
`CE_SPECVERSION`	The version of the `CloudEvents` spec. This value is always `1.0`.
`CE_TIME`	The time that the event was generated.
`CE_TYPE`	The type of the event. For Object Storage events, this is `com.ibm.cloud.cos.document.[ACTION]` where `[ACTION]` is either `write` or `delete`.

`CE_DATA` environment variable

Notice that the following example for CE_DATA is formatted for readability.

{
"bucket":"mybucket",
"endpoint":"",
"key":"Notes.rtf",
"notification": {
    "bucket_name":"mybucket",
    "content_type":"text/rtf",
    "event_type":"Object:Delete",
    "format":"2.0",
    "object_length":"4642",
    "object_name":"Notes.rtf",
    "request_id":"b59727ee-9c4e-446a-9261-5616f6d1283b",
    "request_time":"2021-04-13T20:10:37.631Z"
},
"operation":"Object:Delete"}

The following table describes the CE_DATA environment attribute.

Environment variables for IBM Cloud Object Storage
Attribute	Description
`bucket`	The bucket name for the object that is related to the event.
`endpont`	This value is always an empty string.
`key`	The name of the object in the bucket.
`operaton`	The event type or operation, either type `Object:Write` or `Object:Delete`. Create or upload events are tagged as `Object:Write` operations.
`Notification.bucket_name`	The bucket name for the object that is related to the event.
`Notification.content_type`	The MIME type of the object, for example, `text/rtf`.
`Notification.event_type`	The event type or operation, either type `Object:Write` or `Object:Delete`. Create or upload events are tagged as `Object:Write` operations.
`Notification.format`	This value is always `2.0`.
`Notification.object_etag`	A unique value that changes each time that the object is modified. This value does not display for an `Object:Delete` operation.
`Notification.object_length`	The size of the object, in bytes.
`Notification.request_id`	The unique ID that is related to the object change.
`Notification.request_time`	The time that the object change occurred.

Example output

CE_DATA={"bucket":"mybucket","endpoint":"","key":"Notes.rtf","notification":{"bucket_name":"mybucket","content_type":"text/rtf","event_type":"Object:Delete","format":"2.0","object_length":"4642","object_name":"Notes.rtf","request_id":"b59727ee-9c4e-446a-9261-5616f6d1283b","request_time":"2021-04-13T20:10:37.631Z"},"operation":"Object:Delete"}
CE_ID=b59727ee-9c4e-446a-9261-5616f6d1283b
CE_SOURCE=https://cloud.ibm.com/catalog/services/cloud-object-storage/mybucket
CE_SPECVERSION=1.0
CE_TIME=2021-04-13T20:10:37.631Z
CE_TYPE=com.ibm.cloud.cos.document.delete

Defining additional event attributes

When you create a subscription, you can define additional CloudEvent attributes to be included in any events that are generated. These attributes appear similar to any other CloudEvent attribute in the event delivery. If you choose to specify the name of an existing CloudEvent attribute, then it overrides the original value that was included in the event.

To define addition attributes, use the --extension options with the ibmcloud ce subscription cos create CLI command.

For more information, see Can I use other CloudEvents specifications?

Deleting a subscription

When you no longer need an IBM Cloud Object Storage subscription, you can delete it.

Deleting a subscription from the console

From the Code Engine Projects page, go to your project.
From the Overview page, click Event subscriptions to view a listing of defined subscriptions.
From the list of subscriptions, delete the subscription that you want to remove from your application or job.

If you delete an app or a job that is associated with the subscription, the subscription is not deleted. If you re-create the application or job (or another app or job with the same name), your subscription reconnects with the app or job.

Deleting a subscription with the CLI

You can delete an IBM Cloud Object Storage subscription by running the ibmcloud ce subscription cos delete command.

For example, use the following command to delete an IBM Cloud Object Storage subscription that is called mycosevent,

ibmcloud ce subscription cos delete --name mycosevent

If you delete an app or a job that is associated with the subscription, the subscription is not deleted. Instead, it moves to ready state of false because the subscription depends on the availability of the app or job. If you re-create the application or job (or another app or job with the same name), your subscription reconnects and the Ready state is true.

Working with the IBM Cloud Object Storage event producer

Setting up the IBM Cloud Object Storage event producer

Assigning the Notifications Manager role to Code Engine

Subscribing to IBM Cloud Object Storage events for an application

Subscribing to IBM Cloud Object Storage events for an application from the console

Subscribing to IBM Cloud Object Storage events for an application with the CLI

Viewing event information for an application from the console

Viewing event information for an application with the CLI

IBM Cloud Object Storage header and body information for events delivered to applications

Headers

HTTP body

Subscribing to IBM Cloud Object Storage events for a job

Subscribing to IBM Cloud Object Storage events for a job from the console

Subscribing to IBM Cloud Object Storage events for a job with the CLI

Viewing event information for a job from the console

Viewing event information for a job with the CLI

Environment variables for events delivered to jobs

CE_DATA environment variable

Defining additional event attributes

Deleting a subscription

Deleting a subscription from the console

Deleting a subscription with the CLI

`CE_DATA` environment variable