Creating bucket notifications

Bucket notifications are S3 operations that can be created at the bucket level. The notification configuration has the IBM Storage Ceph Object Gateway S3 events, ObjectCreated, ObjectRemoved, and ObjectLifecycle:Expiration. The configurations need to be published with the destination to send the bucket notifications.

Before you begin

Before you begin, make sure that you have the following prerequisites in place:

A running IBM Storage Ceph cluster with Ceph Object Gateway.
A running HTTP Server, a RabbitMQ server, or a Kafka server.
Root-level access.
User access key and secret key.
Endpoint parameters.

About this task

Important: IBM supports ObjectCreate events, such as put, post, multipartUpload, and copy. IBM also supports ObjectRemove events, such as object_delete and s3_multi_object_delete.

Create bucket notifications in one of two ways:

Creating bucket notifications by using the boto script
Creating bucket notifications by using the AWS CLI

Creating bucket notifications by using the boto script

Procedure

Install the python3-boto3 package.

dnf install python3-boto3

For example,

[user@client ~]$  dnf install python3-boto3

Create an S3 bucket.

Create a python script topic.py to create an SNS topic for http,amqp, or kafka protocol.

For example,

import boto3
from botocore.client import Config
import sys

# endpoint and keys from vstart
endpoint = 'http://127.0.0.1:8000'
access_key='0555b35654ad1656d804'
secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='

client = boto3.client('sns',
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        config=Config(signature_version='s3'))

attributes = {"push-endpoint": "amqp://localhost:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}

client.create_topic(topic_name="mytopic", Attributes=attributes)

Run the python script for creating topic.
```
python3 topic.py
```

Create a python script notification.py to create S3 bucket notification for s3:objectCreate, s3:objectRemove, and s3:ObjectLifecycle:Expiration events.

For example,

import boto3
import sys

# bucket name as first argument
bucketname = sys.argv[1]
# topic ARN as second argument
topic_arn = sys.argv[2]
# notification id as third argument
notification_id = sys.argv[3]

# endpoint and keys from vstart
endpoint = 'http://127.0.0.1:8000'
access_key='0555b35654ad1656d804'
secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='

client = boto3.client('s3',
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)

# regex filter on the object name and metadata based filtering are extension to AWS S3 API
# bucket and topic should be created beforehand

topic_conf_list = [{'Id': notification_id, 
                    'TopicArn': topic_arn, 
                    'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*'],
                    }]
    client.put_bucket_notification_configuration(
        Bucket=bucketname,
          NotificationConfiguration={
            'TopicConfigurations': [
              {
                'Id': notification_id,
                'TopicArn': topic_arn,
                'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*', 's3:ObjectLifecycle:Expiration:*']
              }]})

Run the python script for creating the bucket notification.
```
python3 notification.py
```
Create S3 objects in the bucket.

Fetch the notification configuration.

import boto3
endpoint = 'http://127.0.0.1:8000'
access_key='0555b35654ad1656d804'
secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='

client = boto3.client('s3',
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)

# getting a specific notification configuration is an extension to AWS S3 API

print(client.get_bucket_notification_configuration(Bucket=bucketname))

Optional: Delete the objects.
Verify the object deletion events at the http, rabbitmq, or kafka receiver.

Creating bucket notifications by using the AWS CLI

Procedure

Create a topic.

aws --endpoint=AWS_END_POINT sns create-topic --name NAME --attributes=ATTRIBUTES_FILE

For example,

[user@client ~]$ aws --endpoint=http://localhost sns create-topic --name test-kafka --attributes=file://topic.json

 sample topic.json:
 {"push-endpoint": "kafka://localhost","verify-ssl": "False", "kafka-ack-level": "broker", "persistent":"true"}
 ref: https://docs.aws.amazon.com/cli/latest/reference/sns/create-topic.html

Create the bucket notification.

aws s3api put-bucket-notification-configuration --bucket BUCKET_NAME --notification-configuration NOTIFICATION_FILE

For example,

[user@client ~]$ aws s3api put-bucket-notification-configuration --bucket my-bucket --notification-configuration file://notification.json

 sample notification.json
 {
     "TopicConfigurations": [
         {
             "Id": "test_notification",
             "TopicArn": "arn:aws:sns:us-west-2:123456789012:test-kafka",
             "Events": [
                 "s3:ObjectCreated:*"
             ]
         }
     ]
 }

Fetch the notification configuration.

aws s3api --endpoint=AWS_ENDPOINT get-bucket-notification-configuration --bucket BUCKET_NAME

For example,

[user@client ~]$ aws s3api --endpoint=http://localhost get-bucket-notification-configuration --bucket my-bucket
 {
     "TopicConfigurations": [
         {
             "Id": "test_notification",
             "TopicArn": "arn:aws:sns:default::test-kafka",
             "Events": [
                 "s3:ObjectCreated:*"
             ]
         }
     ]
 }

For more information, see the S3 bucket notifications for event-driven architectures chapter, within the IBM Storage Ceph Solutions Guide Redpaper publication.