Configuring an S3 Bucket to send events to a Lambda destination for processing

Amazon Simple Storage Service more commonly referred to as S3 is an extremely popular, robust and highly scalable Cloud Object Storage service. I imagine it was one of those benchmark services that started making Amazon Webservices popular along with its cloud compute services. Though cloud based storage is quite common, S3’s real power and effectiveness is in the seamless integration options with other AWS services, like the building block of AWS’ serverless compute offering, AWS Lambda, and that is what this article is going to cover.

A quick word on S3 Buckets and using their lifecycle events

The basic storage unit in S3 is known as a Bucket, aptly named as its just a cloud object container where you can dump anything into it literally. S3 Buckets – among the various other features related to permission management, url signing, site hosting and so on – offers an easy and robust way to send lifecycle event notifications to other AWS services about what is happening inside the bucket. Think at its most basic level – these lifecycle events include new object creations, or object deletes, but S3 support more granularity which we will get into in this article.

The power of this should be obvious – this seamless ability to subscribe to events makes S3 a powerful utility for all kinds of applications that may want to incorporate things like post processing or S3 bucket objects themselves – or for triggering other workflow events such as deleting records from a database when an object is removed – or signing urls to embed videos in sites and so on. But even outside of these classic cloud object processing use cases, there are more unusual examples of applications utilizing the power of S3 event notifications when using other Amazon services like Kinesis Firehose or Athena.

If you are interested for later, I have a full fledged example of an image moderation application which scans incoming images using Rekognition – which the S3 Bucket sending notifications to a Lambda destination setup we are going to review in this article.

Another quick word on S3 Buckets and CloudFormation stacks

This is relevant if you plan to follow along the CloudFormation Template example. If you plan to set this up manually on the console, this is not strictly relevant – but I encourage you to explore working with CloudFormation for all the benefits of Infrastructure-as-Code and DevOps CI/CD.

S3 Event Notifications can be defined for Lambdas in a CloudFormation template only if the bucket was created in the same CloudFormation stack.

Long story short, you cannot use CloudFormation to configure Lambda event destinations for existing buckets.

You will need to do so manually on the console for existing buckets.

Its a limitation AWS is aware of, and as of the time of this writing it still is a limitation.

It is what it is – for the purposes of demonstrating the setup in CloudFormation in this article, we will be creating a brand new bucket.

Listing supported S3 Lifecycle Events

S3 is able to send notifications on a fairly large list of lifecycle events broken down by categories.

These have the format of s3:[Category]:[EventName]. We will be using this format to subscribe to specific events in the CLoudFormation yaml, but in the console you can simply select these.

Do note for the purposes of the example I will only be subscribing to basic object create events though I am listing the rest as a reference here.

However, depending on your application’s specific requirements, as you can see, you can construct fairly fine grained subscriptions for pretty much a kitchen sink of lifecycle options to pick from ranging from the common create/delete to replication and storage reduction rule based deletion events!

Object creation

All object create events s3:ObjectCreated:*

Put s3:ObjectCreated:Put

Post s3:ObjectCreated:Post

Copy s3:ObjectCreated:Copy

Multipart upload completed s3:ObjectCreated:CompleteMultipartUpload

Object removal

All object removal events s3:ObjectRemoved:*

Permanently deleted s3:ObjectRemoved:Delete

Delete marker created s3:ObjectRemoved:DeleteMarkerCreated

Object restore

All restore object events s3:ObjectRestore:*

Restore initiated s3:ObjectRestore:Post

Restore completed s3:ObjectRestore:Completed

Restored object expired s3:ObjectRestore:Delete

Object ACL

Object ACL events s3:ObjectAcl:Put

Object tagging

All object tagging events s3:ObjectTagging:*

Object tags added s3:ObjectTagging:Put

Object tags deleted s3:ObjectTagging:Delete

Reduced Redundancy Storage

Reduced Redundancy Storage (RRS) object lost events s3:ReducedRedundancyLostObject

Replication

All replication events s3:Replication:*

Replication Time Control: Object exceeded 15 minute threshold s3:Replication:OperationMissedThreshold

Replication Time Control: Object replicated after 15 minute threshold s3:Replication:OperationReplicatedAfterThreshold

Object not tracked by replication s3:Replication:OperationNotTracked

Object failed to replicate s3:Replication:OperationFailedReplication

Lifecycle

Lifecycle transition events s3:LifecycleTransition

All lifecycle expiration events s3:LifecycleExpiration:*

Object expired s3:LifecycleExpiration:Delete

Delete marker added by Lifecycle for a versioned object s3:LifecycleExpiration:DeleteMarkerCreated

Intelligent-Tiering

Intelligent-Tiering archive events

s3:IntelligentTiering

Coding a Lambda to process object created events in an S3 Bucket

This is the Lambda we will be wiring up to process objects being created in the S3 Bucket.

We will get to wiring this next but first lets code a simple Lambda to have the foundations in place for more complex processing in a real application use case.

I am coding this in Python – but the principle workflow will be the same regardless of your own Lambda runtime language choice. Likewise the CloudFormation definitions will be the same except of course the specified runtime.

import boto3
import json
import urllib.parse

s3 = boto3.client('s3')

def s3_processor_lambda(event, context):
    
    try:

        #Get the bucket name and the object key from the incoming S3 event
        bucket = event['Records'][0]['s3']['bucket']['name']
        key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
        
        print(f"Object created in {bucket} with key {key}")
		
        s3.download_file(bucket, key, 'downloaded_file_name')
        #Do further processing as required
        
    except Exception as e:
        print(e)

It is a bare bones Lambda but still the foundation for any S3 processing application – the point is S3 will invoke this Lambda and provide details of the event to the Lambda – in this case we are extracting the bucket name and key to use the boto3 s3 client to download the file from the Bucket. As already linked earlier – in this Rekognition example, you can see how I make a subsequent call with the bucket and key to AWS Rekognition to run an image scan of the created file in the S3 Bucket. Like so, you can use this foundation to build on other needs.

Note – if you are planning to manually use the console, please create this Lambda directly in the AWS Lambda Management console and remember Lambda needs read access to the Bucket you plan to use – give your Lambda Admin access for its role, not specifically covering how to create a Lambda in the console here, if you don’t want to create a role that, for the purposes of following along, please simply comment out this line and everything else should work just fine.

#s3.download_file(bucket, key, 'downloaded_file_name')

If you are planning to follow the CloudFormation example – see the next section – roles and all are covered in the yaml definitions as you will see. For doing the same in the console, see the section after.

Defining the S3 Bucket and Lambda in the CloudFormation Template

Lets first get a base template going – first with the S3 Bucket in the console – as I mentioned earlier, this needs to be in the same template as the Lambda for the wiring to be done in CloudFormation.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: S3 Event Processing Stack

Globals:
  Function:
    Architectures:
      - arm64
    Runtime: python3.9
    Timeout: 300

Parameters:
  
  BucketName:
    Type: String
    Default: <USE_AN_ACCOUNT_AND_REGION_SPECIFIC_SUFFIX_AS_A_BEST_PRACTICE>

Resources:

  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: 
      	Ref: BucketName

As a best practice when it comes to naming your S3 Buckets – its a good idea to have an account and regional specific suffix for an identifier so the stack can be safely deployed to any region/account. This is because Buckets are global and the name cannot be shared.

I have the Bucket name as a parameter so this stack can be replicated elsewhere by passing a separate bucket name for creation

Defining the Lambda in the yaml

We are going to define the Lambda with the Events property subscribing to Object created events in the “S3Bucket resource”.

Doing it this way will have CloudFormation automatically create the permissions in the IAM roles for S3 to invoke the Lambda and we do not need to explicitly do so unlike in some other cases.

However since the Lambda downloads the file from S3, we need to explicitly give it read access to the S3 Bucket, which is done on the Policies property of the Lambda.

I like to set the DependsOn property to avoid CloudFormation dependency errors though in general it does a good job of figuring things out on its own without it.

S3ProcessorLambda:
    Type: AWS::Serverless::Function
    DependsOn: S3Bucket
    Properties:
      CodeUri: ./s3-processor-lambda/
      Handler: s3-processor-lambda.s3_processor_lambda
      FunctionName: s3_processor_lambda
      Policies:
        - Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: s3:GetObject*
              Resource: !Sub
                - "arn:aws:s3:::${bucketparam}*"
                - bucketparam:
                	Ref: BucketName
      Events:
        Upload:
          Properties:
            Bucket:
              Ref: S3Bucket
            Events: s3:ObjectCreated:*
          Type: S3

That is it really.

Build and deploy this template and you are done and good to test covered in the next section after the console guide below.

Creating an S3 Bucket Event Destination via the Console

Navigate to the Bucket you want to test and select the properties tab.

Then scroll all the way down to the “Event Notifications” section and hit the Create Notification button.

In the modal that is brought up, first enter a name and select the event types we want.

In our case, all created object events.

Next select the Lambda function that you created earlier.

Ignore the warning about permissions – this applies only to SNS/SQS. The permissions to invoke the Lambda will be done automatically once you select a Lambda and hit the Save changes button.

And now we are good to test.

Testing everything works

Simply upload an image to the bucket and navigate to CloudWatch for your Lambda function to see the details from the line below in our Lambda printed out.

print(f"Object created in {bucket} with key {key}")

If you see this, your setup is pretty much complete and you are good to expand on working with this CloudFormation stack for more complex real world applications and use cases.