S3 to Lambda via Notifications

Create the Terraform File

We'll be doing all of our work in one Terraform file. Create a new directory on your computer somewhere, and then create a file named main.tf in it.

Create the Terraform & Provider Block

Next, we will create a Terraform configuration that will allow us to use the AWS provider. This configuration will require us to specify the version of the AWS provider that we want to use, as well as the version of Terraform that we are using. We will also specify the AWS profile and region that we want to use. This code will ensure that the correct versions of Terraform and the AWS provider are used, and that the AWS provider is configured correctly.

Append this code to the main.tf file:

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.9"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 2.0"
    }
  }
}

provider "aws" {
  profile = "smx-temp"
  region = "us-west-2"
}

Create the Data Resource Blocks

Next, we will create a data source that will allow us to access the AWS caller identity of the current user. This data source will be called "aws_caller_identity" and will be set to the "current" provider, which is the AWS Cross Account provider. This data source will allow us to access the identity of the current user, which can be used for authentication and authorization purposes.

Append this code to the main.tf file:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

Create the Python File

We will need a file Python file to use in this project. For now, this can be a simple Python file that just logs event information.

In your project directory, create a folder named src. In that folder create a file named app.py. Put this code into that file:

import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def lambda_handler(event, context):
    logging.info(json.dumps(event, indent=2))

    # [ERROR] TypeError: Object of type LambdaContext is not JSON serializable
    # logging.info(json.dumps(context, indent=2))

    eventObject = {
        "hello": "Hello Python! Hello Terraform!",
        "functionName": context.function_name,
        "event": event
    }

    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": json.dumps(eventObject)
    }

Create the Lambda Function

Next, we will create the Lambda function module. This module will create a Lambda function with the name generated from the random_pet resource, a description of "My awesome lambda function", a handler of "index.lambda_handler", a runtime of "python3.8", and will be published. The source path for the Lambda function will be set to the terraform-fixtures/python directory. Additionally, the module will allow execution from the S3 bucket created in the S3 bucket module, and will be tagged with the Pattern and Module tags.

Append this code to the main.tf file:

module "lambda_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 4.0"

  function_name = "${random_pet.this.id}-lambda"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"
  publish       = true

  source_path = "${path.module}/src"

  allowed_triggers = {
    AllowExecutionFromS3Bucket = {
      service    = "s3"
      source_arn = module.s3_bucket.s3_bucket_arn
    }
  }

  tags = {
    Pattern = "terraform-s3-lambda"
    Module  = "lambda_function"
  }
}

Create the S3 Bucket

Next, we will create the S3 bucket using the Terraform module "s3_bucket". This module will create an S3 bucket with the name generated from the random_pet resource, and will also add two tags to the bucket for easier identification. The force_destroy parameter will ensure that the bucket is deleted when the Terraform configuration is destroyed.

Append this code to the main.tf file:

module "s3_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 3.0"

  bucket        = "${random_pet.this.id}-bucket"
  force_destroy = true

  tags = {
    Pattern = "terraform-s3-lambda"
    Module  = "s3_bucket"
  }
}

Create S3 Notifications

Next, we will create the S3 notification module. This module will enable us to set up an event bridge that will trigger a Lambda function when an object is created in the S3 bucket. We will specify the Lambda function's ARN and name, as well as the events and filters that will trigger the Lambda function.

Append this code to the main.tf file:

module "s3_notification" {
  source  = "terraform-aws-modules/s3-bucket/aws//modules/notification"
  version = "~> 3.0"

  bucket = module.s3_bucket.s3_bucket_id

  eventbridge = true

  lambda_notifications = {
    lambda1 = {
      function_arn  = module.lambda_function.lambda_function_arn
      function_name = module.lambda_function.lambda_function_name
      events        = ["s3:ObjectCreated:*"]
      filter_prefix = "data/"
      filter_suffix = ".json"
    }
  }
}

Create Pet Name Resource

Next, we will create a random pet resource. This code will generate a random pet name with two words. The random_pet resource will create a random pet name that is composed of two words, each of which is randomly selected from a list of words. This resource can be used to generate unique names for resources in your Terraform configuration.

Append this code to the main.tf file:

resource "random_pet" "this" {
  length = 2
}

Create Outputs

Let's output some helpful information. Append this to the end of the main.tf file.

# Lambda Function
output "lambda_function_arn" {
  description = "The ARN of the Lambda Function"
  value       = module.lambda_function.lambda_function_arn
}

output "lambda_function_invoke_arn" {
  description = "The Invoke ARN of the Lambda Function"
  value       = module.lambda_function.lambda_function_invoke_arn
}

output "lambda_function_name" {
  description = "The name of the Lambda Function"
  value       = module.lambda_function.lambda_function_name
}

output "lambda_function_qualified_arn" {
  description = "The ARN identifying your Lambda Function Version"
  value       = module.lambda_function.lambda_function_qualified_arn
}

output "lambda_function_version" {
  description = "Latest published version of Lambda Function"
  value       = module.lambda_function.lambda_function_version
}

# IAM Role of Lambda Function
output "lambda_role_arn" {
  description = "The ARN of the IAM role created for the Lambda Function"
  value       = module.lambda_function.lambda_role_arn
}

# S3 bucket
output "s3_bucket_id" {
  description = "The name of the bucket."
  value       = module.s3_bucket.s3_bucket_id
}

output "s3_bucket_arn" {
  description = "The ARN of the bucket. Will be of format arn:aws:s3:::bucketname."
  value       = module.s3_bucket.s3_bucket_arn
}

Deploying the Project

Now that we have all of our code written, we can deploy the project. Open a terminal, navigate to the project, and run these commands.

# initialize the project 
$ terraform init 

# plan the project 
$ terraform plan 

# apply the project 
$ terraform apply

Test the Project

After deployment, upload file to the S3 bucket. Make sure that file matches filter prefix and suffix to trigger a notification. The filter prefix is filter_prefix = "data/" and the suffix is filter_suffix = ".json". A matching file would be data/test.json.

Here's how you can upload using the AWS CLI:

$ echo "Test file" > test.json
$ aws s3 cp test.json s3://<bucket-name>/data/

Then, go to the CloudWatch Logs for the deployed Lambda function. You will see the event is logged out containing the item data.

Destroy the Project

Time to destroy the project!

$ terraform destroy