Lambda to Step Functions via Modules

How to Build This Solution

Here are the steps you can follow to build this solution on your own.

Get Your AWS Credentials

If you're using the Skillmix Labs feature, open the lab settings (the beaker icon) on the right side of the code editor. Then, click the Start Lab button to start hte lab environment.

Wait for the credentials to load. Then run this in the terminal:

$ aws configure --profile smx-lab
AWS Access Key ID [None]: AKIA3E3W34P42CSHXDH5
AWS Secret Access Key [None]: vTmqpOqefgJfse8i6QwzgpjgswPjHZ6h/oiQq4zf
Default region name [None]: us-west-2
Default output format [None]: json

Be sure to name your credentials profile 'smx-lab'.

Note: If you're using your own AWS account you'll need to ensure that you've created and configured a named AWS CLI profile named smx-lab.

Create the Terraform File

We'll be doing all of our work in one Terraform file. Create a new directory on your computer somewhere, and then create a file named main.tf in it.

Create the Terraform & Provider Block

Next, we will create a Terraform configuration that will allow us to use the AWS provider. This configuration will require us to specify the version of the AWS provider that we want to use, as well as the version of Terraform that we are using. We will also specify the AWS profile and region that we want to use. This code will ensure that the correct versions of Terraform and the AWS provider are used, and that the AWS provider is configured correctly.

Append this code to the main.tf file:

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.9"
    }
    random = {
      source  = "hashicorp/random"
      version = ">= 2.0"
    }
  }
}

provider "aws" {
  profile = "smx-temp"
  region = "us-west-2"
}

Create the Data Resource Blocks

Next, we will create a data source that will allow us to access the AWS caller identity of the current user. This data source will be called "aws_caller_identity" and will be set to the "current" provider, which is the AWS Cross Account provider. This data source will allow us to access the identity of the current user, which can be used for authentication and authorization purposes.

Append this code to the main.tf file:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

Create the Python File

In this project, the Lambda function will start the Step Function state machine. We need to create a Python file that can be used as this Lambda fuction.

In your project directory, create a folder named src. In that folder create a file named app.py. Put this code into that file:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0

import json
import boto3
import os
from aws_lambda_powertools import Logger

logger = Logger()
client = boto3.client('stepfunctions')
sfnArn = os.environ['SFN_ARN']

def lambda_handler(event, context):
    # TODO implement
    
    logger.info(f"Received Choice: {event['Choice']}")
    response = client.start_execution(
        stateMachineArn=sfnArn,
        input=json.dumps(event)
    )
    
    logger.info(f"Received Response: {response}")
    
    return {
        'statusCode': 200,
        'body': json.dumps(response,default=str)
    }

Create the Lambda Function Resource

Next, we will create the Terraform module "lambda_function". This uses the terraform-aws-modules/lambda/aws module on the public registry.

This Lambda function will have a runtime of "python3.8" and will be published. The source path for the Lambda function will be "${path.module}/src". The environment variables for the Lambda function will include "SFN_ARN = module.step_function.state_machine_arn" and it will also attach policy statements with the effect of "Allow" and the actions of "states:StartExecution" and the resources of "module.step_function.state_machine_arn".

Append this code to the main.tf file:

module "lambda_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 4.0"

  function_name = "${random_pet.this.id}-lambda"
  description   = "My awesome lambda function"
  handler       = "LambdaFunction.lambda_handler"
  runtime       = "python3.8"
  publish       = true

  source_path = "${path.module}/src"

  layers = ["arn:aws:lambda:${data.aws_region.current.name}:017000801446:layer:AWSLambdaPowertoolsPython:15"]

  environment_variables = {
    SFN_ARN = module.step_function.state_machine_arn
  }

  attach_policy_statements = true
  policy_statements = {
    step_function = {
      effect    = "Allow",
      actions   = ["states:StartExecution"],
      resources = [module.step_function.state_machine_arn]
    }
  }

  tags = {
    Pattern = "terraform-lambda-sfn"
    Module  = "lambda_function"
  }
}

Create the Step Function Definition

Next, we will create the locals block which defines a JSON-formatted string that contains the definition of the state machine. This block will be used to create the state machine in the next step.

This state machine will take different actions depending on the payload that is sent to the Lambda function. You can read through the state machine to see how it will perform based on different choice values.

Append this code to the main.tf file:

locals {
  definition = <<SFN
{
  "Comment": "State Machine example with various state types",
  "StartAt": "Pass State",
  "States": {
    "Pass State": {
      "Comment": "A Pass state passes its input to its output, without performing work. Pass states are useful when constructing and debugging state machines.",
      "Type": "Pass",
      "Next": "Wait State"
    },
    "Wait State": {
      "Comment": "A Wait state delays the state machine from continuing for a specified time. You can choose either a relative time, specified in seconds from when the state begins, or an absolute end time, specified as a timestamp.",
      "Type": "Wait",
      "Seconds": 3,
      "Next": "Choice State"
    },
    "Choice State": {
      "Comment": "A Choice state adds branching logic to a state machine.",
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.Choice",
          "StringEquals": "A",
          "Next": "Succeed State"
        },
        {
          "Variable": "$.Choice",
          "StringEquals": "B",
          "Next": "Parallel State"
        }
      ],
      "Default": "Error Handling State"
    },
    "Parallel State": {
      "Comment": "A Parallel state can be used to create parallel branches of execution in your state machine.",
      "Type": "Parallel",
      "Next": "Succeed State",
      "Branches": [
        {
          "StartAt": "Branch 1",
          "States": {
            "Branch 1": {
              "Type": "Pass",
              "Parameters": {
                "comment.$": "States.Format('Branch 1 Processing of Choice {}', $.Choice)"
              },
              "End": true
            }
          }
        },
        {
          "StartAt": "Branch 2",
          "States": {
            "Branch 2": {
              "Type": "Pass",
              "Parameters": {
                "comment.$": "States.Format('Branch 2 Processing of Choice {}', $.Choice)"
              },
              "End": true
            }
          }
        }
      ]
    },
    "Succeed State": {
      "Type": "Succeed",
      "Comment": "A Succeed state stops an execution successfully. The Succeed state is a useful target for Choice state branches that don't do anything but stop the execution."
    },
    "Error Handling State": {
      "Type": "Pass",
      "Parameters": {
        "error.$": "States.Format('{} is an invalid Choice.',$.Choice)"
      },
      "Next": "Fail State"
    },
    "Fail State": {
      "Type": "Fail"
    }
  }
}
SFN
}

Create the State Machine Resource

Next, we will create the Terraform module "step_function" which will create an AWS Step Function using the terraform-aws-modules/step-functions/aws public registry module. This module will define the name, definition, logging configuration, service integrations, and tags for the Step Function. The logging configuration will include execution data and the log level will be set to "ALL". The service integrations will include a Lambda function and X-Ray. Finally, the tags will include a pattern and module name.

Append this code to the main.tf file:

module "step_function" {
  source  = "terraform-aws-modules/step-functions/aws"
  version = "~> 2.0"

  name       = random_pet.this.id
  definition = local.definition

  logging_configuration = {
    include_execution_data = true
    level                  = "ALL"
  }

  service_integrations = {
    lambda = {
      lambda = [module.lambda_function.lambda_function_arn_static]
    }

    xray = {
      xray = true
    }
  }

  tags = {
    Pattern = "terraform-lambda-sfn"
    Module  = "step_function"
  }
}

Create Pet Name Resource

Next, we will create a random pet resource using Terraform. This code will generate a random pet name with two words. The random_pet resource will create a random pet name that is composed of two words, each of which is randomly selected from a list of words. This resource can be used to generate unique names for resources in your Terraform configuration.

Append this code to the main.tf file:

resource "random_pet" "this" {
  length = 2
}

Create Outputs

Let's output some helpful information. Append this to the end of the main.tf file.

# Lambda Function
output "lambda_function_arn" {
  description = "The ARN of the Lambda Function"
  value       = module.lambda_function.lambda_function_arn
}

output "lambda_function_arn_static" {
  description = "The static ARN of the Lambda Function. Use this to avoid cycle errors between resources (e.g., Step Functions)"
  value       = module.lambda_function.lambda_function_arn_static
}

output "lambda_function_invoke_arn" {
  description = "The Invoke ARN of the Lambda Function"
  value       = module.lambda_function.lambda_function_invoke_arn
}

output "lambda_function_name" {
  description = "The name of the Lambda Function"
  value       = module.lambda_function.lambda_function_name
}

output "lambda_function_qualified_arn" {
  description = "The ARN identifying your Lambda Function Version"
  value       = module.lambda_function.lambda_function_qualified_arn
}

output "lambda_function_version" {
  description = "Latest published version of Lambda Function"
  value       = module.lambda_function.lambda_function_version
}

# IAM Role of Lambda Function
output "lambda_role_arn" {
  description = "The ARN of the IAM role created for the Lambda Function"
  value       = module.lambda_function.lambda_role_arn
}

# State Machine
output "state_machine_id" {
  description = "The ARN of the State Machine"
  value       = module.step_function.state_machine_id
}

output "state_machine_arn" {
  description = "The ARN of the State Machine"
  value       = module.step_function.state_machine_arn
}

# IAM Role
output "step_function_role_arn" {
  description = "The ARN of the IAM role created for the State Machine"
  value       = module.step_function.role_arn
}

output "step_function_role_name" {
  description = "The name of the IAM role created for the State Machine"
  value       = module.step_function.role_name
}

Deploying the Project

Now that we have all of our code written, we can deploy the project. Open a terminal, navigate to the project, and run these commands.

# initialize the project 
$ terraform init 

# plan the project 
$ terraform plan 

# apply the project 
$ terraform apply

Test the Project

After deployment, invoke Lambda function with multiple inputs. Then go to the Step Function Console and view the different invocations to note the different behavior with the different inputs.

To do this, you can run these commands in the terminal (replace <function-name> with the value returned in lambda_function_name):

aws lambda invoke --function-name <function-name> --payload '{"Choice": "A"}' response.json
aws lambda invoke --function-name <function-name> --payload '{"Choice": "B"}' response.json
aws lambda invoke --function-name <function-name> --payload '{"Choice": "C"}' response.json

Destroy the Project

Time to destroy the project!

$ terraform destroy

Lambda to Step Functions via Modules

About this Architecture

How to Build This Solution

Get Your AWS Credentials

Create the Terraform File

Create the Terraform & Provider Block

Create the Data Resource Blocks

Create the Python File

Create the Lambda Function Resource

Create the Step Function Definition

Create the State Machine Resource

Create Pet Name Resource

Create Outputs

Deploying the Project

Test the Project

Destroy the Project