API Gateway to SQS

How to Build This Solution

Here are the steps you can follow to build this solution on your own.

Here are the steps needed to build this architecture.

Get Your AWS Credentials

If you're using the Skillmix Labs feature, open the lab settings (the beaker icon) on the right side of the code editor. Then, click the Start Lab button to start hte lab environment.

Wait for the credentials to load. Then run this in the terminal:

$ aws configure --profile smx-lab
AWS Access Key ID [None]: AKIA3E3W34P42CSHXDH5
AWS Secret Access Key [None]: vTmqpOqefgJfse8i6QwzgpjgswPjHZ6h/oiQq4zf
Default region name [None]: us-west-2
Default output format [None]: json

Be sure to name your credentials profile 'smx-lab'.

Note: If you're using your own AWS account you'll need to ensure that you've created and configured a named AWS CLI profile named smx-lab.

Create the Terraform File

We'll be doing all of our work in one Terraform file. Create a new directory on your computer somewhere, and then create a file named main.tf in it.

Create the Terraform & Provider Block

The first step is to create a Terraform and Provider blocks. These blocks bootstrap our project with the required plugins and provider configuration.

This code will:

Add the AWS provider to the project
Set the required version of Terraform
Configure the AWS provider to use a specific credential profile and region

Append this code to the main.tf file:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "smx-lab"
  region  = "us-west-2"
}

Create the Data Resources

The AWS provider makes several data resources available. Here, we configure two of them. The first is aws_caller_identity, and it lets us access things like the Account ID, User ID, and ARN in which Terraform is authorized (see the provider block above).

Second is the aws_region data block which gets the current region.

Append this code to the main.tf file:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

Create the API Gateway Resource

The first major resource we'll create is the API Gateway resource. This resource will deploy and configure our gateway instance.

The main configuration element is the body argument. Here, we use the OpenAPI spec to configure the gateway. Specifically, we will configure:

Basic info such as title and version
Set a base path for our paths aka routes
Create a /submit path that accepts POSTS requests
Integrate the /submit route with SQS using the x-amazon-apigateway-integration block.

Append this code to the main.tf file:

resource "aws_api_gateway_rest_api" "MyApiGatewayRestApi" {
  name = "APIGW SQS Serverless Pattern Demo"
  body = jsonencode({
    "openapi" : "3.0.1",
    "info" : {
      "title" : "APIGW SQS Serverless Pattern Demo",
      "version" : "2022-03-03T00:00:00Z"
    },
    "servers" : [{
      "variables" : {
        "basePath" : {
          "default" : "/default"
        }
      }
    }],
    "paths" : {
      "/submit" : {
        "post" : {
          "responses" : {
            "200" : {
              "description" : "200 response",
              "content" : {
                "application/json" : {
                  "schema" : {
                    "$ref" : "#/components/schemas/Empty"
                  }
                }
              }
            }
          },
          "x-amazon-apigateway-integration" : {
            "type" : "aws",
            "credentials" : "${aws_iam_role.APIGWRole.arn}",
            "httpMethod" : "POST",
            "uri" : "arn:aws:apigateway:${data.aws_region.current.name}:sqs:path/${data.aws_caller_identity.current.account_id}/${aws_sqs_queue.MySQSqueue.name}",
            "responses" : {
              "default" : {
                "statusCode" : "200"
              }
            },
            "requestParameters" : {
              "integration.request.header.Content-Type" : "'application/x-www-form-urlencoded'"
            },
            "requestTemplates" : {
              "application/json" : "Action=SendMessage&MessageBody=$input.body"
            },
            "passthroughBehavior" : "never"
          }
        }
      }
    },
    "components" : {
      "schemas" : {
        "Empty" : {
          "title" : "Empty Schema",
          "type" : "object"
        }
      }
    }
  })
}

The x-amazon-apigateway-integration is where the integration with SQS is made. Here are some notes about how this is configured:

type: This property specifies the type of integration, which is "aws" in this case, meaning that it's an integration with an AWS service.
credentials: This property provides the ARN (Amazon Resource Name) of the IAM (Identity and Access Management) role that will be used by the API Gateway to access the SQS queue.
httpMethod: This property specifies the HTTP method used to call the integration. In this case, it's "POST".
uri: This property provides the ARN of the SQS queue that the API Gateway will interact with. This SQS queue will be created later in this project.
responses: This property defines the default response that will be returned when the integration is successful. In this case, it's a 200 status code.
requestParameters: This property defines the request parameters that will be passed to the integration. In this case, it specifies that the "Content-Type" header should be set to "application/x-www-form-urlencoded".
requestTemplates: This property defines the request templates that will be used to transform the incoming request before it's sent to the integration. In this case, it specifies that the request body will be transformed into a URL-encoded string with the action set to "SendMessage" and the message body equal to the incoming request body.
passthroughBehavior: This property specifies the behavior of the integration when the response from the backend is an error. In this case, it's set to "never", meaning that errors will not be passed through to the caller of the API Gateway.

Create the API Gateway Deployment Resource

Next, we will create the deployment resource. A deployment is a snapshot of the REST API configuration. This deployment is published to stages, which we'll create next. For now all we have to do is create the deployment.

Append this code to the main.tf file:

resource "aws_api_gateway_deployment" "MyApiGatewayDeployment" {
  rest_api_id = aws_api_gateway_rest_api.MyApiGatewayRestApi.id
}

Create the API Gateway Stage Resource

Next, let's create the stage resource. Stages are lifecycle states, such as dev, test, production. They are paired with a deployment configuration.

In this resource we will define the deployment resource, gateway resource, and stage name. We also set the CloudWatch log group ARN, and the logging format.

Append this code to the main.tf file:

resource "aws_api_gateway_stage" "MyApiGatewayStage" {
  deployment_id = aws_api_gateway_deployment.MyApiGatewayDeployment.id
  rest_api_id   = aws_api_gateway_rest_api.MyApiGatewayRestApi.id
  stage_name    = "default"

  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.MyLogGroup.arn
    format          = "{ \"requestId\":\"$context.requestId\", \"ip\": \"$context.identity.sourceIp\", \"requestTime\":\"$context.requestTime\", \"httpMethod\":\"$context.httpMethod\",\"routeKey\":\"$context.routeKey\", \"status\":\"$context.status\",\"protocol\":\"$context.protocol\", \"responseLength\":\"$context.responseLength\" }"
  }
}

Create the API Gateway Account Setting, IAM Role & Policy Resources

The API Gateway REST version has an account setting parameter. Here we will configure the CloudWatch role in the gateway account settings. We will also create an IAM Role and attach a policy to it.

The Role will give permission to the apigateway.amazonaws.com to assume the role and take actions on its behalf. The policy will give logging access to the role.

Append this code to the main.tf file:

resource "aws_api_gateway_account" "ApiGatewayAccountSetting" {
  cloudwatch_role_arn = aws_iam_role.APIGatewayCloudWatchRole.arn
}

resource "aws_iam_role" "APIGatewayCloudWatchRole" {
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "APIGatewayCloudWatchPolicy" {
  role = aws_iam_role.APIGatewayCloudWatchRole.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:FilterLogEvents"
            ],
            "Resource": "*"
        }
    ]
}
EOF
}

Create the API Gateway Method Settings

Next, we will create a method settings resource. This resource is used to set the logging levels for gateway paths. In this configuration we are setting logging for all paths.

Append this code to the main.tf file:

resource "aws_api_gateway_method_settings" "MyApiGatewaySetting" {
  rest_api_id = aws_api_gateway_rest_api.MyApiGatewayRestApi.id
  stage_name  = aws_api_gateway_stage.MyApiGatewayStage.stage_name
  method_path = "*/*"

  settings {
    # Enable CloudWatch logging and metrics
    metrics_enabled = true
    logging_level   = "INFO"
  }
}

Create the API Gateway IAM Role & Policy

Next, we need to create an IAM Role and Policy for the gateway itself. The role is configured to allow the apigateway.amazonaws.com to assume it. The policy gives permission for the gateway to send messages to SQS.

Append this code to the main.tf file:

resource "aws_iam_role" "APIGWRole" {
  # uncomment the 'permissions_boundary' argument if running this lab on skillmix.io 
  # permissions_boundary = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:policy/LabUserNewResourceBoundaryPolicy"
  assume_role_policy = <<POLICY1
{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Effect" : "Allow",
      "Principal" : {
        "Service" : "apigateway.amazonaws.com"
      },
      "Action" : "sts:AssumeRole"
    }
  ]
}
POLICY1
}

resource "aws_iam_policy" "APIGWPolicy" {
  policy = <<POLICY2
{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Effect" : "Allow",
      "Action" : [
        "sqs:SendMessage",
        "sqs:SendMessageBatch"
      ],
      "Resource" : "${aws_sqs_queue.MySQSqueue.arn}"
    }
  ]
}
POLICY2
}

resource "aws_iam_role_policy_attachment" "APIGWPolicyAttachment" {
  role       = aws_iam_role.APIGWRole.name
  policy_arn = aws_iam_policy.APIGWPolicy.arn
}

Create the SQS Queue

We do need a SQS queue for this project. This queue will receive messages from our API Gateway. Append this code to the main.tf file:

resource "aws_sqs_queue" "MySQSqueue" {
}

Create the CloudWatch Log Group

We also need a CloudWatch group. Append this code to the main.tf file:

resource "aws_cloudwatch_log_group" "MyLogGroup" {
  name_prefix = "/aws/APIGW/terraform"
}

Create the CloudWatch Log Resource Policy

Next, we will create a CloudWatch log policy that allows CloudWatch to create log streams and put items into that stream.

Append this code to the main.tf file:

resource "aws_cloudwatch_log_resource_policy" "MyCloudWatchLogPolicy" {
  policy_name     = "Terraform-CloudWatchLogPolicy-${data.aws_caller_identity.current.account_id}"
  policy_document = <<POLICY3
{
  "Version": "2012-10-17",
  "Id": "CWLogsPolicy",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [ 
          "apigateway.amazonaws.com",
          "delivery.logs.amazonaws.com"
          ]
      },
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
        ],
      "Resource": "${aws_cloudwatch_log_group.MyLogGroup.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_api_gateway_rest_api.MyApiGatewayRestApi.arn}"
        }
      }
    }
  ]
}
POLICY3  
}

Create the Terraform Outputs

The last step here is to create the Terraform outputs.

Append this code to the main.tf file:

output "SQS-QUEUE" {
  value       = aws_sqs_queue.MySQSqueue.id
  description = "The SQS Queue URL"
}

output "APIGW-URL" {
  value       = aws_api_gateway_stage.MyApiGatewayStage.invoke_url
  description = "The API Gateway Invocation URL Queue URL"
}

# Command for testing to send data to api gateway
output "Test-Command1" {
  value       = "curl --location --request POST '${aws_api_gateway_stage.MyApiGatewayStage.invoke_url}/submit' --header 'Content-Type: application/json'  --data-raw '{ \"TestMessage\": \"Hello From ApiGateway!\" }'"
  description = "Command to invoke the API Gateway"
}

# Command for testing to retrieve the message from the SQS queue
output "Test-Command2" {
  value       = "aws sqs receive-message --queue-url ${aws_sqs_queue.MySQSqueue.id}"
  description = "Command to query the SQS Queue for messages"
}

Deploy the Solution

Let's deploy this thing! If you haven't done so, start the Skillmix lab session and get the account credentials. Configure your Terraform environment to use those credentials.

Then, open a terminal or command prompt, navigate to the folder with your Terraform file, and execute these commands:

# initiatlize the project 
$ terraform init 

# show the plan 
$ terraform plan 

# apply the changes 
$ terraform apply

Wait for the changes to be applied before proceeding.

Test the Solution

To test the solution we need to execute a few commands at the command line. Open your terminal and execute these. Remember to replace values with the values from your Terraform outputs.

You will need the AWS CLI installed and configured.

# create a SQS message by sending a request to the gateway 
$ curl --location --request POST '{MyHttpAPI}/submit' > --header 'Content-Type: application/json' \ > --data-raw '{ "isMessageReceived": "Yes" }' # read the message queue aws sqs receive-message --queue-url ENTER_YOUR_QUEUE_URL --profile smx-lab

Source

This project was sourced from the AWS Repo: https://github.com/aws-samples/serverless-patterns

API Gateway to SQS

About this Architecture

How to Build This Solution

Get Your AWS Credentials

Create the Terraform File

Create the Terraform & Provider Block

Create the Data Resources

Create the API Gateway Resource

Create the API Gateway Deployment Resource

Create the API Gateway Stage Resource

Create the API Gateway Account Setting, IAM Role & Policy Resources

Create the API Gateway Method Settings

Create the API Gateway IAM Role & Policy

Create the SQS Queue

Create the CloudWatch Log Group

Create the CloudWatch Log Resource Policy

Create the Terraform Outputs

Deploy the Solution

Test the Solution