Here is some information about this architecture.
Here are the steps you can follow to build this solution on your own.
This project focuses on scheduling batch computing jobs using Amazon EventBridge and AWS Batch. You'll learn how to leverage EventBridge to schedule and run batch jobs in a highly scalable environment. The lessons will guide you through the configuration and execution of batch computing workflows, providing a strong foundation in managed batch processing.
If you're using the Skillmix Labs feature, open the lab settings (the beaker icon) on the right side of the code editor. Then, click the Start Lab button to start hte lab environment.
Wait for the credentials to load. Then run this in the terminal.
Be sure to enter in your own access key and secret key and name your profile 'smx-lab'.
$ aws configure --profile smx-lab
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: us-west-2
Default output format [None]: Note: If you're using your own AWS account you'll need to ensure that you've created and configured a named AWS CLI profile named smx-lab.
Next, we'll create the Terraform config. This config is used to specify the required provider plugins for our Terraform project. In this case, we are specifying the 'aws' provider from HashiCorp with a version constraint of '~>4.0'.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~>4.0"
}
}
}
Next, we'll create the aws config which is used to configure the AWS provider in Terraform. This config specifies the AWS profile and region to be used for provisioning resources.
provider "aws" {
profile = "smx-lab"
region = "${var.region}"
}
Next, we'll create the region config. This config is used to specify the AWS Region where we will be deploying our resources.
variable "region" {
type = string
description = "AWS Region where deploying resources"
default = "us-east-1"
}
Next, we'll create the aws_profile_name config. This config is used to specify the AWS CLI credentials profile name. By default, it is set to default.
variable "aws_profile_name" {
type = string
description = "AWS CLI credentials profile name"
default = "default"
}
Next, we'll create the vpc_cidr config. This config is used to define the CIDR block for the Batch VPC. The default value is set to '10.0.0.0/16'.
variable "vpc_cidr" {
type = string
description = "CIDR block for Batch VPC"
default = "10.0.0.0/16"
}
Next, we'll create the subnet_cidr config. This config is used to define the CIDR block for the Batch subnet.
variable "subnet_cidr" {
type = string
description = "CIDR block for Batch subnet"
default = "10.0.0.0/24"
}
Next, we'll create the aws_iam_policy_document config. This config is used to define an IAM policy document that specifies the permissions required for a role to be assumed by the ecs-tasks.amazonaws.com service.
data "aws_iam_policy_document" "assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
Next, we'll create the aws_iam_policy_document config. This config is used to define an IAM policy document that allows EC2 instances to assume a role.
data "aws_iam_policy_document" "ec2_assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
Next, we'll create the aws_iam_policy_document config. This config is used to define an IAM policy document that allows the batch.amazonaws.com service to assume the batch_assume_role role with the sts:AssumeRole action.
data "aws_iam_policy_document" "batch_assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["batch.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
Next, we'll create the project_name config. This config is used to define the local variable project_name with the value 'batch-job'.
locals {
project_name = "batch-job"
}
Next, we'll create the random_string config. This config is used to generate a random string of a specified length. In this case, the length is set to 16 characters, and the generated string will not contain any special characters or uppercase letters.
resource "random_string" "random" {
length = 16
special = false
upper = false
}
Next, we'll create the aws_iam_role config. This config is used to create an IAM role in AWS. In this case, we are creating a role called batch_job_execution_role with a dynamically generated name using the random_string resource. The assume_role_policy is set to the value of the json attribute of the assume_role_policy data source.
resource "aws_iam_role" "batch_job_execution_role" {
name = "StepFunctions-BatchJobManagementRole-${random_string.random.result}"
assume_role_policy = "${data.aws_iam_policy_document.assume_role_policy.json}"
}
Next, we'll create the aws_iam_role_policy_attachment config. This config is used to attach an IAM policy to an IAM role. In this case, we are attaching the AmazonECSTaskExecutionRolePolicy policy to the batch_job_execution_role IAM role.
resource "aws_iam_role_policy_attachment" "batch_job_execution_policy" {
role = "${aws_iam_role.batch_job_execution_role.name}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
Next, we'll create the aws_batch_job_definition config. This config is used to define a job definition for AWS Batch.
resource "aws_batch_job_definition" "batch_job" {
name = "StepFunctions-BatchJobManagement-${random_string.random.result}"
type = "container"
container_properties = "${jsonencode({
'command': ['ls', '-la'],
'image': 'busybox',
'resourceRequirements': [
{'type': 'VCPU', 'value': '1'},
{'type': 'MEMORY', 'value': '512'}
],
'volumes': [
{'host': {'sourcePath': '/tmp'}, 'name': 'tmp'}
],
'environment': [
{'name': 'VARNAME', 'value': 'VARVAL'}
],
'mountPoints': [
{'sourceVolume': 'tmp', 'containerPath': '/tmp', 'readOnly': false}
],
'ulimits': [
{'hardLimit': 1024, 'name': 'nofile', 'softLimit': 1024}
],
'executionRoleArn': '${aws_iam_role.batch_job_execution_role.arn}'
})}"
}
Next, we'll create the aws_iam_role config. This config is used to create an IAM role named ecs_instance_role_${random_string.random.result}. The role is used to define the permissions and policies that can be associated with EC2 instances.
resource "aws_iam_role" "ecs_instance_role" {
name = "ecs_instance_role_${random_string.random.result}"
assume_role_policy = "${data.aws_iam_policy_document.ec2_assume_role.json}"
}
Next, we'll create the aws_iam_role_policy_attachment config. This config is used to attach an IAM policy to an IAM role. In this case, we are attaching the AmazonEC2ContainerServiceforEC2Role policy to the ecs_instance_role IAM role.
resource "aws_iam_role_policy_attachment" "ecs_instance_role" {
role = "${aws_iam_role.ecs_instance_role.name}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}
Next, we'll create the aws_iam_instance_profile config. This config is used to create an IAM instance profile named ecs_instance_role_${random_string.random.result} and associate it with the ecs_instance_role IAM role.
resource "aws_iam_instance_profile" "ecs_instance_role" {
name = "ecs_instance_role_${random_string.random.result}"
role = "${aws_iam_role.ecs_instance_role.name}"
}
Next, we'll create the aws_vpc config which is used to define an Amazon Virtual Private Cloud (VPC) in Terraform.
resource "aws_vpc" "vpc" {
cidr_block = "${var.vpc_cidr}"
}
Next, we'll create the aws_internet_gateway config. This config is used to create an internet gateway in AWS.
resource "aws_internet_gateway" "gw" {
vpc_id = "${aws_vpc.vpc.id}"
}
Next, we'll create the aws_route_table config. This config is used to define a public route table in AWS.
resource "aws_route_table" "public_route" {
vpc_id = "${aws_vpc.vpc.id}"
route {
cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.gw.id}"
}
}
Next, we'll create the aws_subnet config. This config is used to define a subnet in an AWS VPC.
resource "aws_subnet" "subnet" {
vpc_id = "${aws_vpc.vpc.id}"
cidr_block = "${var.subnet_cidr}"
map_public_ip_on_launch = true
}
Next, we'll create the aws_route_table_association config. This config is used to associate a subnet with a route table in AWS.
resource "aws_route_table_association" "subnet_rt" {
subnet_id = "${aws_subnet.subnet.id}"
route_table_id = "${aws_route_table.public_route.id}"
}
Next, we'll create the aws_security_group config called 'batch_sg'. This config is used to create a security group for the Batch environment. The security group will have the following attributes: - Name: ${local.project_name}-sg - Description: Security group for Batch environment - VPC ID: ${aws_vpc.vpc.id} - Egress rules: Allow all traffic to any destination IP address:
resource "aws_security_group" "batch_sg" {
name = "${local.project_name}-sg"
description = "Security group for Batch environment"
vpc_id = "${aws_vpc.vpc.id}"
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
Next, we'll create the aws_iam_role config. This config is used to create an AWS IAM role for the Batch service. The role is named ${local.project_name}-batch-svc-role and has an assume role policy that allows the Batch service to assume this role. The role also has a managed policy attached, AWSBatchServiceRole, which provides the necessary permissions for the Batch service to perform its tasks.
resource "aws_iam_role" "batch_role" {
name = "${local.project_name}-batch-svc-role"
assume_role_policy = "${jsonencode({
'Version': '2012-10-17',
'Statement': [
{
'Action': 'sts:AssumeRole',
'Effect': 'Allow',
'Principal': {
'Service': 'batch.amazonaws.com'
}
}
]
})}"
managed_policy_arns = [
'arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole'
]
}
Next, we'll create the aws_iam_role config. This config is used to create an AWS IAM role called 'batch_ecs_role'. The role is used to allow EC2 instances to assume this role and perform actions on behalf of the role.
resource "aws_iam_role" "batch_ecs_role" {
name = "${local.project_name}-batch-ecs-role"
assume_role_policy = "${jsonencode({
'Version': '2012-10-17',
'Statement': [{
'Action': 'sts:AssumeRole',
'Effect': 'Allow',
'Principal': {'Service': 'ec2.amazonaws.com'}
}]
})}"
managed_policy_arns = [
'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role'
]
}
Next, we'll create the aws_iam_instance_profile config. This config is used to define an IAM instance profile, which is a container for an IAM role that you can use to pass role information to an EC2 instance when the instance starts.
resource "aws_iam_instance_profile" "ecs_profile" {
name = "${local.project_name}-batch-profile"
role = "${aws_iam_role.batch_ecs_role.name}"
}
Next, we'll create the aws_batch_compute_environment config. This config is used to define a managed compute environment for AWS Batch.
resource "aws_batch_compute_environment" "batch_compute_env" {
compute_environment_name = "BatchComputeEnvironment-${random_string.random.result}"
type = "MANAGED"
service_role = "${aws_iam_role.batch_role.arn}"
compute_resources {
max_vcpus = 64
min_vcpus = 0
desired_vcpus = 2
type = "EC2"
instance_role = "${aws_iam_instance_profile.ecs_profile.arn}"
security_group_ids = ["${aws_security_group.batch_sg.id}"]
instance_type = ["optimal"]
subnets = ["${aws_subnet.subnet.id}"]
}
depends_on = [
"${aws_iam_role.batch_ecs_role}",
"${aws_subnet.subnet}",
"${aws_security_group.batch_sg}",
"${aws_iam_role.batch_role}"
]
}
This config creates a managed compute environment named BatchComputeEnvironment-${random_string.random.result}. It uses the MANAGED type and requires a service role specified by ${aws_iam_role.batch_role.arn}. The compute resources are defined with a maximum of 64 vCPUs, a minimum of 0 vCPUs, and a desired count of 2 vCPUs. The compute resources use EC2 instances with the instance role specified by ${aws_iam_instance_profile.ecs_profile.arn}. The instances are associated with the security group ${aws_security_group.batch_sg.id} and are launched in the subnet ${aws_subnet.subnet.id}. Finally, this config has dependencies on ${aws_iam_role.batch_ecs_role}, ${aws_subnet.subnet}, ${aws_security_group.batch_sg}, and ${aws_iam_role.batch_role}.
Next, we'll create the aws_batch_job_queue config. This config is used to define a job queue in AWS Batch.
resource "aws_batch_job_queue" "batch_job_queue" {
name = "Scheduler-Batch-Queue-${random_string.random.result}"
state = "ENABLED"
priority = 1
compute_environments = ["${aws_batch_compute_environment.batch_compute_env.arn}"]
}
Next, we'll create the aws_scheduler_schedule config. This config is used to define a scheduled event that submits a job to an AWS Batch job queue.
resource "aws_scheduler_schedule" "batch-schedule" {
name = "batch-submit-job-schedule"
flexible_time_window = [{"mode": "OFF"}]
schedule_expression = "rate(5 minutes)"
schedule_expression_timezone= "US/Eastern"
description = "submitJob Batch event"
target {
arn = "arn:aws:scheduler:::aws-sdk:batch:submitJob"
role_arn = "${aws_iam_role.scheduler-batch-role.arn}"
input = "${jsonencode({
'JobName': 'scheduled-job',
'JobDefinition': '${aws_batch_job_definition.batch_job.arn}',
'JobQueue': '${aws_batch_job_queue.batch_job_queue.arn}'
})}"
}
}
This config creates a scheduler schedule named batch-submit-job-schedule. It uses a flexible time window mode of OFF, meaning the schedule will not be flexible. The schedule expression is set to run every 5 minutes in the US/Eastern timezone. The description is set to "submitJob Batch event".
The target of this schedule is an AWS Batch job with the ARN arn:aws:scheduler:::aws-sdk:batch:submitJob. It requires an IAM role with the ARN aws_iam_role.scheduler-batch-role.arn. The input for the job is a JSON-encoded string containing the job name, job definition ARN, and job queue ARN.
This config allows you to schedule the submission of jobs to an AWS Batch job queue at regular intervals.
Next, we'll create the aws_iam_policy config. This config is used to create an IAM policy named scheduler_batch_policy. The policy allows actions such as batch:SubmitJob, batch:DescribeJobs, and batch:TerminateJob on all resources, and also allows actions such as events:PutTargets, events:PutRule, and events:DescribeRule on all resources.
resource "aws_iam_policy" "scheduler_batch_policy" {
name = "scheduler_batch_policy"
policy = "${jsonencode({
'Version': '2012-10-17',
'Statement': [
{
'Action': ['batch:SubmitJob', 'batch:DescribeJobs', 'batch:TerminateJob'],
'Resource': '*',
'Effect': 'Allow'
},
{
'Action': ['events:PutTargets', 'events:PutRule', 'events:DescribeRule'],
'Resource': ['*'],
'Effect': 'Allow'
}
]
})}"
}
Next, we'll create the aws_iam_role config. This config is used to create an IAM role in AWS. The role is named scheduler-batch-role and it has a managed policy attached to it, specified by the ARN ${aws_iam_policy.scheduler_batch_policy.arn}. The assume_role_policy defines the permissions for the role, allowing the service scheduler.amazonaws.com to assume this role.
resource "aws_iam_role" "scheduler-batch-role" {
name = "scheduler-batch-role"
managed_policy_arns = ["${aws_iam_policy.scheduler_batch_policy.arn}"]
assume_role_policy = "${jsonencode({
'Version': '2012-10-17',
'Statement': [
{
'Action': 'sts:AssumeRole',
'Effect': 'Allow',
'Sid': '',
'Principal': {
'Service': 'scheduler.amazonaws.com'
}
}
]
})}"
}Deploy the Solution
Let's deploy this thing! If you haven't done so, start the Skillmix lab session and get the account credentials. Configure your Terraform environment to use those credentials.
Then, open a terminal or command prompt, navigate to the folder with your Terraform file, and execute these commands:
# initiatlize the project
$ terraform init
# show the plan
$ terraform plan
# apply the changes
$ terraform applyWait for the changes to be applied before proceeding.
Test the Solution
After deployment, view the schedule created in the Amazon EventBridge console under Scheduler>Schedules.
From the AWS Batch console, navigate to the Jobs section. The schedule will trigger a new job every 5 minutes, which will be visable in the Jobs section. You can also view the job status from the Dashboard section of the AWS Batch console in the Job queue overview.