Building and deploying a web application on Fargate with Terraform Takashi Yamahara @shonansurvivors

● Takashi Yamahara - @shonansurvivors ● Site Reliability Engineer ● AWS User Group - Japan, Beginner branch member About Me

Agenda 0. Precondition 1. Separate Terraform state by: ・Component ・Environment ・[Appendix] Tips for Workspaces 2. Deploy ECS application with ecspresso 3. Summary

0. Precondition

Repositories Today’s session Note: This is not to say that the above case is superior.

Out of scope ● Terraform Cloud ● Terragrunt

1. Separate Terraform state

Separate Terraform state Monolith Separated

Separate state by component

Separate state by component ● Consider: ○ Stability ○ Stateful or not ○ Lifecycle . ├── app # ECR / ECS / IAM Roles(for ECS) / S3(for .env file) ├── cicd # IAM Role(for ci tool) / Data resources(for ecspresso) ├── datastore # RDS / IAM Role(for RDS) / ElastiCache ├── log # CloudWatch log groups / S3 ├── monitor # CloudWatch Alarm / SNS ├── network # VPC / Subnets / VPC Endpoints / Security Groups etc... ├── ops # IAM Role(with SAML) └── routing # Route53 / ACM / CloudFront / ALB

Separate state by component Moreover, create component directories under each category while considering separation of concern. It helps to minimize the blast radius of failures. If you would to refer to other component attributes, use terraform_remote_state or data sources. . ├── app │ └── service1 │ ├── backend.tf │ ├── data.tf │ ├── ecr.tf │ ├── … │ ├── locals.tf │ ├── outputs.tf │ ├── provider.tf -> ../../provider.tf │ └── shared_locals.tf -> ../../shared_locals.tf ├── cicd │ └── service1 ├── datasrore │ ├── service1_elasticache │ └── service1_rds ├── log │ ├── alb │ ├── app_service1 │ ├── cloudfront │ └── datastore_service1_rds ├── network │ └── main ├── routing │ ├── example_com │ └── example_internal ├── provider.tf └── shared_locals.tf

Separate state by environment

Separate state by environment There are several ways to do this. 1. Environment directories + Shared modules 2. Environment directories + Symbolic links 3. Only environment directories 4. Workspaces

1. Env dirs + Shared modules . └── envs ├── dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── main.tf # calls shared module │ │ ├── something.tf # env-specific resources │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── provider.tf │ └── shared_locals.tf ├── stg │ └── app │ └── service1 ├── prod │ └── app │ └── service1 └── shared_modules ├── app │ └── service1 │ ├── ecr.tf │ ├── ... │ ├── outputs.tf │ └── variables.tf ├── ... └── routing Create directories for each env and modularize each component. If you need environment-specific resources, put them under the environment subdirectory side.

2. Env dirs + Symbolic links Using symbolic link, refer to shared files from each environment. If you need environment-specific resources, put them under the environment subdirectory side. . └── envs ├── dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── main.tf -> ../../../shared/app/service1/main.tf │ │ ├── something.tf # env-specific resources │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── stg │ └── app │ └── service1 │ ├── main.tf -> ../../../shared/app/service1/main.tf │ └── ... ├── prod │ └── app │ └── service1 │ ├── main.tf -> ../../../shared/app/service1/main.tf │ └── ... └── shared ├── app │ └── service1 │ └── main.tf ├── ... └── routing

3. Only environment directories Put resources under each environment subdirectory. It’s not DRY. However, this structure is easy to reflect requirements of each environment. You don't have to think about shared-modules I/O or using state mv command for migtation from shared to env-specific. . └── envs ├── dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── stg │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf └── prod └── ...

4. Workspaces Using workspaces, apply same code base to multiple environments. . ├── app │ └── service1 │ ├── backend.tf │ ├── ecr.tf │ ├── ... │ ├── locals.tf │ ├── outputs.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf $ terraform workspace new dev $ terraform workspace select dev $ terraform plan $ terraform apply

[Appendix] Tips for Workspaces

[Appendix] Tips for Workspaces 1. Configure for each environment ○ terraform.workspace + Local Values ○ Variables(Input Values) + tfvars 2. Destination bucket for states 3. Wrapper script for workspaces

terraform.workspace + Local Values . ├── datastore │ └── service1_rds │ ├── rds.tf │ ├── ... │ ├── locals.tf │ ├── backend.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf # Execute $ terraform \ workspace select dev $ terraform plan $ terraform apply resource "aws_db_instance" "this" { engine = "mysql" instance_class = local.aws_db_instance.instance_class[terraform.workspace] // } locals { aws_db_instance = { instance_class = { dev = "db.t3.small" stg = "db.m5.large" prod = "db.m5.large" } // } }

Variables(Input Values) + tfvars . ├── datastore │ └── service1_rds │ ├── rds.tf │ ├── ... │ ├── variables.tf │ ├── dev.tfvars │ ├── stg.tfvars │ ├── prod.tfvars │ ├── backend.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf # dev.tfvars aws_db_instance = { instance_class = "db.t3.small" // } } # Execute $ terraform \ workspace select dev $ terraform plan \ -var-file=dev.tfvars $ terraform apply \ -var-file=dev.tfvars resource "aws_db_instance" "this" { engine = "mysql" instance_class = var.aws_db_instance.instance_class // } variable "aws_db_instance" { type = object({ instance_class = string // }) }

Destination bucket for states By default, a state for each workspace is stored in a single S3 bucket. Default My requirement 🤔 😲

Do you use variables? Variables cannot be used in backend block. terraform { backend "s3" { bucket = "example-${terraform.workspace}" key = "app/service1.tfstate" region = "ap-northeast-1" profile = terraform.workspace } }

Use -backend-config Switch state bucket while using workspaces. $ terraform init -reconfigure \ -backend-config "bucket=example-dev" \ -backend-config "key=app/service1.tfstate" \ -backend-config "region=ap-northeast-1" \ -backend-config "profile=dev" $ terraform workspace select dev $ terraform plan $ terraform apply $ terraform init -reconfigure \ -backend-config=dev.conf $ terraform workspace select dev $ terraform plan $ terraform apply # dev.conf bucket = "example-dev" key = "app/service1.tfstate" region = "ap-northeast-1" profile = "dev" Or

workspace select ? init -backend-config=? (If you used tfvars) apply -var-file=?.tfvars

Wrapper script for Workspaces
 . ├── components │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── .env.dev ├── .env.stg ├── .env.prod ├── terraform.sh └── modules # .env.dev BUCKET=example-dev REGION=ap-northeast-1 PROFILE=dev # terraform.sh(The whole is about 100 lines.) ... source ${BASE_DIR}/.env.${ENV} ... terraform init -reconfigure \ -backend-config "bucket=${BUCKET}" \ -backend-config "key=${TARGET_DIR}.tfstate" \ -backend-config "region=${REGION}" \ -backend-config "profile=${PROFILE}" ... terraform workspace select ${ENV} ... cd ${BASE_DIR}/${TARGET_DIR} terraform ${TF_CMD} ${TF_ARGS} # Usage ./terraform.sh [...] # Example ./terraform.sh dev components/app/service1 plan

2. Deploy ECS application with ecspresso

About ecspresso ● kayac/ecspresso is a deployment tool for Amazon ECS. ● It reads some files, then register a new task definition revision and update a ECS service.

ecspresso goes well with Terraform ● ecspresso can read a state, so we don't have to do any hard coding. # config.yaml plugins: - name: tfstate config: url: s3://bucket/key # ecs-service-def.json "networkConfiguration": { "awsvpcConfiguration": { "securityGroups": [ "{{ tfstate `aws_security_group.default.id` }}" ] "subnets": [ "{{ tfstate `aws_subnet.private['a'].id` }}", "{{ tfstate `aws_subnet.private['c'].id` }}" ] } },

ecspresso can read one state However, ecspresso can read only one state file. Don’t separated states and ecspresso go together? . ├── app │ └── service1 │ ├── ecr.tf │ ├── iam.tf │ ├── s3.tf │ └── … ├── cicd │ └── app_service1 │ ├── … │ └── … ├── datasrore ├── log │ └── app_service1 │ ├── cloudwatch_log.tf │ └── … ├── network │ └── main │ ├── security_group.tf │ ├── subnet.tf │ └── … └── routing └── example_com ├── lb.tf └── … state state state state ecspresso 🤔

Put data sources into a component Put data sources referring to all resources required for ecs-service-def.json and ecs-task-def.json into a component such as CI/CD. Then you make ecspresso read the state. . ├── app │ └── service1 │ ├── ecr.tf │ ├── iam.tf │ ├── s3.tf │ └── … ├── cicd │ └── app_service1 │ ├── ecspresso.tf │ └── … ├── datasrore ├── log │ └── app_service1 │ ├── cloudwatch_log.tf │ └── … ├── network │ └── main │ ├── security_group.tf │ ├── subnet.tf │ └── … └── routing └── example_com ├── lb.tf └── … state ecspresso # ecspresso.tf data "aws_xxx" "foo" { name = "foo" } ... ☺ Data sources

Summary

Summary ● Separate states by environment and component. ○ Minimize the blast radius of failures. ● There are several ways to do this, therefore think about it. ○ Shared modules, symbolic links, Only environment directories, or workspaces ... ● Use a wrapper script on an as-needed basis. ○ Reduce the complexity of dealing with workspaces or many directories. ● ecspresso — ECS deployment tool — can read a state. ○ Put data sources associated with it into anyone of states.

References ● Japanese ○ 実践Terraform AWSにおけるシステム設計とベストプラクティス - Tomoki Nomura ○ Terraformのコンポーネント分割について検討する - Takashi Narikawa ○ 「それ、どこに出しても恥ずかしくない Terraformコードになってるか？」 - Yuki Yoshida ○ Terraform ベストプラクティス 2020 春 ~moduleやめてみた~ - kenzo0107 ○ layerx-invoice-practical-devops-20211029 - Shinji Takae ○ Terraformなにもわからないけどディレクトリ構成の実例を晒して人類に貢献したい - Yuichiro Fukubayashi ○ CCoE による Terraform を活用した Infrastructure as Code - Yuuki Nagahara ○ ecspresso handbook - FUJIWARA Shunichiro ● English ○ Multi-layering: Terraform IaC from scratch to scale - Antoine Chapusot ○ ozbillwang/terraform-best-practices: Terraform Best Practices for AWS users - Bill Wang ○ Splitting the Terraform monolith - Gustav Karlsson ○ How to manage Terraform state - Yevgeniy Brikman

Thank you!