Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building and deploying a web application on Fa...

Building and deploying a web application on Fargate with Terraform

shonansurvivors

November 20, 2021
Tweet

More Decks by shonansurvivors

Other Decks in Technology

Transcript

  1. • Takashi Yamahara - @shonansurvivors • Site Reliability Engineer •

    AWS User Group - Japan, Beginner branch member About Me
  2. Agenda 0. Precondition 1. Separate Terraform state by: ・Component ・Environment

    ・[Appendix] Tips for Workspaces 2. Deploy ECS application with ecspresso 3. Summary
  3. Separate state by component • Consider: ◦ Stability ◦ Stateful

    or not ◦ Lifecycle . ├── app # ECR / ECS / IAM Roles(for ECS) / S3(for .env file) ├── cicd # IAM Role(for ci tool) / Data resources(for ecspresso) ├── datastore # RDS / IAM Role(for RDS) / ElastiCache ├── log # CloudWatch log groups / S3 ├── monitor # CloudWatch Alarm / SNS ├── network # VPC / Subnets / VPC Endpoints / Security Groups etc... ├── ops # IAM Role(with SAML) └── routing # Route53 / ACM / CloudFront / ALB
  4. Separate state by component Moreover, create component directories under each

    category while considering separation of concern. It helps to minimize the blast radius of failures. If you would to refer to other component attributes, use terraform_remote_state or data sources. . ├── app │ └── service1 │ ├── backend.tf │ ├── data.tf │ ├── ecr.tf │ ├── … │ ├── locals.tf │ ├── outputs.tf │ ├── provider.tf -> ../../provider.tf │ └── shared_locals.tf -> ../../shared_locals.tf ├── cicd │ └── service1 ├── datasrore │ ├── service1_elasticache │ └── service1_rds ├── log │ ├── alb │ ├── app_service1 │ ├── cloudfront │ └── datastore_service1_rds ├── network │ └── main ├── routing │ ├── example_com │ └── example_internal ├── provider.tf └── shared_locals.tf
  5. Separate state by environment There are several ways to do

    this. 1. Environment directories + Shared modules 2. Environment directories + Symbolic links 3. Only environment directories 4. Workspaces
  6. 1. Env dirs + Shared modules . └── envs ├──

    dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── main.tf # calls shared module │ │ ├── something.tf # env-specific resources │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── provider.tf │ └── shared_locals.tf ├── stg │ └── app │ └── service1 ├── prod │ └── app │ └── service1 └── shared_modules ├── app │ └── service1 │ ├── ecr.tf │ ├── ... │ ├── outputs.tf │ └── variables.tf ├── ... └── routing Create directories for each env and modularize each component. If you need environment-specific resources, put them under the environment subdirectory side.
  7. 2. Env dirs + Symbolic links Using symbolic link, refer

    to shared files from each environment. If you need environment-specific resources, put them under the environment subdirectory side. . └── envs ├── dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── main.tf -> ../../../shared/app/service1/main.tf │ │ ├── something.tf # env-specific resources │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── stg │ └── app │ └── service1 │ ├── main.tf -> ../../../shared/app/service1/main.tf │ └── ... ├── prod │ └── app │ └── service1 │ ├── main.tf -> ../../../shared/app/service1/main.tf │ └── ... └── shared ├── app │ └── service1 │ └── main.tf ├── ... └── routing
  8. 3. Only environment directories Put resources under each environment subdirectory.

    It’s not DRY. However, this structure is easy to reflect requirements of each environment. You don't have to think about shared-modules I/O or using state mv command for migtation from shared to env-specific. . └── envs ├── dev │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── stg │ ├── app │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf └── prod └── ...
  9. 4. Workspaces Using workspaces, apply same code base to multiple

    environments. . ├── app │ └── service1 │ ├── backend.tf │ ├── ecr.tf │ ├── ... │ ├── locals.tf │ ├── outputs.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf $ terraform workspace new dev $ terraform workspace select dev $ terraform plan $ terraform apply
  10. [Appendix] Tips for Workspaces 1. Configure for each environment ◦

    terraform.workspace + Local Values ◦ Variables(Input Values) + tfvars 2. Destination bucket for states 3. Wrapper script for workspaces
  11. terraform.workspace + Local Values . ├── datastore │ └── service1_rds

    │ ├── rds.tf │ ├── ... │ ├── locals.tf │ ├── backend.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf # Execute $ terraform \ workspace select dev $ terraform plan $ terraform apply resource "aws_db_instance" "this" { engine = "mysql" instance_class = local.aws_db_instance.instance_class[terraform.workspace] // } locals { aws_db_instance = { instance_class = { dev = "db.t3.small" stg = "db.m5.large" prod = "db.m5.large" } // } }
  12. Variables(Input Values) + tfvars . ├── datastore │ └── service1_rds

    │ ├── rds.tf │ ├── ... │ ├── variables.tf │ ├── dev.tfvars │ ├── stg.tfvars │ ├── prod.tfvars │ ├── backend.tf │ ├── provider.tf -> ../../provider.tf │ ├── shared_locals.tf -> ../../shared_locals.tf │ └── version.tf ├── ... ├── routing ├── provider.tf └── shared_locals.tf # dev.tfvars aws_db_instance = { instance_class = "db.t3.small" // } } # Execute $ terraform \ workspace select dev $ terraform plan \ -var-file=dev.tfvars $ terraform apply \ -var-file=dev.tfvars resource "aws_db_instance" "this" { engine = "mysql" instance_class = var.aws_db_instance.instance_class // } variable "aws_db_instance" { type = object({ instance_class = string // }) }
  13. Destination bucket for states By default, a state for each

    workspace is stored in a single S3 bucket. Default My requirement 🤔 😲
  14. Do you use variables? Variables cannot be used in backend

    block. terraform { backend "s3" { bucket = "example-${terraform.workspace}" key = "app/service1.tfstate" region = "ap-northeast-1" profile = terraform.workspace } }
  15. Use -backend-config Switch state bucket while using workspaces. $ terraform

    init -reconfigure \ -backend-config "bucket=example-dev" \ -backend-config "key=app/service1.tfstate" \ -backend-config "region=ap-northeast-1" \ -backend-config "profile=dev" $ terraform workspace select dev $ terraform plan $ terraform apply $ terraform init -reconfigure \ -backend-config=dev.conf $ terraform workspace select dev $ terraform plan $ terraform apply # dev.conf bucket = "example-dev" key = "app/service1.tfstate" region = "ap-northeast-1" profile = "dev" Or
  16. Wrapper script for Workspaces
 . ├── components │ ├── app

    │ │ └── service1 │ │ ├── backend.tf │ │ ├── data.tf │ │ ├── ecr.tf │ │ ├── ... │ │ ├── locals.tf │ │ ├── outputs.tf │ │ ├── provider.tf -> ../../provider.tf │ │ ├── shared_locals.tf -> ../../shared_locals.tf │ │ └── version.tf │ ├── ... │ ├── routing │ ├── provider.tf │ └── shared_locals.tf ├── .env.dev ├── .env.stg ├── .env.prod ├── terraform.sh └── modules # .env.dev BUCKET=example-dev REGION=ap-northeast-1 PROFILE=dev # terraform.sh(The whole is about 100 lines.) ... source ${BASE_DIR}/.env.${ENV} ... terraform init -reconfigure \ -backend-config "bucket=${BUCKET}" \ -backend-config "key=${TARGET_DIR}.tfstate" \ -backend-config "region=${REGION}" \ -backend-config "profile=${PROFILE}" ... terraform workspace select ${ENV} ... cd ${BASE_DIR}/${TARGET_DIR} terraform ${TF_CMD} ${TF_ARGS} # Usage ./terraform.sh <ENV> <TARGET_DIR> <TF_CMD> [<TF_ARGS>...] # Example ./terraform.sh dev components/app/service1 plan
  17. About ecspresso • kayac/ecspresso is a deployment tool for Amazon

    ECS. • It reads some files, then register a new task definition revision and update a ECS service.
  18. ecspresso goes well with Terraform • ecspresso can read a

    state, so we don't have to do any hard coding. # config.yaml plugins: - name: tfstate config: url: s3://bucket/key # ecs-service-def.json "networkConfiguration": { "awsvpcConfiguration": { "securityGroups": [ "{{ tfstate `aws_security_group.default.id` }}" ] "subnets": [ "{{ tfstate `aws_subnet.private['a'].id` }}", "{{ tfstate `aws_subnet.private['c'].id` }}" ] } },
  19. ecspresso can read one state However, ecspresso can read only

    one state file. Don’t separated states and ecspresso go together? . ├── app │ └── service1 │ ├── ecr.tf │ ├── iam.tf │ ├── s3.tf │ └── … ├── cicd │ └── app_service1 │ ├── … │ └── … ├── datasrore ├── log │ └── app_service1 │ ├── cloudwatch_log.tf │ └── … ├── network │ └── main │ ├── security_group.tf │ ├── subnet.tf │ └── … └── routing └── example_com ├── lb.tf └── … state state state state ecspresso 🤔
  20. Put data sources into a component Put data sources referring

    to all resources required for ecs-service-def.json and ecs-task-def.json into a component such as CI/CD. Then you make ecspresso read the state. . ├── app │ └── service1 │ ├── ecr.tf │ ├── iam.tf │ ├── s3.tf │ └── … ├── cicd │ └── app_service1 │ ├── ecspresso.tf │ └── … ├── datasrore ├── log │ └── app_service1 │ ├── cloudwatch_log.tf │ └── … ├── network │ └── main │ ├── security_group.tf │ ├── subnet.tf │ └── … └── routing └── example_com ├── lb.tf └── … state ecspresso # ecspresso.tf data "aws_xxx" "foo" { name = "foo" } ... ☺ Data sources
  21. Summary • Separate states by environment and component. ◦ Minimize

    the blast radius of failures. • There are several ways to do this, therefore think about it. ◦ Shared modules, symbolic links, Only environment directories, or workspaces ... • Use a wrapper script on an as-needed basis. ◦ Reduce the complexity of dealing with workspaces or many directories. • ecspresso — ECS deployment tool — can read a state. ◦ Put data sources associated with it into anyone of states.
  22. References • Japanese ◦ 実践Terraform AWSにおけるシステム設計とベストプラクティス - Tomoki Nomura ◦

    Terraformのコンポーネント分割について検討する - Takashi Narikawa ◦ 「それ、どこに出しても恥ずかしくない Terraformコードになってるか?」 - Yuki Yoshida ◦ Terraform ベストプラクティス 2020 春 ~moduleやめてみた~ - kenzo0107 ◦ layerx-invoice-practical-devops-20211029 - Shinji Takae ◦ Terraformなにもわからないけどディレクトリ構成の実例を晒して人類に貢献したい - Yuichiro Fukubayashi ◦ CCoE による Terraform を活用した Infrastructure as Code - Yuuki Nagahara ◦ ecspresso handbook - FUJIWARA Shunichiro • English ◦ Multi-layering: Terraform IaC from scratch to scale - Antoine Chapusot ◦ ozbillwang/terraform-best-practices: Terraform Best Practices for AWS users - Bill Wang ◦ Splitting the Terraform monolith - Gustav Karlsson ◦ How to manage Terraform state - Yevgeniy Brikman