Building and deploying a web application on Fargate with Terraform

JAWS PANKRATION 2021

https://jawspankration2021.jaws-ug.jp/sessions/11

Transcript

1. Building and deploying
   a web application on Fargate
   with Terraform
   Takashi Yamahara
   @shonansurvivors
   

   View Slide

2. ● Takashi Yamahara - @shonansurvivors
   ● Site Reliability Engineer
   ● AWS User Group - Japan, Beginner branch member
   About Me
   

   View Slide

3. Agenda
   0. Precondition
   1. Separate Terraform state by:
   ・Component
   ・Environment
   ・[Appendix] Tips for Workspaces
   2. Deploy ECS application with ecspresso
   3. Summary
   

   View Slide

4. 0. Precondition
   

   View Slide

5. Repositories
   Today’s session
   Note:
   This is not to say that the
   above case is superior.
   

   View Slide

6. Out of scope
   ● Terraform Cloud
   ● Terragrunt
   

   View Slide

7. 1. Separate Terraform state
   

   View Slide

8. Separate Terraform state
   Monolith Separated
   

   View Slide

9. Separate state by component
   

   View Slide

10. Separate state by component
    ● Consider:
    ○ Stability
    ○ Stateful or not
    ○ Lifecycle
    .
    ├── app # ECR / ECS / IAM Roles(for ECS) / S3(for .env file)
    ├── cicd # IAM Role(for ci tool) / Data resources(for ecspresso)
    ├── datastore # RDS / IAM Role(for RDS) / ElastiCache
    ├── log # CloudWatch log groups / S3
    ├── monitor # CloudWatch Alarm / SNS
    ├── network # VPC / Subnets / VPC Endpoints / Security Groups etc...
    ├── ops # IAM Role(with SAML)
    └── routing # Route53 / ACM / CloudFront / ALB
    

    View Slide

11. Separate state by component
    Moreover, create component directories
    under each category while considering
    separation of concern.
    It helps to minimize the blast radius of
    failures.
    If you would to refer to other
    component attributes, use
    terraform_remote_state or data
    sources.
    .
    ├── app
    │ └── service1
    │ ├── backend.tf
    │ ├── data.tf
    │ ├── ecr.tf
    │ ├── …
    │ ├── locals.tf
    │ ├── outputs.tf
    │ ├── provider.tf -> ../../provider.tf
    │ └── shared_locals.tf -> ../../shared_locals.tf
    ├── cicd
    │ └── service1
    ├── datasrore
    │ ├── service1_elasticache
    │ └── service1_rds
    ├── log
    │ ├── alb
    │ ├── app_service1
    │ ├── cloudfront
    │ └── datastore_service1_rds
    ├── network
    │ └── main
    ├── routing
    │ ├── example_com
    │ └── example_internal
    ├── provider.tf
    └── shared_locals.tf
    

    View Slide

12. Separate state by environment
    

    View Slide

13. Separate state by environment
    There are several ways to do this.
    1. Environment directories + Shared modules
    2. Environment directories + Symbolic links
    3. Only environment directories
    4. Workspaces
    

    View Slide

14. 1. Env dirs + Shared modules
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── main.tf # calls shared module
    │ │ ├── something.tf # env-specific resources
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ └── app
    │ └── service1
    ├── prod
    │ └── app
    │ └── service1
    └── shared_modules
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── ...
    │ ├── outputs.tf
    │ └── variables.tf
    ├── ...
    └── routing
    Create directories for each env and
    modularize each component.
    If you need environment-specific
    resources, put them under the
    environment subdirectory side.
    

    View Slide

15. 2. Env dirs + Symbolic links
    Using symbolic link, refer to shared
    files from each environment.
    If you need environment-specific
    resources, put them under the
    environment subdirectory side.
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ │ ├── something.tf # env-specific resources
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ └── app
    │ └── service1
    │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ └── ...
    ├── prod
    │ └── app
    │ └── service1
    │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ └── ...
    └── shared
    ├── app
    │ └── service1
    │ └── main.tf
    ├── ...
    └── routing
    

    View Slide

16. 3. Only environment directories
    Put resources under each environment
    subdirectory.
    It’s not DRY.
    However, this structure is easy to reflect
    requirements of each environment.
    You don't have to think about
    shared-modules I/O or using state mv
    command for migtation from shared to
    env-specific.
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    └── prod
    └── ...
    

    View Slide

17. 4. Workspaces
    Using workspaces, apply
    same code base to multiple
    environments.
    .
    ├── app
    │ └── service1
    │ ├── backend.tf
    │ ├── ecr.tf
    │ ├── ...
    │ ├── locals.tf
    │ ├── outputs.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    $ terraform workspace new dev
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply
    

    View Slide

18. [Appendix] Tips for Workspaces
    

    View Slide

19. [Appendix] Tips for Workspaces
    1. Configure for each environment
    ○ terraform.workspace + Local Values
    ○ Variables(Input Values) + tfvars
    2. Destination bucket for states
    3. Wrapper script for workspaces
    

    View Slide

20. terraform.workspace + Local Values
    .
    ├── datastore
    │ └── service1_rds
    │ ├── rds.tf
    │ ├── ...
    │ ├── locals.tf
    │ ├── backend.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    # Execute
    $ terraform \
    workspace select dev
    $ terraform plan
    $ terraform apply
    resource "aws_db_instance" "this" {
    engine = "mysql"
    instance_class = local.aws_db_instance.instance_class[terraform.workspace]
    //
    }
    locals {
    aws_db_instance = {
    instance_class = {
    dev = "db.t3.small"
    stg = "db.m5.large"
    prod = "db.m5.large"
    }
    //
    }
    }
    

    View Slide

21. Variables(Input Values) + tfvars
    .
    ├── datastore
    │ └── service1_rds
    │ ├── rds.tf
    │ ├── ...
    │ ├── variables.tf
    │ ├── dev.tfvars
    │ ├── stg.tfvars
    │ ├── prod.tfvars
    │ ├── backend.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    # dev.tfvars
    aws_db_instance = {
    instance_class = "db.t3.small"
    //
    }
    }
    # Execute
    $ terraform \
    workspace select dev
    $ terraform plan \
    -var-file=dev.tfvars
    $ terraform apply \
    -var-file=dev.tfvars
    resource "aws_db_instance" "this" {
    engine = "mysql"
    instance_class = var.aws_db_instance.instance_class
    //
    }
    variable "aws_db_instance" {
    type = object({
    instance_class = string
    //
    })
    }
    

    View Slide

22. Destination bucket for states
    By default, a state for each workspace is stored in a single S3 bucket.
    Default
    My requirement
    🤔
    😲
    

    View Slide

23. Do you use variables?
    Variables cannot be used in backend block.
    terraform {
    backend "s3" {
    bucket = "example-${terraform.workspace}"
    key = "app/service1.tfstate"
    region = "ap-northeast-1"
    profile = terraform.workspace
    }
    }
    

    View Slide

24. Use -backend-config
    Switch state bucket while using workspaces.
    $ terraform init -reconfigure \
    -backend-config "bucket=example-dev" \
    -backend-config "key=app/service1.tfstate" \
    -backend-config "region=ap-northeast-1" \
    -backend-config "profile=dev"
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply
    $ terraform init -reconfigure \
    -backend-config=dev.conf
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply
    # dev.conf
    bucket = "example-dev"
    key = "app/service1.tfstate"
    region = "ap-northeast-1"
    profile = "dev"
    Or
    

    View Slide

25. workspace select ?
    init -backend-config=?
    (If you used tfvars)
    apply -var-file=?.tfvars
    

    View Slide

26. Wrapper script for Workspaces

    .
    ├── components
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── .env.dev
    ├── .env.stg
    ├── .env.prod
    ├── terraform.sh
    └── modules
    # .env.dev
    BUCKET=example-dev
    REGION=ap-northeast-1
    PROFILE=dev
    # terraform.sh(The whole is about 100 lines.)
    ...
    source ${BASE_DIR}/.env.${ENV}
    ...
    terraform init -reconfigure \
    -backend-config "bucket=${BUCKET}" \
    -backend-config "key=${TARGET_DIR}.tfstate" \
    -backend-config "region=${REGION}" \
    -backend-config "profile=${PROFILE}"
    ...
    terraform workspace select ${ENV}
    ...
    cd ${BASE_DIR}/${TARGET_DIR}
    terraform ${TF_CMD} ${TF_ARGS}
    # Usage
    ./terraform.sh [...]
    # Example
    ./terraform.sh dev components/app/service1 plan
    

    View Slide

27. 2. Deploy ECS application with ecspresso
    

    View Slide

28. About ecspresso
    ● kayac/ecspresso is a deployment tool for Amazon ECS.
    ● It reads some files, then register a new task definition revision and
    update a ECS service.
    

    View Slide

29. ecspresso goes well with Terraform
    ● ecspresso can read a state, so we don't have to do any hard coding.
    # config.yaml
    plugins:
    - name: tfstate
    config:
    url: s3://bucket/key
    # ecs-service-def.json
    "networkConfiguration": {
    "awsvpcConfiguration": {
    "securityGroups": [
    "{{ tfstate `aws_security_group.default.id` }}"
    ]
    "subnets": [
    "{{ tfstate `aws_subnet.private['a'].id` }}",
    "{{ tfstate `aws_subnet.private['c'].id` }}"
    ]
    }
    },
    

    View Slide

30. ecspresso can read one state
    However, ecspresso can read
    only one state file.
    Don’t separated states and
    ecspresso go together?
    .
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── iam.tf
    │ ├── s3.tf
    │ └── …
    ├── cicd
    │ └── app_service1
    │ ├── …
    │ └── …
    ├── datasrore
    ├── log
    │ └── app_service1
    │ ├── cloudwatch_log.tf
    │ └── …
    ├── network
    │ └── main
    │ ├── security_group.tf
    │ ├── subnet.tf
    │ └── …
    └── routing
    └── example_com
    ├── lb.tf
    └── …
    state
    state
    state
    state
    ecspresso
    🤔
    

    View Slide

31. Put data sources into a component
    Put data sources referring to all resources
    required for ecs-service-def.json and
    ecs-task-def.json into a component such as
    CI/CD.
    Then you make ecspresso read the state.
    .
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── iam.tf
    │ ├── s3.tf
    │ └── …
    ├── cicd
    │ └── app_service1
    │ ├── ecspresso.tf
    │ └── …
    ├── datasrore
    ├── log
    │ └── app_service1
    │ ├── cloudwatch_log.tf
    │ └── …
    ├── network
    │ └── main
    │ ├── security_group.tf
    │ ├── subnet.tf
    │ └── …
    └── routing
    └── example_com
    ├── lb.tf
    └── …
    state
    ecspresso
    # ecspresso.tf
    data "aws_xxx" "foo" {
    name = "foo"
    }
    ...
    ☺
    Data sources
    

    View Slide

32. Summary
    

    View Slide

33. Summary
    ● Separate states by environment and component.
    ○ Minimize the blast radius of failures.
    ● There are several ways to do this, therefore think about it.
    ○ Shared modules, symbolic links, Only environment directories,
    or workspaces ...
    ● Use a wrapper script on an as-needed basis.
    ○ Reduce the complexity of dealing with workspaces or many
    directories.
    ● ecspresso — ECS deployment tool — can read a state.
    ○ Put data sources associated with it into anyone of states.
    

    View Slide

34. References
    ● Japanese
    ○ 実践Terraform AWSにおけるシステム設計とベストプラクティス - Tomoki Nomura
    ○ Terraformのコンポーネント分割について検討する - Takashi Narikawa
    ○ 「それ、どこに出しても恥ずかしくない Terraformコードになってるか？」 - Yuki Yoshida
    ○ Terraform ベストプラクティス 2020 春 ~moduleやめてみた~ - kenzo0107
    ○ layerx-invoice-practical-devops-20211029 - Shinji Takae
    ○ Terraformなにもわからないけどディレクトリ構成の実例を晒して人類に貢献したい - Yuichiro Fukubayashi
    ○ CCoE による Terraform を活用した Infrastructure as Code - Yuuki Nagahara
    ○ ecspresso handbook - FUJIWARA Shunichiro
    ● English
    ○ Multi-layering: Terraform IaC from scratch to scale - Antoine Chapusot
    ○ ozbillwang/terraform-best-practices: Terraform Best Practices for AWS users - Bill Wang
    ○ Splitting the Terraform monolith - Gustav Karlsson
    ○ How to manage Terraform state - Yevgeniy Brikman
    

    View Slide

35. Thank you!
    

    View Slide