Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building and deploying a web application on Fargate with Terraform

Building and deploying a web application on Fargate with Terraform

shonansurvivors

November 20, 2021
Tweet

More Decks by shonansurvivors

Other Decks in Technology

Transcript

  1. Building and deploying
    a web application on Fargate
    with Terraform
    Takashi Yamahara
    @shonansurvivors

    View full-size slide

  2. ● Takashi Yamahara - @shonansurvivors
    ● Site Reliability Engineer
    ● AWS User Group - Japan, Beginner branch member
    About Me

    View full-size slide

  3. Agenda
    0. Precondition
    1. Separate Terraform state by:
    ・Component
    ・Environment
    ・[Appendix] Tips for Workspaces
    2. Deploy ECS application with ecspresso
    3. Summary

    View full-size slide

  4. 0. Precondition

    View full-size slide

  5. Repositories
    Today’s session
    Note:
    This is not to say that the
    above case is superior.

    View full-size slide

  6. Out of scope
    ● Terraform Cloud
    ● Terragrunt

    View full-size slide

  7. 1. Separate Terraform state

    View full-size slide

  8. Separate Terraform state
    Monolith Separated

    View full-size slide

  9. Separate state by component

    View full-size slide

  10. Separate state by component
    ● Consider:
    ○ Stability
    ○ Stateful or not
    ○ Lifecycle
    .
    ├── app # ECR / ECS / IAM Roles(for ECS) / S3(for .env file)
    ├── cicd # IAM Role(for ci tool) / Data resources(for ecspresso)
    ├── datastore # RDS / IAM Role(for RDS) / ElastiCache
    ├── log # CloudWatch log groups / S3
    ├── monitor # CloudWatch Alarm / SNS
    ├── network # VPC / Subnets / VPC Endpoints / Security Groups etc...
    ├── ops # IAM Role(with SAML)
    └── routing # Route53 / ACM / CloudFront / ALB

    View full-size slide

  11. Separate state by component
    Moreover, create component directories
    under each category while considering
    separation of concern.
    It helps to minimize the blast radius of
    failures.
    If you would to refer to other
    component attributes, use
    terraform_remote_state or data
    sources.
    .
    ├── app
    │ └── service1
    │ ├── backend.tf
    │ ├── data.tf
    │ ├── ecr.tf
    │ ├── …
    │ ├── locals.tf
    │ ├── outputs.tf
    │ ├── provider.tf -> ../../provider.tf
    │ └── shared_locals.tf -> ../../shared_locals.tf
    ├── cicd
    │ └── service1
    ├── datasrore
    │ ├── service1_elasticache
    │ └── service1_rds
    ├── log
    │ ├── alb
    │ ├── app_service1
    │ ├── cloudfront
    │ └── datastore_service1_rds
    ├── network
    │ └── main
    ├── routing
    │ ├── example_com
    │ └── example_internal
    ├── provider.tf
    └── shared_locals.tf

    View full-size slide

  12. Separate state by environment

    View full-size slide

  13. Separate state by environment
    There are several ways to do this.
    1. Environment directories + Shared modules
    2. Environment directories + Symbolic links
    3. Only environment directories
    4. Workspaces

    View full-size slide

  14. 1. Env dirs + Shared modules
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── main.tf # calls shared module
    │ │ ├── something.tf # env-specific resources
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ └── app
    │ └── service1
    ├── prod
    │ └── app
    │ └── service1
    └── shared_modules
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── ...
    │ ├── outputs.tf
    │ └── variables.tf
    ├── ...
    └── routing
    Create directories for each env and
    modularize each component.
    If you need environment-specific
    resources, put them under the
    environment subdirectory side.

    View full-size slide

  15. 2. Env dirs + Symbolic links
    Using symbolic link, refer to shared
    files from each environment.
    If you need environment-specific
    resources, put them under the
    environment subdirectory side.
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ │ ├── something.tf # env-specific resources
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ └── app
    │ └── service1
    │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ └── ...
    ├── prod
    │ └── app
    │ └── service1
    │ ├── main.tf -> ../../../shared/app/service1/main.tf
    │ └── ...
    └── shared
    ├── app
    │ └── service1
    │ └── main.tf
    ├── ...
    └── routing

    View full-size slide

  16. 3. Only environment directories
    Put resources under each environment
    subdirectory.
    It’s not DRY.
    However, this structure is easy to reflect
    requirements of each environment.
    You don't have to think about
    shared-modules I/O or using state mv
    command for migtation from shared to
    env-specific.
    .
    └── envs
    ├── dev
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── stg
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    └── prod
    └── ...

    View full-size slide

  17. 4. Workspaces
    Using workspaces, apply
    same code base to multiple
    environments.
    .
    ├── app
    │ └── service1
    │ ├── backend.tf
    │ ├── ecr.tf
    │ ├── ...
    │ ├── locals.tf
    │ ├── outputs.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    $ terraform workspace new dev
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply

    View full-size slide

  18. [Appendix] Tips for Workspaces

    View full-size slide

  19. [Appendix] Tips for Workspaces
    1. Configure for each environment
    ○ terraform.workspace + Local Values
    ○ Variables(Input Values) + tfvars
    2. Destination bucket for states
    3. Wrapper script for workspaces

    View full-size slide

  20. terraform.workspace + Local Values
    .
    ├── datastore
    │ └── service1_rds
    │ ├── rds.tf
    │ ├── ...
    │ ├── locals.tf
    │ ├── backend.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    # Execute
    $ terraform \
    workspace select dev
    $ terraform plan
    $ terraform apply
    resource "aws_db_instance" "this" {
    engine = "mysql"
    instance_class = local.aws_db_instance.instance_class[terraform.workspace]
    //
    }
    locals {
    aws_db_instance = {
    instance_class = {
    dev = "db.t3.small"
    stg = "db.m5.large"
    prod = "db.m5.large"
    }
    //
    }
    }

    View full-size slide

  21. Variables(Input Values) + tfvars
    .
    ├── datastore
    │ └── service1_rds
    │ ├── rds.tf
    │ ├── ...
    │ ├── variables.tf
    │ ├── dev.tfvars
    │ ├── stg.tfvars
    │ ├── prod.tfvars
    │ ├── backend.tf
    │ ├── provider.tf -> ../../provider.tf
    │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ └── version.tf
    ├── ...
    ├── routing
    ├── provider.tf
    └── shared_locals.tf
    # dev.tfvars
    aws_db_instance = {
    instance_class = "db.t3.small"
    //
    }
    }
    # Execute
    $ terraform \
    workspace select dev
    $ terraform plan \
    -var-file=dev.tfvars
    $ terraform apply \
    -var-file=dev.tfvars
    resource "aws_db_instance" "this" {
    engine = "mysql"
    instance_class = var.aws_db_instance.instance_class
    //
    }
    variable "aws_db_instance" {
    type = object({
    instance_class = string
    //
    })
    }

    View full-size slide

  22. Destination bucket for states
    By default, a state for each workspace is stored in a single S3 bucket.
    Default
    My requirement
    🤔
    😲

    View full-size slide

  23. Do you use variables?
    Variables cannot be used in backend block.
    terraform {
    backend "s3" {
    bucket = "example-${terraform.workspace}"
    key = "app/service1.tfstate"
    region = "ap-northeast-1"
    profile = terraform.workspace
    }
    }

    View full-size slide

  24. Use -backend-config
    Switch state bucket while using workspaces.
    $ terraform init -reconfigure \
    -backend-config "bucket=example-dev" \
    -backend-config "key=app/service1.tfstate" \
    -backend-config "region=ap-northeast-1" \
    -backend-config "profile=dev"
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply
    $ terraform init -reconfigure \
    -backend-config=dev.conf
    $ terraform workspace select dev
    $ terraform plan
    $ terraform apply
    # dev.conf
    bucket = "example-dev"
    key = "app/service1.tfstate"
    region = "ap-northeast-1"
    profile = "dev"
    Or

    View full-size slide

  25. workspace select ?
    init -backend-config=?
    (If you used tfvars)
    apply -var-file=?.tfvars

    View full-size slide

  26. Wrapper script for Workspaces

    .
    ├── components
    │ ├── app
    │ │ └── service1
    │ │ ├── backend.tf
    │ │ ├── data.tf
    │ │ ├── ecr.tf
    │ │ ├── ...
    │ │ ├── locals.tf
    │ │ ├── outputs.tf
    │ │ ├── provider.tf -> ../../provider.tf
    │ │ ├── shared_locals.tf -> ../../shared_locals.tf
    │ │ └── version.tf
    │ ├── ...
    │ ├── routing
    │ ├── provider.tf
    │ └── shared_locals.tf
    ├── .env.dev
    ├── .env.stg
    ├── .env.prod
    ├── terraform.sh
    └── modules
    # .env.dev
    BUCKET=example-dev
    REGION=ap-northeast-1
    PROFILE=dev
    # terraform.sh(The whole is about 100 lines.)
    ...
    source ${BASE_DIR}/.env.${ENV}
    ...
    terraform init -reconfigure \
    -backend-config "bucket=${BUCKET}" \
    -backend-config "key=${TARGET_DIR}.tfstate" \
    -backend-config "region=${REGION}" \
    -backend-config "profile=${PROFILE}"
    ...
    terraform workspace select ${ENV}
    ...
    cd ${BASE_DIR}/${TARGET_DIR}
    terraform ${TF_CMD} ${TF_ARGS}
    # Usage
    ./terraform.sh [...]
    # Example
    ./terraform.sh dev components/app/service1 plan

    View full-size slide

  27. 2. Deploy ECS application with ecspresso

    View full-size slide

  28. About ecspresso
    ● kayac/ecspresso is a deployment tool for Amazon ECS.
    ● It reads some files, then register a new task definition revision and
    update a ECS service.

    View full-size slide

  29. ecspresso goes well with Terraform
    ● ecspresso can read a state, so we don't have to do any hard coding.
    # config.yaml
    plugins:
    - name: tfstate
    config:
    url: s3://bucket/key
    # ecs-service-def.json
    "networkConfiguration": {
    "awsvpcConfiguration": {
    "securityGroups": [
    "{{ tfstate `aws_security_group.default.id` }}"
    ]
    "subnets": [
    "{{ tfstate `aws_subnet.private['a'].id` }}",
    "{{ tfstate `aws_subnet.private['c'].id` }}"
    ]
    }
    },

    View full-size slide

  30. ecspresso can read one state
    However, ecspresso can read
    only one state file.
    Don’t separated states and
    ecspresso go together?
    .
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── iam.tf
    │ ├── s3.tf
    │ └── …
    ├── cicd
    │ └── app_service1
    │ ├── …
    │ └── …
    ├── datasrore
    ├── log
    │ └── app_service1
    │ ├── cloudwatch_log.tf
    │ └── …
    ├── network
    │ └── main
    │ ├── security_group.tf
    │ ├── subnet.tf
    │ └── …
    └── routing
    └── example_com
    ├── lb.tf
    └── …
    state
    state
    state
    state
    ecspresso
    🤔

    View full-size slide

  31. Put data sources into a component
    Put data sources referring to all resources
    required for ecs-service-def.json and
    ecs-task-def.json into a component such as
    CI/CD.
    Then you make ecspresso read the state.
    .
    ├── app
    │ └── service1
    │ ├── ecr.tf
    │ ├── iam.tf
    │ ├── s3.tf
    │ └── …
    ├── cicd
    │ └── app_service1
    │ ├── ecspresso.tf
    │ └── …
    ├── datasrore
    ├── log
    │ └── app_service1
    │ ├── cloudwatch_log.tf
    │ └── …
    ├── network
    │ └── main
    │ ├── security_group.tf
    │ ├── subnet.tf
    │ └── …
    └── routing
    └── example_com
    ├── lb.tf
    └── …
    state
    ecspresso
    # ecspresso.tf
    data "aws_xxx" "foo" {
    name = "foo"
    }
    ...

    Data sources

    View full-size slide

  32. Summary
    ● Separate states by environment and component.
    ○ Minimize the blast radius of failures.
    ● There are several ways to do this, therefore think about it.
    ○ Shared modules, symbolic links, Only environment directories,
    or workspaces ...
    ● Use a wrapper script on an as-needed basis.
    ○ Reduce the complexity of dealing with workspaces or many
    directories.
    ● ecspresso — ECS deployment tool — can read a state.
    ○ Put data sources associated with it into anyone of states.

    View full-size slide

  33. References
    ● Japanese
    ○ 実践Terraform AWSにおけるシステム設計とベストプラクティス - Tomoki Nomura
    ○ Terraformのコンポーネント分割について検討する - Takashi Narikawa
    ○ 「それ、どこに出しても恥ずかしくない Terraformコードになってるか?」 - Yuki Yoshida
    ○ Terraform ベストプラクティス 2020 春 ~moduleやめてみた~ - kenzo0107
    ○ layerx-invoice-practical-devops-20211029 - Shinji Takae
    ○ Terraformなにもわからないけどディレクトリ構成の実例を晒して人類に貢献したい - Yuichiro Fukubayashi
    ○ CCoE による Terraform を活用した Infrastructure as Code - Yuuki Nagahara
    ○ ecspresso handbook - FUJIWARA Shunichiro
    ● English
    ○ Multi-layering: Terraform IaC from scratch to scale - Antoine Chapusot
    ○ ozbillwang/terraform-best-practices: Terraform Best Practices for AWS users - Bill Wang
    ○ Splitting the Terraform monolith - Gustav Karlsson
    ○ How to manage Terraform state - Yevgeniy Brikman

    View full-size slide