Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Terraform Ops for Microservices

Terraform Ops for Microservices

BABAROT

July 19, 2018
Tweet

More Decks by BABAROT

Other Decks in Technology

Transcript

  1. Mercari Meetup for Microservices Platform
    July 19, 2018 / @b4b4r07
    Terraform Ops for Microservices

    View Slide

  2. 2
    About me
    @b4b4r07 / babarot
    Blog / tellme.tokyo
    SRE in Microservices Platform
    Team at Mercari, Inc.

    View Slide

  3. 3
    Topics
    1. Microservice Starter Kit
    2. mercari/microservices-terraform

    View Slide

  4. 4
    At first,
    In Mercari, we’re migrating our architecture from Monolithic one to
    Microservices one now...

    View Slide

  5. 5
    Monolithic App
    Data Access
    Layer
    Business Logic
    UI
    Database

    View Slide

  6. 6
    Monolithic App
    Data Access
    Layer
    Business Logic
    UI
    Database
    Microservices Apps
    Units that can be deployed isolatedly

    View Slide

  7. 7
    Monolithic App
    Data Access
    Layer
    Business Logic
    UI
    Database
    Microservices Apps
    Units that can be deployed isolatedly

    View Slide

  8. 8
    Problems of Microservices Architecture
    ● Every time a new microservice is developed, it’s need to prepare
    the infrastructure
    ○ In the case of monolithic architecture, since the code base
    for adding new functions is the same, there is no need to
    newly prepare infrastructure for deployment
    ○ On the other hand, in the case of microservices
    architecture, it is costly to prepare new infrastructure
    ○ The infra includes not only the server but also 3rd party
    tools (PagerDuty,

    View Slide

  9. 9
    Our platform: Centralized GKE Cluster
    GCP project for GKE
    Centralized cluster
    Namespace: Service A
    Namespace: Service B
    IAM: SRE IAM: Team A
    IAM: Team B
    Service A
    Service B
    RBAC: Team A
    RBAC: Team B

    View Slide

  10. 10
    Our platform: Centralized GKE Cluster
    GCP project for GKE
    Centralized cluster
    Namespace: Service A
    Namespace: Service B
    IAM: SRE IAM: Team A
    IAM: Team B
    Service A
    Service B
    RBAC: Team A
    RBAC: Team B
    New
    New
    Cost

    View Slide

  11. 11
    Cost for migrating to Microservices
    ● Microservices developers have to … by themselves
    ○ create GCP project for their service (1 Service : 1 GCP)
    ○ prepare for common prerequisites
    ■ On-call, Alert, Monitoring, …
    ○ prepare for GCP specific features (e.g. Spanner, ...)
    ○ connect the service to Centralized GKE Cluster
    ● Microservices platformer want to …
    ○ abstract these Terraform Ops and automate it
    ○ encourage Infrastructure as Code to the developers

    View Slide

  12. 12
    Isolation Policy
    ● 1 Microservice
    ○ 1 GCP Project (+ some GCP resources)
    ○ 1 Cloud Resource (PagerDuty, Sentry, …)
    ○ 1 Kubernetes Namespace (in Centralized GKE Cluster)
    ○ 1 Team (with some Roles)
    It’s hard to do these manually! = Need to be automated
    (provisioning)

    View Slide

  13. 13
    Topic
    Microservice
    Starter Kit

    View Slide

  14. 14
    ● Provisioning Tool
    ○ Provide Cloud Resources (GCP, Sentry, PagerDuty, ...)
    ○ Provide Kubernetes Resources (Namespace, Secret, ...)
    ○ Provide Team (Service owners = GitHub Teams)
    ○ ...
    ● Created as “Terraform Module” + “Terraform Template
    Provider”
    ○ Hosted on S3
    Microservice Starter Kit

    View Slide

  15. 15
    Workflow
    ● Generate Microservice skeleton from Template Provider
    ● Configure Microservice settings with Module (Starter Kit)
    ○ Fill in Team member list (= Service Owners)
    ○ Enable GCP flag?
    ○ Enable Sentry flag?
    ○ ...

    View Slide

  16. 16
    Workflow
    GCP project for GKE
    mercari-echo-us
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    Centralized GKE Cluster

    View Slide

  17. 17
    Workflow
    Namespace
    Starter Kit
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  18. 18
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  19. 19
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  20. 20
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    terraform apply
    mercari-echo-jp
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  21. 21
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  22. 22
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  23. 23
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster

    View Slide

  24. 24
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    mercari-echo-jp
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    Centralized GKE Cluster

    View Slide

  25. 25
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    mercari-echo-jp
    Centralized GKE Cluster

    View Slide

  26. 26
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster

    View Slide

  27. 27
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster

    View Slide

  28. 28
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster

    View Slide

  29. 29
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    mercari/tfnotify

    View Slide

  30. 30
    https://tech.mercari.com/entry/2018/04/09/110000

    View Slide

  31. 31
    Topic
    mercari/
    microservices-terraform

    View Slide

  32. 32
    mercari/microservices-terraform
    ● What?
    ○ All microservices infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline

    View Slide

  33. 33
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    terraform apply
    @mercari-echo-jp
    mercari/microservices-terraform
    ● What?
    ○ All microservices infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline

    View Slide

  34. 34
    mercari/microservices-terraform
    ● What?
    ○ All Microservices Infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline
    ● Why?
    ○ To centralize all Microservices Infra code
    ■ Eliminate CI pipeline setting cost
    ○ To make it easy to review P-R for platform team
    ■ Encourage the culture of Infra as Code to the developer

    View Slide

  35. 35

    View Slide

  36. 36
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/

    View Slide

  37. 37
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/
    Generated by ./script/new
    Developers can freely change or add
    Terraform resource files under their
    microservice directory

    View Slide

  38. 38
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/
    Generated by ./script/new
    Developers can freely change or add
    Terraform resource files under their
    microservice directory
    The approval and merge authority for P-R are
    defined by CODEOWNERS and master is protected
    mercari-echo-jp/
    ├── development/
    │ ├── …
    │ └── module_microservice_starter_kit.tf
    └── production/
    └── …

    View Slide

  39. 39
    mercari/microservices-terraform
    * @mercari/microservices-platform
    /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
    /terraform/microservices-platform/development/ @mercari/microservices-platform
    /terraform/microservices-platform/production/ @mercari/microservices-platform
    # mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
    $ cat .github/CODEOWNERS
    https://help.github.com/articles/about-codeowners/

    View Slide

  40. 40
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/

    View Slide

  41. 41
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/

    View Slide

  42. 42
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/

    View Slide

  43. 43
    mercari/microservices-terraform
    * @mercari/microservices-platform
    /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
    /terraform/microservices-platform/development/ @mercari/microservices-platform
    /terraform/microservices-platform/production/ @mercari/microservices-platform
    # mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
    https://help.github.com/articles/about-codeowners/
    $ cat .github/CODEOWNERS
    ☝Generated by Starter Kit

    View Slide

  44. 44
    mercari/microservices-terraform
    ● リポジトリは中央集権・分散分権モデル
    ○ 中央集権
    ■ 必要ならPlatformチームのTerraform Code Review
    ■ CI Pipelineなどのセットアップが不要
    ■ 統一的なTerraform Code管理ができる
    ○ 分散分権
    ■ 各Microservice dir以下だけApprove/Mergeを各チームに
    委譲する→Platformチームをボトルネックにしない
    ■ Terraform Stateは分離することで事故を防ぐ

    View Slide

  45. 45
    Topic
    Conclusion

    View Slide

  46. 46
    Conclusion
    ● For accelerating Microservices,
    ○ Develop Starter Kit to to make it easy to build the infra
    ○ Promote Infrastructure as Code to the developers
    ○ Improve Developer productivities

    View Slide

  47. View Slide