Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Terraform Ops for Microservices

Terraform Ops for Microservices

BABAROT

July 19, 2018
Tweet

More Decks by BABAROT

Other Decks in Technology

Transcript

  1. 2 About me @b4b4r07 / babarot Blog / tellme.tokyo SRE

    in Microservices Platform Team at Mercari, Inc.
  2. 4 At first, In Mercari, we’re migrating our architecture from

    Monolithic one to Microservices one now...
  3. 6 Monolithic App Data Access Layer Business Logic UI Database

    Microservices Apps Units that can be deployed isolatedly
  4. 7 Monolithic App Data Access Layer Business Logic UI Database

    Microservices Apps Units that can be deployed isolatedly
  5. 8 Problems of Microservices Architecture • Every time a new

    microservice is developed, it’s need to prepare the infrastructure ◦ In the case of monolithic architecture, since the code base for adding new functions is the same, there is no need to newly prepare infrastructure for deployment ◦ On the other hand, in the case of microservices architecture, it is costly to prepare new infrastructure ◦ The infra includes not only the server but also 3rd party tools (PagerDuty,
  6. 9 Our platform: Centralized GKE Cluster GCP project for GKE

    Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B
  7. 10 Our platform: Centralized GKE Cluster GCP project for GKE

    Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B New New Cost
  8. 11 Cost for migrating to Microservices • Microservices developers have

    to … by themselves ◦ create GCP project for their service (1 Service : 1 GCP) ◦ prepare for common prerequisites ▪ On-call, Alert, Monitoring, … ◦ prepare for GCP specific features (e.g. Spanner, ...) ◦ connect the service to Centralized GKE Cluster • Microservices platformer want to … ◦ abstract these Terraform Ops and automate it ◦ encourage Infrastructure as Code to the developers
  9. 12 Isolation Policy • 1 Microservice ◦ 1 GCP Project

    (+ some GCP resources) ◦ 1 Cloud Resource (PagerDuty, Sentry, …) ◦ 1 Kubernetes Namespace (in Centralized GKE Cluster) ◦ 1 Team (with some Roles) It’s hard to do these manually! = Need to be automated (provisioning)
  10. 14 • Provisioning Tool ◦ Provide Cloud Resources (GCP, Sentry,

    PagerDuty, ...) ◦ Provide Kubernetes Resources (Namespace, Secret, ...) ◦ Provide Team (Service owners = GitHub Teams) ◦ ... • Created as “Terraform Module” + “Terraform Template Provider” ◦ Hosted on S3 Microservice Starter Kit
  11. 15 Workflow • Generate Microservice skeleton from Template Provider •

    Configure Microservice settings with Module (Starter Kit) ◦ Fill in Team member list (= Service Owners) ◦ Enable GCP flag? ◦ Enable Sentry flag? ◦ ...
  12. 16 Workflow GCP project for GKE mercari-echo-us Namespace 1. Run

    ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
  13. 17 Workflow Namespace Starter Kit 1. Run ./script/new locally 2.

    Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  14. 18 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  15. 19 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  16. 20 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    terraform apply mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  17. 21 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  18. 22 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  19. 23 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  20. 24 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
  21. 25 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team mercari-echo-jp Centralized GKE Cluster
  22. 26 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  23. 27 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  24. 28 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  25. 29 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster mercari/tfnotify
  26. 32 mercari/microservices-terraform • What? ◦ All microservices infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
  27. 33 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster Starter Kit mercari/microservices-terraform Circle CI terraform plan terraform apply @mercari-echo-jp mercari/microservices-terraform • What? ◦ All microservices infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
  28. 34 mercari/microservices-terraform • What? ◦ All Microservices Infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline • Why? ◦ To centralize all Microservices Infra code ▪ Eliminate CI pipeline setting cost ◦ To make it easy to review P-R for platform team ▪ Encourage the culture of Infra as Code to the developer
  29. 35

  30. 36 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/
  31. 37 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory
  32. 38 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory The approval and merge authority for P-R are defined by CODEOWNERS and master is protected mercari-echo-jp/ ├── development/ │ ├── … │ └── module_microservice_starter_kit.tf └── production/ └── …
  33. 39 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp $ cat .github/CODEOWNERS https://help.github.com/articles/about-codeowners/
  34. 43 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp https://help.github.com/articles/about-codeowners/ $ cat .github/CODEOWNERS ☝Generated by Starter Kit
  35. 44 mercari/microservices-terraform • リポジトリは中央集権・分散分権モデル ◦ 中央集権 ▪ 必要ならPlatformチームのTerraform Code Review

    ▪ CI Pipelineなどのセットアップが不要 ▪ 統一的なTerraform Code管理ができる ◦ 分散分権 ▪ 各Microservice dir以下だけApprove/Mergeを各チームに 委譲する→Platformチームをボトルネックにしない ▪ Terraform Stateは分離することで事故を防ぐ
  36. 46 Conclusion • For accelerating Microservices, ◦ Develop Starter Kit

    to to make it easy to build the infra ◦ Promote Infrastructure as Code to the developers ◦ Improve Developer productivities