Terraform Ops for Microservices

8238c3c0be55b887aa9d6d59bfefa504?s=47 BABAROT
July 19, 2018

Terraform Ops for Microservices

8238c3c0be55b887aa9d6d59bfefa504?s=128

BABAROT

July 19, 2018
Tweet

Transcript

  1. Mercari Meetup for Microservices Platform July 19, 2018 / @b4b4r07

    Terraform Ops for Microservices
  2. 2 About me @b4b4r07 / babarot Blog / tellme.tokyo SRE

    in Microservices Platform Team at Mercari, Inc.
  3. 3 Topics 1. Microservice Starter Kit 2. mercari/microservices-terraform

  4. 4 At first, In Mercari, we’re migrating our architecture from

    Monolithic one to Microservices one now...
  5. 5 Monolithic App Data Access Layer Business Logic UI Database

  6. 6 Monolithic App Data Access Layer Business Logic UI Database

    Microservices Apps Units that can be deployed isolatedly
  7. 7 Monolithic App Data Access Layer Business Logic UI Database

    Microservices Apps Units that can be deployed isolatedly
  8. 8 Problems of Microservices Architecture • Every time a new

    microservice is developed, it’s need to prepare the infrastructure ◦ In the case of monolithic architecture, since the code base for adding new functions is the same, there is no need to newly prepare infrastructure for deployment ◦ On the other hand, in the case of microservices architecture, it is costly to prepare new infrastructure ◦ The infra includes not only the server but also 3rd party tools (PagerDuty,
  9. 9 Our platform: Centralized GKE Cluster GCP project for GKE

    Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B
  10. 10 Our platform: Centralized GKE Cluster GCP project for GKE

    Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B New New Cost
  11. 11 Cost for migrating to Microservices • Microservices developers have

    to … by themselves ◦ create GCP project for their service (1 Service : 1 GCP) ◦ prepare for common prerequisites ▪ On-call, Alert, Monitoring, … ◦ prepare for GCP specific features (e.g. Spanner, ...) ◦ connect the service to Centralized GKE Cluster • Microservices platformer want to … ◦ abstract these Terraform Ops and automate it ◦ encourage Infrastructure as Code to the developers
  12. 12 Isolation Policy • 1 Microservice ◦ 1 GCP Project

    (+ some GCP resources) ◦ 1 Cloud Resource (PagerDuty, Sentry, …) ◦ 1 Kubernetes Namespace (in Centralized GKE Cluster) ◦ 1 Team (with some Roles) It’s hard to do these manually! = Need to be automated (provisioning)
  13. 13 Topic Microservice Starter Kit

  14. 14 • Provisioning Tool ◦ Provide Cloud Resources (GCP, Sentry,

    PagerDuty, ...) ◦ Provide Kubernetes Resources (Namespace, Secret, ...) ◦ Provide Team (Service owners = GitHub Teams) ◦ ... • Created as “Terraform Module” + “Terraform Template Provider” ◦ Hosted on S3 Microservice Starter Kit
  15. 15 Workflow • Generate Microservice skeleton from Template Provider •

    Configure Microservice settings with Module (Starter Kit) ◦ Fill in Team member list (= Service Owners) ◦ Enable GCP flag? ◦ Enable Sentry flag? ◦ ...
  16. 16 Workflow GCP project for GKE mercari-echo-us Namespace 1. Run

    ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
  17. 17 Workflow Namespace Starter Kit 1. Run ./script/new locally 2.

    Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  18. 18 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  19. 19 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  20. 20 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    terraform apply mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  21. 21 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  22. 22 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  23. 23 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
  24. 24 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
  25. 25 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team mercari-echo-jp Centralized GKE Cluster
  26. 26 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  27. 27 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  28. 28 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
  29. 29 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster mercari/tfnotify
  30. 30 https://tech.mercari.com/entry/2018/04/09/110000

  31. 31 Topic mercari/ microservices-terraform

  32. 32 mercari/microservices-terraform • What? ◦ All microservices infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
  33. 33 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster Starter Kit mercari/microservices-terraform Circle CI terraform plan terraform apply @mercari-echo-jp mercari/microservices-terraform • What? ◦ All microservices infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
  34. 34 mercari/microservices-terraform • What? ◦ All Microservices Infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline • Why? ◦ To centralize all Microservices Infra code ▪ Eliminate CI pipeline setting cost ◦ To make it easy to review P-R for platform team ▪ Encourage the culture of Infra as Code to the developer
  35. 35

  36. 36 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/
  37. 37 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory
  38. 38 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory The approval and merge authority for P-R are defined by CODEOWNERS and master is protected mercari-echo-jp/ ├── development/ │ ├── … │ └── module_microservice_starter_kit.tf └── production/ └── …
  39. 39 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp $ cat .github/CODEOWNERS https://help.github.com/articles/about-codeowners/
  40. 40 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

  41. 41 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

  42. 42 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

  43. 43 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp https://help.github.com/articles/about-codeowners/ $ cat .github/CODEOWNERS ☝Generated by Starter Kit
  44. 44 mercari/microservices-terraform • リポジトリは中央集権・分散分権モデル ◦ 中央集権 ▪ 必要ならPlatformチームのTerraform Code Review

    ▪ CI Pipelineなどのセットアップが不要 ▪ 統一的なTerraform Code管理ができる ◦ 分散分権 ▪ 各Microservice dir以下だけApprove/Mergeを各チームに 委譲する→Platformチームをボトルネックにしない ▪ Terraform Stateは分離することで事故を防ぐ
  45. 45 Topic Conclusion

  46. 46 Conclusion • For accelerating Microservices, ◦ Develop Starter Kit

    to to make it easy to build the infra ◦ Promote Infrastructure as Code to the developers ◦ Improve Developer productivities
  47. None