Terraform Ops for Microservices

Transcript

1. Mercari Meetup for Microservices Platform July 19, 2018 / @b4b4r07

   Terraform Ops for Microservices

2. 2 About me @b4b4r07 / babarot Blog / tellme.tokyo SRE

   in Microservices Platform Team at Mercari, Inc.

3. 3 Topics 1. Microservice Starter Kit 2. mercari/microservices-terraform

4. 4 At first, In Mercari, we’re migrating our architecture from

   Monolithic one to Microservices one now...

5. 5 Monolithic App Data Access Layer Business Logic UI Database

6. 6 Monolithic App Data Access Layer Business Logic UI Database

   Microservices Apps Units that can be deployed isolatedly

7. 7 Monolithic App Data Access Layer Business Logic UI Database

   Microservices Apps Units that can be deployed isolatedly

8. 8 Problems of Microservices Architecture • Every time a new

   microservice is developed, it’s need to prepare the infrastructure ◦ In the case of monolithic architecture, since the code base for adding new functions is the same, there is no need to newly prepare infrastructure for deployment ◦ On the other hand, in the case of microservices architecture, it is costly to prepare new infrastructure ◦ The infra includes not only the server but also 3rd party tools (PagerDuty,

9. 9 Our platform: Centralized GKE Cluster GCP project for GKE

   Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B

10. 10 Our platform: Centralized GKE Cluster GCP project for GKE

    Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B New New Cost

11. 11 Cost for migrating to Microservices • Microservices developers have

    to … by themselves ◦ create GCP project for their service (1 Service : 1 GCP) ◦ prepare for common prerequisites ▪ On-call, Alert, Monitoring, … ◦ prepare for GCP specific features (e.g. Spanner, ...) ◦ connect the service to Centralized GKE Cluster • Microservices platformer want to … ◦ abstract these Terraform Ops and automate it ◦ encourage Infrastructure as Code to the developers

12. 12 Isolation Policy • 1 Microservice ◦ 1 GCP Project

    (+ some GCP resources) ◦ 1 Cloud Resource (PagerDuty, Sentry, …) ◦ 1 Kubernetes Namespace (in Centralized GKE Cluster) ◦ 1 Team (with some Roles) It’s hard to do these manually! = Need to be automated (provisioning)

13. 13 Topic Microservice Starter Kit

14. 14 • Provisioning Tool ◦ Provide Cloud Resources (GCP, Sentry,

    PagerDuty, ...) ◦ Provide Kubernetes Resources (Namespace, Secret, ...) ◦ Provide Team (Service owners = GitHub Teams) ◦ ... • Created as “Terraform Module” + “Terraform Template Provider” ◦ Hosted on S3 Microservice Starter Kit

15. 15 Workflow • Generate Microservice skeleton from Template Provider •

    Configure Microservice settings with Module (Starter Kit) ◦ Fill in Team member list (= Service Owners) ◦ Enable GCP flag? ◦ Enable Sentry flag? ◦ ...

16. 16 Workflow GCP project for GKE mercari-echo-us Namespace 1. Run

    ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster

17. 17 Workflow Namespace Starter Kit 1. Run ./script/new locally 2.

    Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

18. 18 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

19. 19 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

20. 20 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    terraform apply mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

21. 21 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

22. 22 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

23. 23 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan

    mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster

24. 24 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster

25. 25 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team mercari-echo-jp Centralized GKE Cluster

26. 26 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster

27. 27 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster

28. 28 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster

29. 29 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster mercari/tfnotify

30. 30 https://tech.mercari.com/entry/2018/04/09/110000

31. 31 Topic mercari/ microservices-terraform

32. 32 mercari/microservices-terraform • What? ◦ All microservices infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline

33. 33 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit

    mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster Starter Kit mercari/microservices-terraform Circle CI terraform plan terraform apply @mercari-echo-jp mercari/microservices-terraform • What? ◦ All microservices infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline

34. 34 mercari/microservices-terraform • What? ◦ All Microservices Infra are managed

    by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline • Why? ◦ To centralize all Microservices Infra code ▪ Eliminate CI pipeline setting cost ◦ To make it easy to review P-R for platform team ▪ Encourage the culture of Infra as Code to the developer

35. 35

36. 36 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/

37. 37 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory

38. 38 mercari/microservices-terraform . ├── script/ │ ├── … │ └──

    new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory The approval and merge authority for P-R are defined by CODEOWNERS and master is protected mercari-echo-jp/ ├── development/ │ ├── … │ └── module_microservice_starter_kit.tf └── production/ └── …

39. 39 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp $ cat .github/CODEOWNERS https://help.github.com/articles/about-codeowners/

40. 40 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

41. 41 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

42. 42 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/

43. 43 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform

    # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp https://help.github.com/articles/about-codeowners/ $ cat .github/CODEOWNERS ☝Generated by Starter Kit

44. 44 mercari/microservices-terraform • リポジトリは中央集権・分散分権モデル ◦ 中央集権 ▪ 必要ならPlatformチームのTerraform Code Review

    ▪ CI Pipelineなどのセットアップが不要 ▪ 統一的なTerraform Code管理ができる ◦ 分散分権 ▪ 各Microservice dir以下だけApprove/Mergeを各チームに 委譲する→Platformチームをボトルネックにしない ▪ Terraform Stateは分離することで事故を防ぐ

45. 45 Topic Conclusion

46. 46 Conclusion • For accelerating Microservices, ◦ Develop Starter Kit

    to to make it easy to build the infra ◦ Promote Infrastructure as Code to the developers ◦ Improve Developer productivities
47. None