Terraform Ops for Microservices

Talk at https://mercari.connpass.com/event/92168/

Transcript

1. Mercari Meetup for Microservices Platform July 19, 2018 / @b4b4r07 Terraform Ops for Microservices
2. 2 About me @b4b4r07 / babarot Blog / tellme.tokyo SRE in Microservices Platform Team at Mercari, Inc.
3. 3 Topics 1. Microservice Starter Kit 2. mercari/microservices-terraform
4. 4 At first, In Mercari, we’re migrating our architecture from Monolithic one to Microservices one now...
5. 5 Monolithic App Data Access Layer Business Logic UI Database
6. 6 Monolithic App Data Access Layer Business Logic UI Database Microservices Apps Units that can be deployed isolatedly
7. 7 Monolithic App Data Access Layer Business Logic UI Database Microservices Apps Units that can be deployed isolatedly
8. 8 Problems of Microservices Architecture • Every time a new microservice is developed, it’s need to prepare the infrastructure ◦ In the case of monolithic architecture, since the code base for adding new functions is the same, there is no need to newly prepare infrastructure for deployment ◦ On the other hand, in the case of microservices architecture, it is costly to prepare new infrastructure ◦ The infra includes not only the server but also 3rd party tools (PagerDuty,
9. 9 Our platform: Centralized GKE Cluster GCP project for GKE Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B
10. 10 Our platform: Centralized GKE Cluster GCP project for GKE Centralized cluster Namespace: Service A Namespace: Service B IAM: SRE IAM: Team A IAM: Team B Service A Service B RBAC: Team A RBAC: Team B New New Cost
11. 11 Cost for migrating to Microservices • Microservices developers have to … by themselves ◦ create GCP project for their service (1 Service : 1 GCP) ◦ prepare for common prerequisites ▪ On-call, Alert, Monitoring, … ◦ prepare for GCP specific features (e.g. Spanner, ...) ◦ connect the service to Centralized GKE Cluster • Microservices platformer want to … ◦ abstract these Terraform Ops and automate it ◦ encourage Infrastructure as Code to the developers
12. 12 Isolation Policy • 1 Microservice ◦ 1 GCP Project (+ some GCP resources) ◦ 1 Cloud Resource (PagerDuty, Sentry, …) ◦ 1 Kubernetes Namespace (in Centralized GKE Cluster) ◦ 1 Team (with some Roles) It’s hard to do these manually! = Need to be automated (provisioning)
13. 13 Topic Microservice Starter Kit
14. 14 • Provisioning Tool ◦ Provide Cloud Resources (GCP, Sentry, PagerDuty, ...) ◦ Provide Kubernetes Resources (Namespace, Secret, ...) ◦ Provide Team (Service owners = GitHub Teams) ◦ ... • Created as “Terraform Module” + “Terraform Template Provider” ◦ Hosted on S3 Microservice Starter Kit
15. 15 Workflow • Generate Microservice skeleton from Template Provider • Configure Microservice settings with Module (Starter Kit) ◦ Fill in Team member list (= Service Owners) ◦ Enable GCP flag? ◦ Enable Sentry flag? ◦ ...
16. 16 Workflow GCP project for GKE mercari-echo-us Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
17. 17 Workflow Namespace Starter Kit 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
18. 18 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
19. 19 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
20. 20 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan terraform apply mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
21. 21 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
22. 22 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
23. 23 Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team GCP project for GKE mercari-echo-us Centralized GKE Cluster
24. 24 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace mercari-echo-jp 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team Centralized GKE Cluster
25. 25 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team mercari-echo-jp Centralized GKE Cluster
26. 26 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
27. 27 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
28. 28 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster
29. 29 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster mercari/tfnotify
30. 30 https://tech.mercari.com/entry/2018/04/09/110000
31. 31 Topic mercari/ microservices-terraform
32. 32 mercari/microservices-terraform • What? ◦ All microservices infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
33. 33 GCP project for GKE mercari-echo-us Workflow Namespace Starter Kit mercari/microservices-terraform Circle CI terraform plan mercari-echo-jp terraform apply Cloud SQL Cloud Spanner Logging Service Account Cloud IAM Namespace 1. Run ./script/new locally 2. Push to GitHub 3. Merge P-R into master 4. Run terraform apply on CI 5. Create GCP project (and some Cloud resources) 6. Create Service Account 7. Create Kubernetes Resources (Namespace, ...) 8. Set Service Account to Secret 9. Create GitHub Team @mercari-echo-jp mercari-echo-jp Centralized GKE Cluster Starter Kit mercari/microservices-terraform Circle CI terraform plan terraform apply @mercari-echo-jp mercari/microservices-terraform • What? ◦ All microservices infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline
34. 34 mercari/microservices-terraform • What? ◦ All Microservices Infra are managed by Terraform code ◦ The ops (terraform apply) is automated by CI pipeline • Why? ◦ To centralize all Microservices Infra code ▪ Eliminate CI pipeline setting cost ◦ To make it easy to review P-R for platform team ▪ Encourage the culture of Infra as Code to the developer
35. 35
36. 36 mercari/microservices-terraform . ├── script/ │ ├── … │ └── new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/
37. 37 mercari/microservices-terraform . ├── script/ │ ├── … │ └── new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory
38. 38 mercari/microservices-terraform . ├── script/ │ ├── … │ └── new* ├── terraform/ │ └── microservices/ │ └── mercari-echo-jp/ │ ├── development/ │ │ ├── … │ │ └── module_microservice_starter_kit.tf │ └── production/ │ └── … ├── … └── modules/ Generated by ./script/new Developers can freely change or add Terraform resource files under their microservice directory The approval and merge authority for P-R are defined by CODEOWNERS and master is protected mercari-echo-jp/ ├── development/ │ ├── … │ └── module_microservice_starter_kit.tf └── production/ └── …
39. 39 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp $ cat .github/CODEOWNERS https://help.github.com/articles/about-codeowners/
40. 40 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/
41. 41 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/
42. 42 mercari/microservices-terraform https://blog.github.com/2017-07-06-introducing-code-owners/
43. 43 mercari/microservices-terraform * @mercari/microservices-platform /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform /terraform/microservices-platform/development/ @mercari/microservices-platform /terraform/microservices-platform/production/ @mercari/microservices-platform # mercari-echo-jp /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp https://help.github.com/articles/about-codeowners/ $ cat .github/CODEOWNERS ☝Generated by Starter Kit
44. 44 mercari/microservices-terraform • リポジトリは中央集権・分散分権モデル ◦ 中央集権 ▪ 必要ならPlatformチームのTerraform Code Review ▪ CI Pipelineなどのセットアップが不要 ▪ 統一的なTerraform Code管理ができる ◦ 分散分権 ▪ 各Microservice dir以下だけApprove/Mergeを各チームに 委譲する→Platformチームをボトルネックにしない ▪ Terraform Stateは分離することで事故を防ぐ
45. 45 Topic Conclusion
46. 46 Conclusion • For accelerating Microservices, ◦ Develop Starter Kit to to make it easy to build the infra ◦ Promote Infrastructure as Code to the developers ◦ Improve Developer productivities