Terraform Ops for Microservices

Talk at https://mercari.connpass.com/event/92168/

Transcript

1. Mercari Meetup for Microservices Platform
   July 19, 2018 / @b4b4r07
   Terraform Ops for Microservices
   

   View Slide

2. 2
   About me
   @b4b4r07 / babarot
   Blog / tellme.tokyo
   SRE in Microservices Platform
   Team at Mercari, Inc.
   

   View Slide

3. 3
   Topics
   1. Microservice Starter Kit
   2. mercari/microservices-terraform
   

   View Slide

4. 4
   At first,
   In Mercari, we’re migrating our architecture from Monolithic one to
   Microservices one now...
   

   View Slide

5. 5
   Monolithic App
   Data Access
   Layer
   Business Logic
   UI
   Database
   

   View Slide

6. 6
   Monolithic App
   Data Access
   Layer
   Business Logic
   UI
   Database
   Microservices Apps
   Units that can be deployed isolatedly
   

   View Slide

7. 7
   Monolithic App
   Data Access
   Layer
   Business Logic
   UI
   Database
   Microservices Apps
   Units that can be deployed isolatedly
   

   View Slide

8. 8
   Problems of Microservices Architecture
   ● Every time a new microservice is developed, it’s need to prepare
   the infrastructure
   ○ In the case of monolithic architecture, since the code base
   for adding new functions is the same, there is no need to
   newly prepare infrastructure for deployment
   ○ On the other hand, in the case of microservices
   architecture, it is costly to prepare new infrastructure
   ○ The infra includes not only the server but also 3rd party
   tools (PagerDuty,
   

   View Slide

9. 9
   Our platform: Centralized GKE Cluster
   GCP project for GKE
   Centralized cluster
   Namespace: Service A
   Namespace: Service B
   IAM: SRE IAM: Team A
   IAM: Team B
   Service A
   Service B
   RBAC: Team A
   RBAC: Team B
   

   View Slide

10. 10
    Our platform: Centralized GKE Cluster
    GCP project for GKE
    Centralized cluster
    Namespace: Service A
    Namespace: Service B
    IAM: SRE IAM: Team A
    IAM: Team B
    Service A
    Service B
    RBAC: Team A
    RBAC: Team B
    New
    New
    Cost
    

    View Slide

11. 11
    Cost for migrating to Microservices
    ● Microservices developers have to … by themselves
    ○ create GCP project for their service (1 Service : 1 GCP)
    ○ prepare for common prerequisites
    ■ On-call, Alert, Monitoring, …
    ○ prepare for GCP specific features (e.g. Spanner, ...)
    ○ connect the service to Centralized GKE Cluster
    ● Microservices platformer want to …
    ○ abstract these Terraform Ops and automate it
    ○ encourage Infrastructure as Code to the developers
    

    View Slide

12. 12
    Isolation Policy
    ● 1 Microservice
    ○ 1 GCP Project (+ some GCP resources)
    ○ 1 Cloud Resource (PagerDuty, Sentry, …)
    ○ 1 Kubernetes Namespace (in Centralized GKE Cluster)
    ○ 1 Team (with some Roles)
    It’s hard to do these manually! = Need to be automated
    (provisioning)
    

    View Slide

13. 13
    Topic
    Microservice
    Starter Kit
    

    View Slide

14. 14
    ● Provisioning Tool
    ○ Provide Cloud Resources (GCP, Sentry, PagerDuty, ...)
    ○ Provide Kubernetes Resources (Namespace, Secret, ...)
    ○ Provide Team (Service owners = GitHub Teams)
    ○ ...
    ● Created as “Terraform Module” + “Terraform Template
    Provider”
    ○ Hosted on S3
    Microservice Starter Kit
    

    View Slide

15. 15
    Workflow
    ● Generate Microservice skeleton from Template Provider
    ● Configure Microservice settings with Module (Starter Kit)
    ○ Fill in Team member list (= Service Owners)
    ○ Enable GCP flag?
    ○ Enable Sentry flag?
    ○ ...
    

    View Slide

16. 16
    Workflow
    GCP project for GKE
    mercari-echo-us
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    Centralized GKE Cluster
    

    View Slide

17. 17
    Workflow
    Namespace
    Starter Kit
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

18. 18
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

19. 19
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

20. 20
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    terraform apply
    mercari-echo-jp
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

21. 21
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

22. 22
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

23. 23
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    GCP project for GKE
    mercari-echo-us
    Centralized GKE Cluster
    

    View Slide

24. 24
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    mercari-echo-jp
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    Centralized GKE Cluster
    

    View Slide

25. 25
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    mercari-echo-jp
    Centralized GKE Cluster
    

    View Slide

26. 26
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    

    View Slide

27. 27
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    

    View Slide

28. 28
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    

    View Slide

29. 29
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    mercari/tfnotify
    

    View Slide

30. 30
    https://tech.mercari.com/entry/2018/04/09/110000
    

    View Slide

31. 31
    Topic
    mercari/
    microservices-terraform
    

    View Slide

32. 32
    mercari/microservices-terraform
    ● What?
    ○ All microservices infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline
    

    View Slide

33. 33
    GCP project for GKE
    mercari-echo-us
    Workflow
    Namespace
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    mercari-echo-jp
    terraform apply
    Cloud
    SQL
    Cloud
    Spanner
    Logging
    Service Account
    Cloud IAM
    Namespace
    1. Run ./script/new locally
    2. Push to GitHub
    3. Merge P-R into master
    4. Run terraform apply on CI
    5. Create GCP project
    (and some Cloud resources)
    6. Create Service Account
    7. Create Kubernetes Resources
    (Namespace, ...)
    8. Set Service Account to Secret
    9. Create GitHub Team
    @mercari-echo-jp
    mercari-echo-jp
    Centralized GKE Cluster
    Starter Kit mercari/microservices-terraform Circle CI
    terraform plan
    terraform apply
    @mercari-echo-jp
    mercari/microservices-terraform
    ● What?
    ○ All microservices infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline
    

    View Slide

34. 34
    mercari/microservices-terraform
    ● What?
    ○ All Microservices Infra are managed by Terraform code
    ○ The ops (terraform apply) is automated by CI pipeline
    ● Why?
    ○ To centralize all Microservices Infra code
    ■ Eliminate CI pipeline setting cost
    ○ To make it easy to review P-R for platform team
    ■ Encourage the culture of Infra as Code to the developer
    

    View Slide

35. View Slide

36. 36
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/
    

    View Slide

37. 37
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/
    Generated by ./script/new
    Developers can freely change or add
    Terraform resource files under their
    microservice directory
    

    View Slide

38. 38
    mercari/microservices-terraform
    .
    ├── script/
    │ ├── …
    │ └── new*
    ├── terraform/
    │ └── microservices/
    │ └── mercari-echo-jp/
    │ ├── development/
    │ │ ├── …
    │ │ └── module_microservice_starter_kit.tf
    │ └── production/
    │ └── …
    ├── …
    └── modules/
    Generated by ./script/new
    Developers can freely change or add
    Terraform resource files under their
    microservice directory
    The approval and merge authority for P-R are
    defined by CODEOWNERS and master is protected
    mercari-echo-jp/
    ├── development/
    │ ├── …
    │ └── module_microservice_starter_kit.tf
    └── production/
    └── …
    

    View Slide

39. 39
    mercari/microservices-terraform
    * @mercari/microservices-platform
    /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
    /terraform/microservices-platform/development/ @mercari/microservices-platform
    /terraform/microservices-platform/production/ @mercari/microservices-platform
    # mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
    $ cat .github/CODEOWNERS
    https://help.github.com/articles/about-codeowners/
    

    View Slide

40. 40
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/
    

    View Slide

41. 41
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/
    

    View Slide

42. 42
    mercari/microservices-terraform
    https://blog.github.com/2017-07-06-introducing-code-owners/
    

    View Slide

43. 43
    mercari/microservices-terraform
    * @mercari/microservices-platform
    /terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
    /terraform/microservices-platform/development/ @mercari/microservices-platform
    /terraform/microservices-platform/production/ @mercari/microservices-platform
    # mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
    /terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
    https://help.github.com/articles/about-codeowners/
    $ cat .github/CODEOWNERS
    ☝Generated by Starter Kit
    

    View Slide

44. 44
    mercari/microservices-terraform
    ● リポジトリは中央集権・分散分権モデル
    ○ 中央集権
    ■ 必要ならPlatformチームのTerraform Code Review
    ■ CI Pipelineなどのセットアップが不要
    ■ 統一的なTerraform Code管理ができる
    ○ 分散分権
    ■ 各Microservice dir以下だけApprove/Mergeを各チームに
    委譲する→Platformチームをボトルネックにしない
    ■ Terraform Stateは分離することで事故を防ぐ
    

    View Slide

45. 45
    Topic
    Conclusion
    

    View Slide

46. 46
    Conclusion
    ● For accelerating Microservices,
    ○ Develop Starter Kit to to make it easy to build the infra
    ○ Promote Infrastructure as Code to the developers
    ○ Improve Developer productivities
    

    View Slide

47. View Slide