Slide 1
Slide 1 text
Mercari Meetup for Microservices Platform
July 19, 2018 / @b4b4r07
Terraform Ops for Microservices
Slide 2
Slide 2 text
2
About me
@b4b4r07 / babarot
Blog / tellme.tokyo
SRE in Microservices Platform
Team at Mercari, Inc.
Slide 3
Slide 3 text
3
Topics
1. Microservice Starter Kit
2. mercari/microservices-terraform
Slide 4
Slide 4 text
4
At first,
In Mercari, we’re migrating our architecture from Monolithic one to
Microservices one now...
Slide 5
Slide 5 text
5
Monolithic App
Data Access
Layer
Business Logic
UI
Database
Slide 6
Slide 6 text
6
Monolithic App
Data Access
Layer
Business Logic
UI
Database
Microservices Apps
Units that can be deployed isolatedly
Slide 7
Slide 7 text
7
Monolithic App
Data Access
Layer
Business Logic
UI
Database
Microservices Apps
Units that can be deployed isolatedly
Slide 8
Slide 8 text
8
Problems of Microservices Architecture
● Every time a new microservice is developed, it’s need to prepare
the infrastructure
○ In the case of monolithic architecture, since the code base
for adding new functions is the same, there is no need to
newly prepare infrastructure for deployment
○ On the other hand, in the case of microservices
architecture, it is costly to prepare new infrastructure
○ The infra includes not only the server but also 3rd party
tools (PagerDuty,
Slide 9
Slide 9 text
9
Our platform: Centralized GKE Cluster
GCP project for GKE
Centralized cluster
Namespace: Service A
Namespace: Service B
IAM: SRE IAM: Team A
IAM: Team B
Service A
Service B
RBAC: Team A
RBAC: Team B
Slide 10
Slide 10 text
10
Our platform: Centralized GKE Cluster
GCP project for GKE
Centralized cluster
Namespace: Service A
Namespace: Service B
IAM: SRE IAM: Team A
IAM: Team B
Service A
Service B
RBAC: Team A
RBAC: Team B
New
New
Cost
Slide 11
Slide 11 text
11
Cost for migrating to Microservices
● Microservices developers have to … by themselves
○ create GCP project for their service (1 Service : 1 GCP)
○ prepare for common prerequisites
■ On-call, Alert, Monitoring, …
○ prepare for GCP specific features (e.g. Spanner, ...)
○ connect the service to Centralized GKE Cluster
● Microservices platformer want to …
○ abstract these Terraform Ops and automate it
○ encourage Infrastructure as Code to the developers
Slide 12
Slide 12 text
12
Isolation Policy
● 1 Microservice
○ 1 GCP Project (+ some GCP resources)
○ 1 Cloud Resource (PagerDuty, Sentry, …)
○ 1 Kubernetes Namespace (in Centralized GKE Cluster)
○ 1 Team (with some Roles)
It’s hard to do these manually! = Need to be automated
(provisioning)
Slide 13
Slide 13 text
13
Topic
Microservice
Starter Kit
Slide 14
Slide 14 text
14
● Provisioning Tool
○ Provide Cloud Resources (GCP, Sentry, PagerDuty, ...)
○ Provide Kubernetes Resources (Namespace, Secret, ...)
○ Provide Team (Service owners = GitHub Teams)
○ ...
● Created as “Terraform Module” + “Terraform Template
Provider”
○ Hosted on S3
Microservice Starter Kit
Slide 15
Slide 15 text
15
Workflow
● Generate Microservice skeleton from Template Provider
● Configure Microservice settings with Module (Starter Kit)
○ Fill in Team member list (= Service Owners)
○ Enable GCP flag?
○ Enable Sentry flag?
○ ...
Slide 16
Slide 16 text
16
Workflow
GCP project for GKE
mercari-echo-us
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
Centralized GKE Cluster
Slide 17
Slide 17 text
17
Workflow
Namespace
Starter Kit
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 18
Slide 18 text
18
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 19
Slide 19 text
19
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 20
Slide 20 text
20
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
terraform apply
mercari-echo-jp
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 21
Slide 21 text
21
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 22
Slide 22 text
22
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 23
Slide 23 text
23
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
GCP project for GKE
mercari-echo-us
Centralized GKE Cluster
Slide 24
Slide 24 text
24
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
mercari-echo-jp
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
Centralized GKE Cluster
Slide 25
Slide 25 text
25
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
mercari-echo-jp
Centralized GKE Cluster
Slide 26
Slide 26 text
26
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
@mercari-echo-jp
mercari-echo-jp
Centralized GKE Cluster
Slide 27
Slide 27 text
27
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
@mercari-echo-jp
mercari-echo-jp
Centralized GKE Cluster
Slide 28
Slide 28 text
28
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
@mercari-echo-jp
mercari-echo-jp
Centralized GKE Cluster
Slide 29
Slide 29 text
29
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
@mercari-echo-jp
mercari-echo-jp
Centralized GKE Cluster
mercari/tfnotify
Slide 30
Slide 30 text
30
https://tech.mercari.com/entry/2018/04/09/110000
Slide 31
Slide 31 text
31
Topic
mercari/
microservices-terraform
Slide 32
Slide 32 text
32
mercari/microservices-terraform
● What?
○ All microservices infra are managed by Terraform code
○ The ops (terraform apply) is automated by CI pipeline
Slide 33
Slide 33 text
33
GCP project for GKE
mercari-echo-us
Workflow
Namespace
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
mercari-echo-jp
terraform apply
Cloud
SQL
Cloud
Spanner
Logging
Service Account
Cloud IAM
Namespace
1. Run ./script/new locally
2. Push to GitHub
3. Merge P-R into master
4. Run terraform apply on CI
5. Create GCP project
(and some Cloud resources)
6. Create Service Account
7. Create Kubernetes Resources
(Namespace, ...)
8. Set Service Account to Secret
9. Create GitHub Team
@mercari-echo-jp
mercari-echo-jp
Centralized GKE Cluster
Starter Kit mercari/microservices-terraform Circle CI
terraform plan
terraform apply
@mercari-echo-jp
mercari/microservices-terraform
● What?
○ All microservices infra are managed by Terraform code
○ The ops (terraform apply) is automated by CI pipeline
Slide 34
Slide 34 text
34
mercari/microservices-terraform
● What?
○ All Microservices Infra are managed by Terraform code
○ The ops (terraform apply) is automated by CI pipeline
● Why?
○ To centralize all Microservices Infra code
■ Eliminate CI pipeline setting cost
○ To make it easy to review P-R for platform team
■ Encourage the culture of Infra as Code to the developer
Slide 35
Slide 35 text
35
Slide 36
Slide 36 text
36
mercari/microservices-terraform
.
├── script/
│ ├── …
│ └── new*
├── terraform/
│ └── microservices/
│ └── mercari-echo-jp/
│ ├── development/
│ │ ├── …
│ │ └── module_microservice_starter_kit.tf
│ └── production/
│ └── …
├── …
└── modules/
Slide 37
Slide 37 text
37
mercari/microservices-terraform
.
├── script/
│ ├── …
│ └── new*
├── terraform/
│ └── microservices/
│ └── mercari-echo-jp/
│ ├── development/
│ │ ├── …
│ │ └── module_microservice_starter_kit.tf
│ └── production/
│ └── …
├── …
└── modules/
Generated by ./script/new
Developers can freely change or add
Terraform resource files under their
microservice directory
Slide 38
Slide 38 text
38
mercari/microservices-terraform
.
├── script/
│ ├── …
│ └── new*
├── terraform/
│ └── microservices/
│ └── mercari-echo-jp/
│ ├── development/
│ │ ├── …
│ │ └── module_microservice_starter_kit.tf
│ └── production/
│ └── …
├── …
└── modules/
Generated by ./script/new
Developers can freely change or add
Terraform resource files under their
microservice directory
The approval and merge authority for P-R are
defined by CODEOWNERS and master is protected
mercari-echo-jp/
├── development/
│ ├── …
│ └── module_microservice_starter_kit.tf
└── production/
└── …
Slide 39
Slide 39 text
39
mercari/microservices-terraform
* @mercari/microservices-platform
/terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
/terraform/microservices-platform/development/ @mercari/microservices-platform
/terraform/microservices-platform/production/ @mercari/microservices-platform
# mercari-echo-jp
/terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
/terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
$ cat .github/CODEOWNERS
https://help.github.com/articles/about-codeowners/
Slide 40
Slide 40 text
40
mercari/microservices-terraform
https://blog.github.com/2017-07-06-introducing-code-owners/
Slide 41
Slide 41 text
41
mercari/microservices-terraform
https://blog.github.com/2017-07-06-introducing-code-owners/
Slide 42
Slide 42 text
42
mercari/microservices-terraform
https://blog.github.com/2017-07-06-introducing-code-owners/
Slide 43
Slide 43 text
43
mercari/microservices-terraform
* @mercari/microservices-platform
/terraform/modules/microservices/starter-kit/ @mercari/microservices-platform
/terraform/microservices-platform/development/ @mercari/microservices-platform
/terraform/microservices-platform/production/ @mercari/microservices-platform
# mercari-echo-jp
/terraform/microservices/mercari-echo-jp/development/ @mercari/mercari-echo-jp
/terraform/microservices/mercari-echo-jp/production/ @mercari/mercari-echo-jp
https://help.github.com/articles/about-codeowners/
$ cat .github/CODEOWNERS
☝Generated by Starter Kit
Slide 44
Slide 44 text
44
mercari/microservices-terraform
● リポジトリは中央集権・分散分権モデル
○ 中央集権
■ 必要ならPlatformチームのTerraform Code Review
■ CI Pipelineなどのセットアップが不要
■ 統一的なTerraform Code管理ができる
○ 分散分権
■ 各Microservice dir以下だけApprove/Mergeを各チームに
委譲する→Platformチームをボトルネックにしない
■ Terraform Stateは分離することで事故を防ぐ
Slide 45
Slide 45 text
45
Topic
Conclusion
Slide 46
Slide 46 text
46
Conclusion
● For accelerating Microservices,
○ Develop Starter Kit to to make it easy to build the infra
○ Promote Infrastructure as Code to the developers
○ Improve Developer productivities
Slide 47
Slide 47 text
No content