How We Harden Platform Security at Mercari

Slide 1

Slide 1 text

How We Harden Platform Security CloudNative Days Tokyo 2021

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

How We Harden Platform Security CloudNative Days Tokyo 2021

Slide 4

Slide 4 text

Taichi Nakashima @deeeet / @tcnksm Engineering head of Developer Productivity Engineering

Slide 5

Slide 5 text

https://e34.fm

Slide 6

Slide 6 text

Table of Contents  ● Microservices Platform Overview ● 3 Cases of Harden Platform Security ○ Multi Tenant Security ○ Production Operation Security ○ Supply Chain Security ● Lessons Learned

Slide 7

Slide 7 text

Microservices Platform Overview

Slide 8

Slide 8 text

Service C Service D Service B Service E Service A Mercari and Merpay Microservices Google Kubernetes Engine 200+ Microservices 4000+Kubernetes Pods 2 main business

Slide 9

Slide 9 text

Service A Team Mercari SRE Merpay SRE Service A Service B Team Service B Service C Team Service C Work closely or embedded Platform Platform Team

Slide 10

Slide 10 text

Harden Platform Security

Slide 11

Slide 11 text

Base Principle: Shared Responsibility 

Slide 12

Slide 12 text

Base Principle: Defense in Depth 

Slide 13

Slide 13 text

Harden Platform Security  ● Multi-tenant Security ● Production Operation Security ● Supply Chain Security

Slide 14

Slide 14 text

Multi-tenancy  Multi-tenancy is architecture pattern which, instead of building platforms per business or services, prepares isolated tenant per services and hosts them together on single platform. While multi-tenancy increase complexity, you can avoid reinventing wheels in the organization, reduce the operational costs, and leverage improvements to all.

Slide 15

Slide 15 text

Principle: Least Privilege  Least privilege means human user or workload must be able to access only resources that are necessary for its legitimate purpose. In multi-tenancy context, it’s important to make sure only tenant owners are able to access its tenant’s resources.

Slide 16

Slide 16 text

Multi-tenant Least Privileges on  ● Kubernetes Cluster ● Infrastructure as Code (IaC) Monorepo and Build System

Slide 17

Slide 17 text

Service A Namespace Kubernetes Cluster Service B Namespace System Namespace Container A Container A Resources Container A Container A Resources Container A Container A Resources Service A Team ✅ RBAC 🚫 🚫

Slide 18

Slide 18 text

Multi-tenant Least Privileges on  ● Kubernetes Cluster ● Infrastructure as Code (IaC) Monorepo and Build System

Slide 19

Slide 19 text

CI System Build System Service A Tenant Container A Container A Resources CI System Service B Tenant Container A Container A Resources CI System System Tenant Container A Container A Resources IaC Monorepo Service A Team Service B Team Platform Team PR Conﬁgure

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Service A Team ✅ CODEOWNER 🚫 🚫 /service-a module.tf google_spanner_database.tf google_storage_bucket.tf ... /service-b module.tf google_bigquery_dataset.tf google_pubsub_topic.tf ... /system module.tf google_container_cluster.tf google_compute_firewall.tf ... Infra as Code Monorepo

Slide 22

Slide 22 text

Slide 23

Slide 23 text

CI System Build System Service A Tenant Container A Container A Resources Build Account IAM CI System Service B Tenant Container A Container A Resources CI System System Tenant Container A Container A Resources IAM IAM

Slide 24

Slide 24 text

CI System Build System Service A Tenant Container A Container A Resources Service A Account (Keyless) Build Account (keyless) CI System Service B Tenant Container A Container A Resources CI System System Tenant Container A Container A Resources Impersonate IAM Service B Account (Keyless) IAM Impersonate System Account (Keyless) IAM Impersonate

Slide 25

Slide 25 text

CI System Build System Service A Tenant Container A Container A Resources CI System Service B Tenant Container A Container A Resources CI System System Tenant Container A Container A Resources Short-lived token 🚫 🚫 ✅ Impersonate IAM Service A Account (Keyless) Build Account (keyless)

Slide 26

Slide 26 text

Harden Platform Security  ● Multi-tenant Security ● Production Operation Security ● Supply Chain Security

Slide 27

Slide 27 text

https://sre.google/books/building-secure-reliable-systems/

Slide 28

Slide 28 text

Goal: Zero Touch Production  The speciﬁc goal of these interfaces—like Zero Touch Production (ZTP) ,..., is to make Google safer and reduce outages by removing direct human access to production roles. Instead, humans have indirect access to production through tooling and automation that make predictable and controlled changes to production infrastructure. - Building Secure and Reliable Systems, Chapter 5

Slide 29

Slide 29 text

Service A Team Service A Namespace Kubernetes Cluster Container A Container A Resources View Edit IaC Repository +Build System Service B Namespace System Namespace

Slide 30

Slide 30 text

Service A Team Service A Namespace Kubernetes Cluster Container A Container A Resources View Edit Edit IaC Repository +Build System Temporary Role Grant Service B Namespace System Namespace

Slide 31

Slide 31 text

Service A Team Service A Namespace Kubernetes Cluster Container A Container A Resources View Edit Edit Edit IaC Repository +Build System Automated Workﬂows Temporary Role Grant Service B Namespace System Namespace

Slide 32

Slide 32 text

Harden Platform Security  ● Multi-tenant Security ● Production Operation Security ● Supply Chain Security

Slide 33

Slide 33 text

Source Build Deploy Registry Cluster Dependency

Slide 34

Slide 34 text

Source Build Deploy Registry Cluster Dependency Compromise build system Compromise artifact registry Inject bad container image Bypass code review Inject bad/vulnerable dependency Compromise source control system Alter code Compromise deploy system Use bad image

Slide 35

Slide 35 text

Practice: Verify Artifacts, Not Just People The controls around the source, build, and test infrastructure have limited effect if adversaries can bypass them by deploying directly to production. It is not sufﬁcient to verify who initiated a deployment, because that actor may make a mistake or may be intentionally deploying a malicious change. Instead, deployment environments should verify what is being deployed. - Building Secure and Reliable Systems, Chapter 14

Slide 36

Slide 36 text

Source Build Deploy Registry Metadata Cluster Kritis Dependency ✅ Sign Check

Slide 37

Slide 37 text

Lessons Learned

Slide 38

Slide 38 text

Secure By Default  Security hardening = “migration” takes lots of time and costs... Build the security policy by allowlist, instead of denylist!

Slide 39

Slide 39 text

Build Abstraction  Hide infrastructure and security complexity from the developers and control them centrally in background by experts. Make the future migration easy!

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text