3
Agenda
● Platform Infra @ Mercari
● Kubernetes Security
● Enhance Platform Security with Gatekeeper
● Promote DevSecOps
● What's next
Slide 4
Slide 4 text
4
Microservice @ Mercari
● Microservices are running on GKE
○ Teams have ownership of their services
● Provisioned with Terraform
Slide 5
Slide 5 text
5
Platform Infra @ Mercari
● Enhance platform security, reliability, observability, …
○ Developers can focus more on application logic
○ Developer can work more productively
● For example
○ Enhance CI security
○ Introduce HPA with External metrics / VPA
○ Visualize Infra Cost
○ etc
● Enhancing kubernetes security is also our responsibility
7
Kubernetes Security
What we need to be aware of when setting up manifests are:
● Additional Linux Capabilities
● Host Network
● Host-path volumes
● Privileged Containers
● etc
Slide 8
Slide 8 text
8
Kubernetes Security
● The number of developers and microservices are increasing
○ Can’t put all security responsibility on the developers
● It is our responsibility to ensure we have basic guardrails in our
system.
To resolve that, we introduced Gatekeeper
Slide 9
Slide 9 text
9
Enhance Platform Security with Gatekeeper
What is Gatekeeper
● General-purpose policy engine hosted by OPA (Open Policy
Agent)
● We can create policies in its own language called Rego
○ For example
■ Enforce pods to have resource limits
■ Restrict the addition of Linux capabilities to containers
■ Limit the container registries to only allowed certain ones
Slide 10
Slide 10 text
10
Enhance Platform Security with Gatekeeper
What is Gatekeeper
● Gatekeeper validates kubernetes resources based on policies
○ Policies are deployed as CRD
○ Gatekeeper is registered as Validation Webhook
Slide 11
Slide 11 text
11
Enhance Platform Security with Gatekeeper
Security risks in containers and mitigations
Policies that we implemented to enforce security best practices in our
Kubernetes workloads:
● Prevent adding extra capabilities
● Restrict the usage of the host network
● Prevent containers from running in privileged mode
Slide 12
Slide 12 text
12
Enhance Platform Security with Gatekeeper
1. Prevent adding extra capabilities
● Docker has already blocked some capabilities such as SYS_ADMIN,
NET_ADMIN, and so on
○ but thoughtlessly adding other capabilities will have
unexpected security risks
○ Whitelist some components
● Related CVE: CVE-2020-14386
○ Allows escape from a container to obtain root privileges
○ This is related to CAP_NET_RAW
■ Enables ARP (Address Resolution Protocol) spoofing
attacks
Slide 13
Slide 13 text
13
Enhance Platform Security with Gatekeeper
1. Prevent adding extra capabilities
● This policy checks PodSpec definitions in all Pod, Deployment,
DaemonSet, and so on
○ Checks `.spec.containers[].securityContext.capabilities.add`
Slide 14
Slide 14 text
14
Enhance Platform Security with Gatekeeper
2. Restrict the usage of the host network
● Containers using the host network can do many things to
jeopardize our cluster
○ Allow processes in the pod to talk on the host’s loopback
adapter
○ Create port conflicts with other containers
● Related CVE: CVE-2020-15257
○ containerd containers running in the host network
namespaces with UID 0 allowed to gain the host root
privileges
Slide 15
Slide 15 text
15
Enhance Platform Security with Gatekeeper
2. Restrict the usage of the host network
● This policy checks PodSpec definitions in all Pod, Deployment,
DaemonSet, and so on
○ limits the value of `.spec.hostNetwork` to false
Slide 16
Slide 16 text
16
Enhance Platform Security with Gatekeeper
3. Prevent containers from running in privileged mode
● Privileged mode allows container processes to access host devices
○ Enables processes to behave like host processes (running
outside a container)
○ Allows bypassing almost all Kubernetes and Docker security
controls
Slide 17
Slide 17 text
17
Enhance Platform Security with Gatekeeper
3. Prevent containers from running in privileged mode
● This policy checks PodSpec definitions in all Pod, Deployment,
DaemonSet, and so on
○ limits the value of
`.spec.containers[].securityContext.privileged` to false
Slide 18
Slide 18 text
18
Enhance Platform Security with Gatekeeper
We have dozens of these Gatekeeper policies which are helping us in
keeping security best practices
● Related to
○ host-path volumes
○ host namespaces
○ Istio
○ node pool access
○ etc
Slide 19
Slide 19 text
19
Promote DevSecOps
Those works are based on the very concept of DevSecOps
● Adds Continuous Security to DevOps
● Spans the entire software development life cycle
Slide 20
Slide 20 text
20
Promote DevSecOps
In our Gatekeeper example
● Validation when deploying services to the Kubernetes in the
development environment
○ Automatic and continuous security checks during the
deployment phase of the development cycle
○ Can find and fix problems early on in your daily development
○ Speeds up releases
● Fundamental idea in DevSecOps: Shift-Left-Security
Slide 21
Slide 21 text
21
Promote DevSecOps
Can also use Conftest in the CI level
● It uses the Rego language for writing assertions
Slide 22
Slide 22 text
22
Promote DevSecOps
● Asked the Security team to help us from the planning stage
○ We proceeded with the plan under their review
● The collaboration between our platform team and the security
team is essential to promote DevSecOps
Slide 23
Slide 23 text
23
What's next
● Not only security, Gatekeeper will become our platform wide
guardrails
○ Dedicated node-pool for a namespace
○ Restrict developers from using alpha release features
○ Force the developer to use a specific option for reliability or
performance
■ VPA: Enforces developers to specify a lower bound of
request resources
■ Enforce to write resource limits
Slide 24
Slide 24 text
24
Reference
● Enhance Kubernetes Security with OPA Gatekeeper, mercari
engineering
● DevSecOps: What Is It and Why Is It Gaining Momentum in the
Industry?, mercari engineering
● open-policy-agent/gatekeeper, GitHub