1 Enhance Kubernetes Security with Gatekeeper Mercari Platform Group Tech Talk Morito Ikeda

2 @_moricho_ (Morito Ikeda) Job - ~2020/10: BASE / AbemaTV / Cybozu... - 2020/10~: Mercari, Microservices Platform team (Platform Infra) - OSS: gVisor, Firecracker, ... Hobby - Camera, GoPro - Music (Bass, Drum)

3 Agenda ● Platform Infra @ Mercari ● Kubernetes Security ● Enhance Platform Security with Gatekeeper ● Promote DevSecOps ● What's next

4 Microservice @ Mercari ● Microservices are running on GKE ○ Teams have ownership of their services ● Provisioned with Terraform

5 Platform Infra @ Mercari ● Enhance platform security, reliability, observability, … ○ Developers can focus more on application logic ○ Developer can work more productively ● For example ○ Enhance CI security ○ Introduce HPA with External metrics / VPA ○ Visualize Infra Cost ○ etc ● Enhancing kubernetes security is also our responsibility

6 Kubernetes Security ● Container Image Scan (ex. Trivy) ● Manifest Configuration (ex. Gatekeeper, Conftest) ● Container Runtime (ex. gVisor) ● etc Today, will focus on Manifest Configuration

7 Kubernetes Security What we need to be aware of when setting up manifests are: ● Additional Linux Capabilities ● Host Network ● Host-path volumes ● Privileged Containers ● etc

8 Kubernetes Security ● The number of developers and microservices are increasing ○ Can’t put all security responsibility on the developers ● It is our responsibility to ensure we have basic guardrails in our system. To resolve that, we introduced Gatekeeper

9 Enhance Platform Security with Gatekeeper What is Gatekeeper ● General-purpose policy engine hosted by OPA (Open Policy Agent) ● We can create policies in its own language called Rego ○ For example ■ Enforce pods to have resource limits ■ Restrict the addition of Linux capabilities to containers ■ Limit the container registries to only allowed certain ones

10 Enhance Platform Security with Gatekeeper What is Gatekeeper ● Gatekeeper validates kubernetes resources based on policies ○ Policies are deployed as CRD ○ Gatekeeper is registered as Validation Webhook

11 Enhance Platform Security with Gatekeeper Security risks in containers and mitigations Policies that we implemented to enforce security best practices in our Kubernetes workloads: ● Prevent adding extra capabilities ● Restrict the usage of the host network ● Prevent containers from running in privileged mode

12 Enhance Platform Security with Gatekeeper 1. Prevent adding extra capabilities ● Docker has already blocked some capabilities such as SYS_ADMIN, NET_ADMIN, and so on ○ but thoughtlessly adding other capabilities will have unexpected security risks ○ Whitelist some components ● Related CVE: CVE-2020-14386 ○ Allows escape from a container to obtain root privileges ○ This is related to CAP_NET_RAW ■ Enables ARP (Address Resolution Protocol) spoofing attacks

13 Enhance Platform Security with Gatekeeper 1. Prevent adding extra capabilities ● This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ○ Checks `.spec.containers[].securityContext.capabilities.add`

14 Enhance Platform Security with Gatekeeper 2. Restrict the usage of the host network ● Containers using the host network can do many things to jeopardize our cluster ○ Allow processes in the pod to talk on the host’s loopback adapter ○ Create port conflicts with other containers ● Related CVE: CVE-2020-15257 ○ containerd containers running in the host network namespaces with UID 0 allowed to gain the host root privileges

15 Enhance Platform Security with Gatekeeper 2. Restrict the usage of the host network ● This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ○ limits the value of `.spec.hostNetwork` to false

16 Enhance Platform Security with Gatekeeper 3. Prevent containers from running in privileged mode ● Privileged mode allows container processes to access host devices ○ Enables processes to behave like host processes (running outside a container) ○ Allows bypassing almost all Kubernetes and Docker security controls

17 Enhance Platform Security with Gatekeeper 3. Prevent containers from running in privileged mode ● This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ○ limits the value of `.spec.containers[].securityContext.privileged` to false

18 Enhance Platform Security with Gatekeeper We have dozens of these Gatekeeper policies which are helping us in keeping security best practices ● Related to ○ host-path volumes ○ host namespaces ○ Istio ○ node pool access ○ etc

19 Promote DevSecOps Those works are based on the very concept of DevSecOps ● Adds Continuous Security to DevOps ● Spans the entire software development life cycle

20 Promote DevSecOps In our Gatekeeper example ● Validation when deploying services to the Kubernetes in the development environment ○ Automatic and continuous security checks during the deployment phase of the development cycle ○ Can find and fix problems early on in your daily development ○ Speeds up releases ● Fundamental idea in DevSecOps: Shift-Left-Security

21 Promote DevSecOps Can also use Conftest in the CI level ● It uses the Rego language for writing assertions

22 Promote DevSecOps ● Asked the Security team to help us from the planning stage ○ We proceeded with the plan under their review ● The collaboration between our platform team and the security team is essential to promote DevSecOps

23 What's next ● Not only security, Gatekeeper will become our platform wide guardrails ○ Dedicated node-pool for a namespace ○ Restrict developers from using alpha release features ○ Force the developer to use a specific option for reliability or performance ■ VPA: Enforces developers to specify a lower bound of request resources ■ Enforce to write resource limits

24 Reference ● Enhance Kubernetes Security with OPA Gatekeeper, mercari engineering ● DevSecOps: What Is It and Why Is It Gaining Momentum in the Industry?, mercari engineering ● open-policy-agent/gatekeeper, GitHub