1 #GitLabCommit Bringing Kubernetes Policy Enforcement to GitLab

2 #GitLabCommit Sandeep Parikh DevRel Engineer, Google Cloud @crcsmnky @crcsmnky

3 #GitLabCommit Policies and Enforcement

4 #GitLabCommit Policies Definitions Policy management Policy enforcement Rules that tell us how we can configure a resource The practice of developing, deploying, and applying rules The scope and actions taken, based on defined rules Constraints & Templates Open Policy Agent OPA Gatekeeper

5 #GitLabCommit Open Policy Agent Decouple policy decisions from services to achieve unified control across the entire stack. Unified Express policies in a high-level declarative language that promotes safe, fine-grained logic. Declarative Leverage arbitrary external data in policies to ensure that important requirements are enforced. Context Aware

6 #GitLabCommit #GitLabCommit Gatekeeper OPA Gatekeeper brings Open Policy Agent to Kubernetes as an admission controller Gatekeeper turns OPA policies into Kubernetes custom resources Resources are managed using the standard Kubernetes Resource Model (KRM) Gatekeeper Kubernetes kubectl AdmissionReview (request) AdmissionReview (response)

7 #GitLabCommit Templates apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: k8sblocknodeport spec: crd: spec: names: kind: K8sBlockNodePort targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sblocknodeport violation[{"msg": msg}] { input.review.kind.kind == "Service" input.review.object.spec.type == "NodePort" msg := "Cannot create service of type NodePort" } Policies are written in Rego and packaged as parameterized ConstraintTemplate objects.

8 #GitLabCommit apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: k8sblocknodeport spec: crd: spec: names: kind: K8sBlockNodePort targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sblocknodeport violation[{"msg": msg}] { input.review.kind.kind == "Service" input.review.object.spec.type == "NodePort" msg := "Cannot create service of type NodePort" } Templates Policies are written in Rego and packaged as parameterized ConstraintTemplate objects. The ConstraintTemplate extends Gatekeeper by adding a new policy that can be invoked via a new CR.

9 #GitLabCommit Constraints apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Service"] --- apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007 Constraints are instantiations of a ConstraintTemplate CR and can be optionally scoped to specific objects and/or namespaces.

10 #GitLabCommit Constraints apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Service"] --- apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007 Constraints are instantiations of a ConstraintTemplate CR and can be optionally scoped to specific objects and/or namespaces. When violated, Constraints can either deny admission or allow entry, and audit the violation in the status field.

11 #GitLabCommit #GitLabCommit Enforcement Gatekeeper reviews the request then denies admission or issues warnings, based on violations. But this only happens when resources are deployed. Gatekeeper Kubernetes Evaluate policy for incoming object Reject object as it violates policy apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007

12 #GitLabCommit #GitLabCommit Runtime enforcement If resources violate any policies they will be rejected. But with GitOps, the controller will continually fail* to sync resources with clusters. Kubernetes Repo Gatekeeper GitOps * pending baked-in backoff, depends on your controller, YMMV, etc.

13 #GitLabCommit Workflows

14 #GitLabCommit Validation workflows Commits are pushed MRs are submitted Push Deploy ↺ Enforcement Push Review & Enforcement Deploy Merge

15 #GitLabCommit Validation tools googlecontainertools.github.io/kpt Kpt is an OSS tool for building declarative workflows on top of resource configuration. conftest.dev Conftest is a utility to help you write tests against structured configuration data. $ conftest test deployment.yaml --policy ./policy $ docker run -i gcr.io/kpt-functions/gatekeeper-validate

16 #GitLabCommit GitLab CI + conftest From the Open Policy Agent team Evaluates single or combined files Support for policy namespaces conftest-policy-validation: stage: test image: openpolicyagent/conftest:latest script: - echo "validating k8s-manifests/ against k8s-policies/" - /conftest test k8s-manifests --policy k8s-policies

17 #GitLabCommit Concat policies and manifests Validate to find violations Package and export workflows GitLab CI + kpt kpt-prep-manifests: stage: test image: gcr.io/kpt-dev/kpt:v0.4.0 script: - /kpt fn source k8s-manifests/ k8s-policies/ > kpt-manifests.yaml kpt-validate-manifests: stage: test image: gcr.io/kpt-functions/gatekeeper-validate script: - /app/gatekeeper_validate --input kpt-manifests.yaml

18 #GitLabCommit Infrastructure repos and clusters Infra Repo Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Validation workflow

19 #GitLabCommit App repos and clusters App Repo Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Validation workflow Infra Repo

20 #GitLabCommit Working Example

21 #GitLabCommit #GitLabCommit Modern CI/CD Prescriptive approach for software delivery Platform admins create and update best practices App developers iterate independently Infra teams implement policy across platform Devs Ops Infra Dev Prod

22 #GitLabCommit Devs Ops Infra Continuous Integration Container Registry Application Config Continuous Deployment Infrastructure Config & Policy Kubernetes App Repo Shared Config Infra Repo Env Repo CI/CD Workflow

23 #GitLabCommit Devs Ops Infra Continuous Integration Container Registry Application Config Continuous Deployment Infrastructure Config & Policy Kubernetes App Repo Shared Config Infra Repo Env Repo CI/CD Workflow + Policy Enforcement

24 #GitLabCommit #GitLabCommit Configuration Terraform + Cloud Build to deploy solution GitLab deployed on GKE Dev, Staging, and multi-region Prod clusters GitOps for app and config delivery Starter repos for CI, CD, and Kustomize Example repo for Go application Devs Ops Infra Dev Prod bit.ly/modern-cicd-repo bit.ly/modern-cicd-guide

25 #GitLabCommit Getting Started

26 #GitLabCommit #GitLabCommit Authoring – Starter The Rego Playground provides a solid editor to get started with OPA and share policies. Try it out at play.openpolicyagent.org

27 #GitLabCommit #GitLabCommit Authoring – IDE OPA has integrations for several editors and IDEs → VS Code, Sublime Text, IntelliJ, Emacs, VIM. Integrations differ depending on the tools but many offer syntax highlighting, query eval, policy coverage, and more.

28 #GitLabCommit Testing – Dev loop Use skaffold configs to add custom tests Build images with conftest and script execution apiVersion: skaffold/v2beta13 kind: Config metadata: name: test build: artifacts: - image: validate-image context: .. docker: dockerfile: validate/Dockerfile test: - image: validate-image custom: - command: docker run --entrypoint bash $IMAGE ./validate.sh

29 #GitLabCommit Donʼt forget to test against Gatekeeper itself Especially to understand the failure scenarios Testing – Gatekeeper

30 #GitLabCommit Example policies ● open-policy-agent/gatekeeper ○ Getting started examples ○ Includes required labels, allowed repos, container limits, unique service selector ● open-policy-agent/gatekeeper-library ○ Community-owned library of policies ○ General examples (see above) plus others (https-only, disallowed tags, unique ingress) ○ Pod Security Policies implemented as Constraints and ConstraintTemplates

31 #GitLabCommit Example policies ● crcsmnky/gatekeeper-istio ○ Gatekeeper policies for Istio resources ○ Require mTLS activation, disallow all inbound sources, port-naming conventions ● GoogleCloudPlatform/acm-policy-controller-library ○ Anthos Service Mesh (Istio) policies, for use with Anthos Config Management (GitOps) Policy Controller (Gatekeeper) ○ Multiple authz controls, peer authentication, mTLS traffic policies

32 #GitLabCommit Closing Thoughts

33 #GitLabCommit Complexity ● Scoping enforcement to the correct ○ Resources ○ Namespaces ○ Labels, etc. ● Understanding “fail open” vs “fail closed” ● Synchronizing resources to Gatekeeper for policy inputs ● RBAC for administering Constraints and ConstraintTemplates

34 #GitLabCommit Defense in depth Policy enforcement Signed images Vulnerability scanning Signature validation Audit controls Encrypted images

35 #GitLabCommit

36 #GitLabCommit Thank you! Questions or comments? Find me at @crcsmnky on Twitter.