Bringing Kubernetes Policy Enforcement to GitLab

Open Policy Agent’s Gatekeeper provides a policy-based approach to manage admission control and identify policy violations, stopping bad things from happening to production Kubernetes deployments. But those checks only happen at deploy time - what if we could move that process back into Gitlab’s workflow? In this session we’ll review the basics of implementing policy controls using OPA Gatekeeper, and go in-depth on how to bring those policy checks back into the branch-commit-merge process. We’ll demonstrate the different approaches to integrating OPA Gatekeeper with Gitlab CI, bringing policy enforcement to the left and identifying policy violations much earlier.

Transcript

1. 1 #GitLabCommit Bringing Kubernetes Policy Enforcement to GitLab
2. 2 #GitLabCommit Sandeep Parikh DevRel Engineer, Google Cloud @crcsmnky @crcsmnky
3. 3 #GitLabCommit Policies and Enforcement
4. 4 #GitLabCommit Policies Definitions Policy management Policy enforcement Rules that tell us how we can configure a resource The practice of developing, deploying, and applying rules The scope and actions taken, based on defined rules Constraints & Templates Open Policy Agent OPA Gatekeeper
5. 5 #GitLabCommit Open Policy Agent Decouple policy decisions from services to achieve unified control across the entire stack. Unified Express policies in a high-level declarative language that promotes safe, fine-grained logic. Declarative Leverage arbitrary external data in policies to ensure that important requirements are enforced. Context Aware
6. 6 #GitLabCommit #GitLabCommit Gatekeeper OPA Gatekeeper brings Open Policy Agent to Kubernetes as an admission controller Gatekeeper turns OPA policies into Kubernetes custom resources Resources are managed using the standard Kubernetes Resource Model (KRM) Gatekeeper Kubernetes kubectl AdmissionReview (request) AdmissionReview (response)
7. 7 #GitLabCommit Templates apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: k8sblocknodeport spec: crd: spec: names: kind: K8sBlockNodePort targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sblocknodeport violation[{"msg": msg}] { input.review.kind.kind == "Service" input.review.object.spec.type == "NodePort" msg := "Cannot create service of type NodePort" } Policies are written in Rego and packaged as parameterized ConstraintTemplate objects.
8. 8 #GitLabCommit apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: k8sblocknodeport spec: crd: spec: names: kind: K8sBlockNodePort targets: - target: admission.k8s.gatekeeper.sh rego: | package k8sblocknodeport violation[{"msg": msg}] { input.review.kind.kind == "Service" input.review.object.spec.type == "NodePort" msg := "Cannot create service of type NodePort" } Templates Policies are written in Rego and packaged as parameterized ConstraintTemplate objects. The ConstraintTemplate extends Gatekeeper by adding a new policy that can be invoked via a new CR.
9. 9 #GitLabCommit Constraints apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Service"] --- apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007 Constraints are instantiations of a ConstraintTemplate CR and can be optionally scoped to specific objects and/or namespaces.
10. 10 #GitLabCommit Constraints apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sBlockNodePort metadata: name: block-node-port spec: enforcementAction: deny match: kinds: - apiGroups: [""] kinds: ["Service"] --- apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007 Constraints are instantiations of a ConstraintTemplate CR and can be optionally scoped to specific objects and/or namespaces. When violated, Constraints can either deny admission or allow entry, and audit the violation in the status field.
11. 11 #GitLabCommit #GitLabCommit Enforcement Gatekeeper reviews the request then denies admission or issues warnings, based on violations. But this only happens when resources are deployed. Gatekeeper Kubernetes Evaluate policy for incoming object Reject object as it violates policy apiVersion: v1 kind: Service metadata: name: my-service-disallowed spec: type: NodePort ports: - port: 80 nodePort: 30007
12. 12 #GitLabCommit #GitLabCommit Runtime enforcement If resources violate any policies they will be rejected. But with GitOps, the controller will continually fail* to sync resources with clusters. Kubernetes Repo Gatekeeper GitOps * pending baked-in backoff, depends on your controller, YMMV, etc.
13. 13 #GitLabCommit Workflows
14. 14 #GitLabCommit Validation workflows Commits are pushed MRs are submitted Push Deploy ↺ Enforcement Push Review & Enforcement Deploy Merge
15. 15 #GitLabCommit Validation tools googlecontainertools.github.io/kpt Kpt is an OSS tool for building declarative workflows on top of resource configuration. conftest.dev Conftest is a utility to help you write tests against structured configuration data. $ conftest test deployment.yaml --policy ./policy $ docker run -i gcr.io/kpt-functions/gatekeeper-validate
16. 16 #GitLabCommit GitLab CI + conftest From the Open Policy Agent team Evaluates single or combined files Support for policy namespaces conftest-policy-validation: stage: test image: openpolicyagent/conftest:latest script: - echo "validating k8s-manifests/ against k8s-policies/" - /conftest test k8s-manifests --policy k8s-policies
17. 17 #GitLabCommit Concat policies and manifests Validate to find violations Package and export workflows GitLab CI + kpt kpt-prep-manifests: stage: test image: gcr.io/kpt-dev/kpt:v0.4.0 script: - /kpt fn source k8s-manifests/ k8s-policies/ > kpt-manifests.yaml kpt-validate-manifests: stage: test image: gcr.io/kpt-functions/gatekeeper-validate script: - /app/gatekeeper_validate --input kpt-manifests.yaml
18. 18 #GitLabCommit Infrastructure repos and clusters Infra Repo Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Validation workflow
19. 19 #GitLabCommit App repos and clusters App Repo Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Kubernetes GitOps Gatekeeper Validation workflow Infra Repo
20. 20 #GitLabCommit Working Example
21. 21 #GitLabCommit #GitLabCommit Modern CI/CD Prescriptive approach for software delivery Platform admins create and update best practices App developers iterate independently Infra teams implement policy across platform Devs Ops Infra Dev Prod
22. 22 #GitLabCommit Devs Ops Infra Continuous Integration Container Registry Application Config Continuous Deployment Infrastructure Config & Policy Kubernetes App Repo Shared Config Infra Repo Env Repo CI/CD Workflow
23. 23 #GitLabCommit Devs Ops Infra Continuous Integration Container Registry Application Config Continuous Deployment Infrastructure Config & Policy Kubernetes App Repo Shared Config Infra Repo Env Repo CI/CD Workflow + Policy Enforcement
24. 24 #GitLabCommit #GitLabCommit Configuration Terraform + Cloud Build to deploy solution GitLab deployed on GKE Dev, Staging, and multi-region Prod clusters GitOps for app and config delivery Starter repos for CI, CD, and Kustomize Example repo for Go application Devs Ops Infra Dev Prod bit.ly/modern-cicd-repo bit.ly/modern-cicd-guide
25. 25 #GitLabCommit Getting Started
26. 26 #GitLabCommit #GitLabCommit Authoring – Starter The Rego Playground provides a solid editor to get started with OPA and share policies. Try it out at play.openpolicyagent.org
27. 27 #GitLabCommit #GitLabCommit Authoring – IDE OPA has integrations for several editors and IDEs → VS Code, Sublime Text, IntelliJ, Emacs, VIM. Integrations differ depending on the tools but many offer syntax highlighting, query eval, policy coverage, and more.
28. 28 #GitLabCommit Testing – Dev loop Use skaffold configs to add custom tests Build images with conftest and script execution apiVersion: skaffold/v2beta13 kind: Config metadata: name: test build: artifacts: - image: validate-image context: .. docker: dockerfile: validate/Dockerfile test: - image: validate-image custom: - command: docker run --entrypoint bash $IMAGE ./validate.sh
29. 29 #GitLabCommit Donʼt forget to test against Gatekeeper itself Especially to understand the failure scenarios Testing – Gatekeeper
30. 30 #GitLabCommit Example policies • open-policy-agent/gatekeeper ◦ Getting started examples ◦ Includes required labels, allowed repos, container limits, unique service selector • open-policy-agent/gatekeeper-library ◦ Community-owned library of policies ◦ General examples (see above) plus others (https-only, disallowed tags, unique ingress) ◦ Pod Security Policies implemented as Constraints and ConstraintTemplates
31. 31 #GitLabCommit Example policies • crcsmnky/gatekeeper-istio ◦ Gatekeeper policies for Istio resources ◦ Require mTLS activation, disallow all inbound sources, port-naming conventions • GoogleCloudPlatform/acm-policy-controller-library ◦ Anthos Service Mesh (Istio) policies, for use with Anthos Config Management (GitOps) Policy Controller (Gatekeeper) ◦ Multiple authz controls, peer authentication, mTLS traffic policies
32. 32 #GitLabCommit Closing Thoughts
33. 33 #GitLabCommit Complexity • Scoping enforcement to the correct ◦ Resources ◦ Namespaces ◦ Labels, etc. • Understanding “fail open” vs “fail closed” • Synchronizing resources to Gatekeeper for policy inputs • RBAC for administering Constraints and ConstraintTemplates
34. 34 #GitLabCommit Defense in depth Policy enforcement Signed images Vulnerability scanning Signature validation Audit controls Encrypted images
35. 35 #GitLabCommit
36. 36 #GitLabCommit Thank you! Questions or comments? Find me at @crcsmnky on Twitter.