Unit Testing Your Kubernetes Configuration with Open Policy Agent

Talk from KubeCon EU 2019, focused on using Open Policy Agent to write tests for structured configuration.

Transcript

1. Using Open Policy Agent
   Gareth Rushgrove | Director, Product Management | Snyk
   snyk.io
   Unit testing your
   Kubernetes configuration
   May, 2019
   

   View Slide

2. snyk.io
   

   View Slide

3. snyk.io
   - A quick introduction to Open Policy Agent
   - Shift-left testing
   - Introducing conftest
   - Rego as a programming language
   - Portability between different Kubernetes solutions
   - Not just Kubernetes
   Agenda
   

   View Slide

4. Open Policy Agent
   snyk.io
   A policy enforcement engine for configuration
   

   View Slide

5. What is Open Policy Agent?
   

   View Slide

6. Policy
   Data
   OPA
   Service
   How Open Policy Agent works
   

   View Slide

7. A growing ecosystem
   

   View Slide

8. View Slide

9. Shift left
   snyk.io
   Faster feedback
   

   View Slide

10. Shift-left testing is an approach to software testing and
    system testing in which testing is performed earlier in the
    lifecycle (i.e., moved left on the project timeline).
    Wikipedia
    

    View Slide

11. Development cycle
    Fast Slow Slower
    Cluster
    Local
    development
    Continuous
    integration
    

    View Slide

12. Open Policy Agent is
    normally used here
    Development cycle
    Cluster
    Local
    development
    Continuous
    integration
    

    View Slide

13. Cluster
    What if we could use Open
    Policy Agent here as well?
    Development cycle
    Local
    development
    Continuous
    integration
    

    View Slide

14. “Everything as code”
    Said several times at the CDF event
    

    View Slide

15. “Would you write code
    without tests?”
    

    View Slide

16. Introducing conftest
    snyk.io
    Test your configuration locally, and in CI
    

    View Slide

17. Introducing conftest
    

    View Slide

18. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: hello-kubernetes
    spec:
    replicas: 3
    selector:
    matchLabels:
    app: hello-kubernetes
    template:
    metadata:
    labels:
    app: hello-kubernetes
    spec:
    containers:
    - name: hello-kubernetes
    image: paulbouwer/hello-kubernetes:1.5
    ports:
    Given a deployment
    snyk.io
    

    View Slide

19. package main
    deny[msg] {
    input.kind = "Deployment"
    not input.spec.template.spec.securityContext.runAsNonRoot = true
    msg = "Containers must not run as root"
    }
    deny[msg] {
    input.kind = "Deployment"
    not input.spec.selector.matchLabels.app
    msg = "Containers must provide app label for pod selectors"
    }
    Write your policies
    snyk.io
    

    View Slide

20. deny[msg] {
    input.kind = "Deployment"
    not input.spec.template.spec.securityContext.runAsNonRoot = true
    msg = "Containers must not run as root"
    }
    Explaining what we just wrote
    snyk.io
    We should deny any input for which
    Deployment is the value for kind
    and
    When runAsNonRoot is set to false
    

    View Slide

21. $ conftest test deployment.yaml
    deployment.yaml
    Containers must not run as root
    $ echo $status
    1
    Run tests locally with conftest
    snyk.io
    

    View Slide

22. snyk.io
    Demo
    

    View Slide

23. Rego as a language
    snyk.io
    The usual pros and cons of a DSL
    

    View Slide

24. Good documentation
    

    View Slide

25. The Rego playground
    

    View Slide

26. package main
    test_deployment_without_security_context {
    deny["Containers must not run as root"] with input as {"kind": "Deployment"}
    }
    test_deployment_with_security_context {
    no_violations with input as {"kind": "Deployment", "spec": {
    "selector": { "matchLabels": { "app": "something", "release": "something" }},
    "template": { "spec": { "securityContext": { "runAsNonRoot": true }}}}}
    }
    test_services_not_denied {
    no_violations with input as {"kind": "Service"}
    }
    test_services_issue_warning {
    warn["Services are not allowed"] with input as {"kind": "Service"}
    Built-in testing tools
    

    View Slide

27. $ opa test --verbose .
    data.main.test_deployment_without_security_context: PASS (1.029µs)
    data.main.test_deployment_with_security_context: PASS (1.058µs)
    data.main.test_services_not_denied: PASS (701ns)
    data.main.test_services_issue_warning: PASS (614ns)
    ---------------------------------------------------------------------
    PASS: 4/4
    Open Policy Agent test runner
    snyk.io
    

    View Slide

28. In:path .rego extension:rego
    546 results
    Not much public rego code yet
    

    View Slide

29. Open Policy Agent Bundles
    

    View Slide

30. package main
    # has_field returns whether an object has a field
    has_field(object, field) {
    object[field]
    }
    # False is a tricky special case, as false responses would create an undefined
    # document unless they are explicitly tested for
    has_field(object, field) {
    object[field] == false
    }
    has_field(object, field) = false {
    General helpers
    

    View Slide

31. package main
    empty(value) {
    count(value) == 0
    }
    no_violations {
    empty(deny)
    }
    no_warnings {
    empty(warn)
    }
    Test helpers
    snyk.io
    

    View Slide

32. package kubernetes
    is_service {
    input.kind = "Service"
    }
    is_deployment {
    input.kind = "Deployment"
    }
    Domain specific helpers
    snyk.io
    

    View Slide

33. Reusing OCI registries
    

    View Slide

34. Proposed OCI media types of OPA
    

    View Slide

35. $ ls
    kubernetes.rego
    $ conftest push instrumenta.azurecr.io/kubernetes-helpers
    $ ls policy
    $ conftest pull instrumenta.azurecr.io/kubernetes-helpers:latest
    $ ls policy
    kubernetes.rego
    Using conftest to share policy
    snyk.io
    

    View Slide

36. Bundle Bar
    

    View Slide

37. Portability
    snyk.io
    Helping to move between different Kubernetes tools
    

    View Slide

38. snyk.io
    Demo
    

    View Slide

39. $ kustomize build | conftest test -
    Kustomize
    snyk.io
    

    View Slide

40. $ helm template | conftest test -
    Helm
    snyk.io
    

    View Slide

41. import {Pod} from 'kubernetes-types/core/v1'
    import {ObjectMeta} from 'kubernetes-types/meta/v1'
    import * as yaml from 'js-yaml'
    let metadata: ObjectMeta = {name: 'example', labels: {}}
    let pod: Pod = {
    apiVersion: 'v1',
    kind: 'Pod',
    metadata,
    spec: {
    containers: [
    /* ... */
    ],
    },
    }
    console.log(yaml.safeDump(pod))
    Typescript
    

    View Slide

42. $ npx ts-node pod.ts | conftest test -
    Typescript
    snyk.io
    

    View Slide

43. $ kubectl get all -o json \
    | jq -cj '.items[] | tostring+"\u0000"' \
    | xargs -n1 -0 -I@ bash -c "echo '@' | conftest test -"
    Kubectl (look away now)
    snyk.io
    

    View Slide

44. $ kubectl krew install conftest
    $ kubectl conftest deployment some-deployment
    Kubectl plugin
    snyk.io
    

    View Slide

45. package kubernetes
    deployment "hello-kubernetes": {
    apiVersion: "apps/v1"
    spec: {
    replicas: 3
    template spec containers: [{
    image: "paulbouwer/hello-kubernetes:1.5"
    ports: [{
    containerPort: 8080
    }]
    }]
    }
    }
    Cue
    snyk.io
    

    View Slide

46. package kubernetes
    import "encoding/yaml"
    command test: {
    task conftest: {
    kind: "exec"
    cmd: "conftest test -"
    stdin: yaml.MarshalStream(objects)
    }
    }
    Cue
    snyk.io
    

    View Slide

47. $ cue test
    Containers must not run as root
    --- .
    command "conftest test -" failed: exit status 1
    terminating because of errors
    Cue
    snyk.io
    

    View Slide

48. Not just Kubernetes
    snyk.io
    Lots of other configurations to care about
    

    View Slide

49. apiVersion: kind: extension:yaml
    1,054,453 results
    Lots of Kubernetes-like docs
    

    View Slide

50. in:path .yml language:YAML
    69,822,306 results
    Lots more YAML
    

    View Slide

51. snyk.io
    Demo
    

    View Slide

52. $ terraform plan -out config.tfplan
    $ terraform show -json config.tfplan | configtest test -
    Terraform
    snyk.io
    

    View Slide

53. package main
    blacklist = [
    "google_iam",
    "google_container"
    ]
    deny[msg] {
    check_resources(input.resource_changes, blacklist)
    banned := concat(", ", blacklist)
    msg = sprintf("Terraform plan will change prohibited resources in: %v", [banned])
    }
    # Checks whether the plan will cause resources with certain prefixes to change
    check_resources(resources, disallowed_prefixes) {
    startswith(resources[_].type, disallowed_prefixes[_])
    }
    Terraform
    snyk.io
    

    View Slide

54. service: aws-python-scheduled-cron
    frameworkVersion: ">=1.2.0 <2.0.0"
    provider:
    name: aws
    runtime: python2.7
    tags:
    author: "this field is required"
    functions:
    cron:
    handler: handler.run
    runtime: python2.7
    events:
    - schedule: cron(0/2 * ? * MON-FRI *)
    Serverless framework
    snyk.io
    

    View Slide

55. package main
    deny[msg] {
    input.provider.runtime = "python2.7"
    msg = "Python 2.7 cannot be the default provider runtime"
    }
    runtime[name] {
    input.functions[i].runtime = name
    }
    deny[msg] {
    runtime["python2.7"]
    msg = "Python 2.7 cannot be used as the runtime for functions"
    }
    deny[msg] {
    not has_field(input.provider.tags, "author")
    msg = "Should set provider tags for author"
    }
    Serverless framework
    snyk.io
    

    View Slide

56. version: "3.4"
    services:
    web:
    build: .
    ports:
    - "5000:5000"
    redis:
    image: "redis:latest"
    Docker Compose
    snyk.io
    

    View Slide

57. package main
    version {
    to_number(input.version)
    }
    deny[msg] {
    endswith(input.services[_].image, ":latest")
    msg = "No images tagged latest"
    }
    deny[msg] {
    version < 3.5
    msg = "Must be using at least version 3.5 of the Compose file format"
    }
    Docker Compose
    snyk.io
    

    View Slide

58. Conclusions
    snyk.io
    If all you remember is...
    

    View Slide

59. @jdolitsky @stevelasker @sometorin
    Thanks open source community!
    

    View Slide

60. snyk.io
    - Open Policy Agent is incredibly flexible
    - Check out conftest at github.com/instrumenta/conftest
    - Expect lots more integrations in the future
    - Managing configuration as code needs better tools
    Come talk to Snyk at booth #S41
    Summary
    

    View Slide

61. Questions?
    snyk.io
    And thanks for listening
    

    View Slide