Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Living on the Edge @ Kubernetes London

Living on the Edge @ Kubernetes London

How to connect edge devices using Mutual TLS (mTLS) to a Kubernetes cluster.

Viktor Petersson

September 24, 2019
Tweet

More Decks by Viktor Petersson

Other Decks in Technology

Transcript

  1. LIVING ON THE EDGE
    When edge devices meet Kubernetes
    wott.io @vpetersson

    View full-size slide

  2. @vpetersson
    $ whoami
    wott.io

    View full-size slide

  3. WHAT'S WOTT?
    @vpetersson
    wott.io
    Security tool for developers
    Cryptographic identity (x509)
    Ongoing security audit of fleet
    A security dashboard

    View full-size slide

  4. THE SCENARIO
    Kubernetes cluster

    (mittenetes?)
    @vpetersson
    Smart oven mittens

    (mittenet?)
    wott.io

    View full-size slide

  5. THE SCENARIO
    We got linux edge devices (i.e. our "smart oven mitten")
    That needs to talk to our Kubernetes cluster
    ...securely
    ...live on stage
    @vpetersson
    wott.io

    View full-size slide

  6. THE SCENARIO
    Nginx Ingress
    Controller
    Pod
    Pod
    Smart Mitten
    Smart Mitten
    @vpetersson
    wott.io

    View full-size slide

  7. CONCEPTS OVERVIEW
    TLS vs Mutual TLS
    Zero Trust Networking vs Perimeter based security
    Public Key Infrastructure (PKI) and root of trust
    @vpetersson
    wott.io

    View full-size slide

  8. HOW TLS WORKS
    @vpetersson
    wott.io

    View full-size slide

  9. HOW MUTUAL TLS WORKS
    @vpetersson
    wott.io

    View full-size slide

  10. PERIMETER SECURITY VS ZERO TRUST
    Perimeter based security
    Defend the moat with firewalls
    Example: Infra from the 90s
    Zero Trust Networking
    Assume everything is hostile (even internal traffic)
    Encrypt and verify everything
    Example: Istio and BeyondCorp
    Read "Zero Trust Networks" by Doug Barth and
    Evan Gilman for more details
    @vpetersson
    wott.io

    View full-size slide

  11. PUBLIC KEY INFRASTRUCTURE (PKI)
    Where you get your SSL certificates
    Let's Encrypt, Commodo, Verisign
    Browsers trust "public CAs"*
    The CA is your "root of trust"
    Don't use a "public CA" for zero trust networking**
    @vpetersson
    wott.io * This is a bit complicated and out of scope
    ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman

    View full-size slide

  12. TODAY'S GOAL
    Use Mutual TLS (mTLS) for transport layer
    Rotate the keys periodically in an automated fashion*
    Access control using cryptographic identity
    No use of API keys or username+password
    @vpetersson
    wott.io * Performed automatically by the WoTT Agent

    View full-size slide

  13. KUBERNETES SETUP
    Preparation before the talk
    Spun up a GKE k8s cluster
    Nginx Ingress Controller
    SSL cert from Let's Encrypt with Certbot (from Jetstack)
    @vpetersson
    wott.io

    View full-size slide

  14. APP OVERVIEW
    Python Flask app
    Uses headers from Nginx for access control
    This is a PoC with known attack vectors, so
    don't blindly use this in production as-is
    @vpetersson
    wott.io

    View full-size slide

  15. APP OVERVIEW
    @vpetersson
    wott.io
    Did we receive a cert? Is that cert in the whitelist?
    Generate whitelist
    1
    2
    3
    THIS IS A PROOF OF CONCEPT

    DON'T USE IN PRODUCTION

    View full-size slide

  16. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    1 3

    View full-size slide

  17. DEPLOYING OUR APP
    $ kubectl create -f k8s/deployment.yaml
    $ kubectl create -f k8s/service.yaml
    $ kubectl create -f k8s/ingress.yaml
    @vpetersson
    wott.io

    View full-size slide

  18. CURL ALL DA THINGZ
    $ curl https://k8slon.vpetersson.com
    No client certificate provided. Access denied.
    @vpetersson
    wott.io

    View full-size slide

  19. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io

    View full-size slide

  20. PREPARING NGINX
    $ curl -s https://api.wott.io/v0.2/ca-bundle | \
    jq -r .ca_bundle > wott-ca.crt
    $ kubectl create secret generic wott-ca \
    -n k8slon \
    --from-file=ca.crt=wott-ca.crt
    $ kubectl get secrets -n k8slon
    NAME TYPE DATA AGE
    [...]
    wott-ca Opaque 1 27h
    @vpetersson
    wott.io

    View full-size slide

  21. INGRESS MAGIC
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    annotations:
    [...]
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    [...]
    spec:
    [...]
    @vpetersson
    wott.io

    View full-size slide

  22. DEPLOYING OUR APP
    $ kubectl apply -f k8s/ingress-mtls.yaml
    @vpetersson
    wott.io

    View full-size slide

  23. CURL ALL DA THINGZ
    $ curl https://k8slon.vpetersson.com

    400 No required SSL certificate was sent
    head>

    400 Bad Request
    No required SSL certificate was sent
    openresty/1.15.8.1


    @vpetersson
    wott.io

    View full-size slide

  24. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io

    View full-size slide

  25. NGINX CONFIGURATION
    @vpetersson
    wott.io
    That gives us these HTTP headers to consume in the appserver:
    HTTP_SSL_CLIENT_VERIFY
    HTTP_SSL_CLIENT_SUBJECT_DN

    View full-size slide

  26. APP OVERVIEW
    @vpetersson
    wott.io
    Did we receive a cert? Is that cert in the whitelist?
    Generate whitelist
    1
    2
    3
    THIS IS A PROOF OF CONCEPT

    DON'T USE IN PRODUCTION

    View full-size slide

  27. ONTO THE EDGE
    @vpetersson
    wott.io

    View full-size slide

  28. INSTALL THE WOTT AGENT
    $ sudo apt-get install -y curl && \
    sudo mkdir -p /opt/wott && \
    echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/
    wott/config.ini && \
    curl -s https://packagecloud.io/install/repositories/wott/agent/
    script.deb.sh | sudo bash && sudo apt install -y wott-agent
    @vpetersson
    wott.io

    View full-size slide

  29. CURL ALL DA THINGZ
    $ sudo curl \

    --key /opt/wott/certs/client.key \
    --cert /opt/wott/certs/client.crt \
    https://k8slon.vpetersson.com
    Access denied!
    @vpetersson
    wott.io

    View full-size slide

  30. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io

    View full-size slide

  31. WHITELIST THE DEVICE
    @vpetersson
    wott.io

    View full-size slide

  32. CURL ALL DA THINGZ
    $ sudo curl \

    --key /opt/wott/certs/client.key \
    --cert /opt/wott/certs/client.crt \
    https://k8slon.vpetersson.com
    Access granted!
    @vpetersson
    wott.io

    View full-size slide

  33. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io

    View full-size slide

  34. CONCLUSION
    mTLS doesn't have to be scary
    Easier and more secure than passwords
    The foundation of Zero Trust Networking
    @vpetersson
    wott.io

    View full-size slide

  35. GET IN TOUCH
    Check out our agent
    Ping us at [email protected] or visit wott.io
    @vpetersson
    wott.io

    View full-size slide

  36. REFERENCES
    https://github.com/vpetersson/k8slon
    https://wott.io/blog/tutorials/2019/07/18/edge-to-kubernetes
    https://wott.io/blog/tutorials/2019/07/15/mtls-with-nginx
    @vpetersson
    wott.io

    View full-size slide