Living on the Edge @ Kubernetes London

How to connect edge devices using Mutual TLS (mTLS) to a Kubernetes cluster.

Transcript

1. LIVING ON THE EDGE
   When edge devices meet Kubernetes
   wott.io @vpetersson
   

   View Slide

2. @vpetersson
   $ whoami
   wott.io
   

   View Slide

3. WHAT'S WOTT?
   @vpetersson
   wott.io
   Security tool for developers
   Cryptographic identity (x509)
   Ongoing security audit of fleet
   A security dashboard
   

   View Slide

4. THE SCENARIO
   Kubernetes cluster

   (mittenetes?)
   @vpetersson
   Smart oven mittens

   (mittenet?)
   wott.io
   

   View Slide

5. THE SCENARIO
   We got linux edge devices (i.e. our "smart oven mitten")
   That needs to talk to our Kubernetes cluster
   ...securely
   ...live on stage
   @vpetersson
   wott.io
   

   View Slide

6. THE SCENARIO
   Nginx Ingress
   Controller
   Pod
   Pod
   Smart Mitten
   Smart Mitten
   @vpetersson
   wott.io
   

   View Slide

7. CONCEPTS OVERVIEW
   TLS vs Mutual TLS
   Zero Trust Networking vs Perimeter based security
   Public Key Infrastructure (PKI) and root of trust
   @vpetersson
   wott.io
   

   View Slide

8. HOW TLS WORKS
   @vpetersson
   wott.io
   

   View Slide

9. HOW MUTUAL TLS WORKS
   @vpetersson
   wott.io
   

   View Slide

10. PERIMETER SECURITY VS ZERO TRUST
    Perimeter based security
    Defend the moat with firewalls
    Example: Infra from the 90s
    Zero Trust Networking
    Assume everything is hostile (even internal traffic)
    Encrypt and verify everything
    Example: Istio and BeyondCorp
    Read "Zero Trust Networks" by Doug Barth and
    Evan Gilman for more details
    @vpetersson
    wott.io
    

    View Slide

11. PUBLIC KEY INFRASTRUCTURE (PKI)
    Where you get your SSL certificates
    Let's Encrypt, Commodo, Verisign
    Browsers trust "public CAs"*
    The CA is your "root of trust"
    Don't use a "public CA" for zero trust networking**
    @vpetersson
    wott.io * This is a bit complicated and out of scope
    ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman
    

    View Slide

12. TODAY'S GOAL
    Use Mutual TLS (mTLS) for transport layer
    Rotate the keys periodically in an automated fashion*
    Access control using cryptographic identity
    No use of API keys or username+password
    @vpetersson
    wott.io * Performed automatically by the WoTT Agent
    

    View Slide

13. KUBERNETES SETUP
    Preparation before the talk
    Spun up a GKE k8s cluster
    Nginx Ingress Controller
    SSL cert from Let's Encrypt with Certbot (from Jetstack)
    @vpetersson
    wott.io
    

    View Slide

14. APP OVERVIEW
    Python Flask app
    Uses headers from Nginx for access control
    This is a PoC with known attack vectors, so
    don't blindly use this in production as-is
    @vpetersson
    wott.io
    

    View Slide

15. APP OVERVIEW
    @vpetersson
    wott.io
    Did we receive a cert? Is that cert in the whitelist?
    Generate whitelist
    1
    2
    3
    THIS IS A PROOF OF CONCEPT

    DON'T USE IN PRODUCTION
    

    View Slide

16. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    1 3
    

    View Slide

17. DEPLOYING OUR APP
    $ kubectl create -f k8s/deployment.yaml
    $ kubectl create -f k8s/service.yaml
    $ kubectl create -f k8s/ingress.yaml
    @vpetersson
    wott.io
    

    View Slide

18. CURL ALL DA THINGZ
    $ curl https://k8slon.vpetersson.com
    No client certificate provided. Access denied.
    @vpetersson
    wott.io
    

    View Slide

19. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    

    View Slide

20. PREPARING NGINX
    $ curl -s https://api.wott.io/v0.2/ca-bundle | \
    jq -r .ca_bundle > wott-ca.crt
    $ kubectl create secret generic wott-ca \
    -n k8slon \
    --from-file=ca.crt=wott-ca.crt
    $ kubectl get secrets -n k8slon
    NAME TYPE DATA AGE
    [...]
    wott-ca Opaque 1 27h
    @vpetersson
    wott.io
    

    View Slide

21. INGRESS MAGIC
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    annotations:
    [...]
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    [...]
    spec:
    [...]
    @vpetersson
    wott.io
    

    View Slide

22. DEPLOYING OUR APP
    $ kubectl apply -f k8s/ingress-mtls.yaml
    @vpetersson
    wott.io
    

    View Slide

23. CURL ALL DA THINGZ
    $ curl https://k8slon.vpetersson.com
    
    400 No required SSL certificate was sent
    head>
    
    400 Bad Request
    No required SSL certificate was sent
    openresty/1.15.8.1
    
    
    @vpetersson
    wott.io
    

    View Slide

24. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    

    View Slide

25. NGINX CONFIGURATION
    @vpetersson
    wott.io
    That gives us these HTTP headers to consume in the appserver:
    HTTP_SSL_CLIENT_VERIFY
    HTTP_SSL_CLIENT_SUBJECT_DN
    

    View Slide

26. APP OVERVIEW
    @vpetersson
    wott.io
    Did we receive a cert? Is that cert in the whitelist?
    Generate whitelist
    1
    2
    3
    THIS IS A PROOF OF CONCEPT

    DON'T USE IN PRODUCTION
    

    View Slide

27. ONTO THE EDGE
    @vpetersson
    wott.io
    

    View Slide

28. INSTALL THE WOTT AGENT
    $ sudo apt-get install -y curl && \
    sudo mkdir -p /opt/wott && \
    echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/
    wott/config.ini && \
    curl -s https://packagecloud.io/install/repositories/wott/agent/
    script.deb.sh | sudo bash && sudo apt install -y wott-agent
    @vpetersson
    wott.io
    

    View Slide

29. CURL ALL DA THINGZ
    $ sudo curl \

    --key /opt/wott/certs/client.key \
    --cert /opt/wott/certs/client.crt \
    https://k8slon.vpetersson.com
    Access denied!
    @vpetersson
    wott.io
    

    View Slide

30. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    

    View Slide

31. WHITELIST THE DEVICE
    @vpetersson
    wott.io
    

    View Slide

32. CURL ALL DA THINGZ
    $ sudo curl \

    --key /opt/wott/certs/client.key \
    --cert /opt/wott/certs/client.crt \
    https://k8slon.vpetersson.com
    Access granted!
    @vpetersson
    wott.io
    

    View Slide

33. ACCESS CONTROL OVERVIEW
    @vpetersson
    wott.io
    

    View Slide

34. CONCLUSION
    mTLS doesn't have to be scary
    Easier and more secure than passwords
    The foundation of Zero Trust Networking
    @vpetersson
    wott.io
    

    View Slide

35. GET IN TOUCH
    Check out our agent
    Ping us at hello@wott.io or visit wott.io
    @vpetersson
    wott.io
    

    View Slide

36. REFERENCES
    https://github.com/vpetersson/k8slon
    https://wott.io/blog/tutorials/2019/07/18/edge-to-kubernetes
    https://wott.io/blog/tutorials/2019/07/15/mtls-with-nginx
    @vpetersson
    wott.io
    

    View Slide