Living on the Edge @ Kubernetes London

Living on the Edge @ Kubernetes London

How to connect edge devices using Mutual TLS (mTLS) to a Kubernetes cluster.


Viktor Petersson

September 24, 2019


  1. LIVING ON THE EDGE When edge devices meet Kubernetes

  2. @vpetersson $ whoami

  3. WHAT'S WOTT? @vpetersson Security tool for developers Cryptographic identity

    (x509) Ongoing security audit of fleet A security dashboard
  4. THE SCENARIO Kubernetes cluster
 (mittenetes?) @vpetersson Smart oven mittens
  5. THE SCENARIO We got linux edge devices (i.e. our "smart

    oven mitten") That needs to talk to our Kubernetes cluster ...securely on stage @vpetersson
  6. THE SCENARIO Nginx Ingress Controller Pod Pod Smart Mitten Smart

    Mitten @vpetersson
  7. CONCEPTS OVERVIEW TLS vs Mutual TLS Zero Trust Networking vs

    Perimeter based security Public Key Infrastructure (PKI) and root of trust @vpetersson
  8. HOW TLS WORKS @vpetersson

  9. HOW MUTUAL TLS WORKS @vpetersson

  10. PERIMETER SECURITY VS ZERO TRUST Perimeter based security Defend the

    moat with firewalls Example: Infra from the 90s Zero Trust Networking Assume everything is hostile (even internal traffic) Encrypt and verify everything Example: Istio and BeyondCorp Read "Zero Trust Networks" by Doug Barth and Evan Gilman for more details @vpetersson
  11. PUBLIC KEY INFRASTRUCTURE (PKI) Where you get your SSL certificates

    Let's Encrypt, Commodo, Verisign Browsers trust "public CAs"* The CA is your "root of trust" Don't use a "public CA" for zero trust networking** @vpetersson * This is a bit complicated and out of scope ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman
  12. TODAY'S GOAL Use Mutual TLS (mTLS) for transport layer Rotate

    the keys periodically in an automated fashion* Access control using cryptographic identity No use of API keys or username+password @vpetersson * Performed automatically by the WoTT Agent
  13. KUBERNETES SETUP Preparation before the talk Spun up a GKE

    k8s cluster Nginx Ingress Controller SSL cert from Let's Encrypt with Certbot (from Jetstack) @vpetersson
  14. APP OVERVIEW Python Flask app Uses headers from Nginx for

    access control This is a PoC with known attack vectors, so don't blindly use this in production as-is @vpetersson
  15. APP OVERVIEW @vpetersson Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
  16. ACCESS CONTROL OVERVIEW @vpetersson 1 3

  17. DEPLOYING OUR APP $ kubectl create -f k8s/deployment.yaml $ kubectl

    create -f k8s/service.yaml $ kubectl create -f k8s/ingress.yaml @vpetersson
  18. CURL ALL DA THINGZ $ curl No client certificate

    provided. Access denied. @vpetersson

  20. PREPARING NGINX $ curl -s | \ jq -r

    .ca_bundle > wott-ca.crt $ kubectl create secret generic wott-ca \ -n k8slon \ --from-file=ca.crt=wott-ca.crt $ kubectl get secrets -n k8slon NAME TYPE DATA AGE [...] wott-ca Opaque 1 27h @vpetersson
  21. INGRESS MAGIC apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: [...]

    "true" k8slon/wott-ca "on" [...] spec: [...] @vpetersson
  22. DEPLOYING OUR APP $ kubectl apply -f k8s/ingress-mtls.yaml @vpetersson

  23. CURL ALL DA THINGZ $ curl <html> <head><title>400 No

    required SSL certificate was sent</title></ head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certificate was sent</center> <hr><center>openresty/</center> </body> </html> @vpetersson

  25. NGINX CONFIGURATION @vpetersson That gives us these HTTP headers

    to consume in the appserver: HTTP_SSL_CLIENT_VERIFY HTTP_SSL_CLIENT_SUBJECT_DN
  26. APP OVERVIEW @vpetersson Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
  27. ONTO THE EDGE @vpetersson

  28. INSTALL THE WOTT AGENT $ sudo apt-get install -y curl

    && \ sudo mkdir -p /opt/wott && \ echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/ wott/config.ini && \ curl -s | sudo bash && sudo apt install -y wott-agent @vpetersson
  29. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ Access denied! @vpetersson

  31. WHITELIST THE DEVICE @vpetersson

  32. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ Access granted! @vpetersson

  34. CONCLUSION mTLS doesn't have to be scary Easier and more

    secure than passwords The foundation of Zero Trust Networking @vpetersson
  35. GET IN TOUCH Check out our agent Ping us at or visit @vpetersson
  36. REFERENCES @vpetersson