Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Living on the Edge @ Kubernetes London

Living on the Edge @ Kubernetes London

How to connect edge devices using Mutual TLS (mTLS) to a Kubernetes cluster.

Viktor Petersson

September 24, 2019

More Decks by Viktor Petersson

Other Decks in Technology


  1. WHAT'S WOTT? @vpetersson wott.io Security tool for developers Cryptographic identity

    (x509) Ongoing security audit of fleet A security dashboard
  2. THE SCENARIO We got linux edge devices (i.e. our "smart

    oven mitten") That needs to talk to our Kubernetes cluster ...securely ...live on stage @vpetersson wott.io
  3. CONCEPTS OVERVIEW TLS vs Mutual TLS Zero Trust Networking vs

    Perimeter based security Public Key Infrastructure (PKI) and root of trust @vpetersson wott.io
  4. PERIMETER SECURITY VS ZERO TRUST Perimeter based security Defend the

    moat with firewalls Example: Infra from the 90s Zero Trust Networking Assume everything is hostile (even internal traffic) Encrypt and verify everything Example: Istio and BeyondCorp Read "Zero Trust Networks" by Doug Barth and Evan Gilman for more details @vpetersson wott.io
  5. PUBLIC KEY INFRASTRUCTURE (PKI) Where you get your SSL certificates

    Let's Encrypt, Commodo, Verisign Browsers trust "public CAs"* The CA is your "root of trust" Don't use a "public CA" for zero trust networking** @vpetersson wott.io * This is a bit complicated and out of scope ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman
  6. TODAY'S GOAL Use Mutual TLS (mTLS) for transport layer Rotate

    the keys periodically in an automated fashion* Access control using cryptographic identity No use of API keys or username+password @vpetersson wott.io * Performed automatically by the WoTT Agent
  7. KUBERNETES SETUP Preparation before the talk Spun up a GKE

    k8s cluster Nginx Ingress Controller SSL cert from Let's Encrypt with Certbot (from Jetstack) @vpetersson wott.io
  8. APP OVERVIEW Python Flask app Uses headers from Nginx for

    access control This is a PoC with known attack vectors, so don't blindly use this in production as-is @vpetersson wott.io
  9. APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
  10. DEPLOYING OUR APP $ kubectl create -f k8s/deployment.yaml $ kubectl

    create -f k8s/service.yaml $ kubectl create -f k8s/ingress.yaml @vpetersson wott.io
  11. PREPARING NGINX $ curl -s https://api.wott.io/v0.2/ca-bundle | \ jq -r

    .ca_bundle > wott-ca.crt $ kubectl create secret generic wott-ca \ -n k8slon \ --from-file=ca.crt=wott-ca.crt $ kubectl get secrets -n k8slon NAME TYPE DATA AGE [...] wott-ca Opaque 1 27h @vpetersson wott.io
  12. INGRESS MAGIC apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: [...] nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream:

    "true" nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" [...] spec: [...] @vpetersson wott.io
  13. CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com <html> <head><title>400 No

    required SSL certificate was sent</title></ head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certificate was sent</center> <hr><center>openresty/</center> </body> </html> @vpetersson wott.io
  14. NGINX CONFIGURATION @vpetersson wott.io That gives us these HTTP headers

    to consume in the appserver: HTTP_SSL_CLIENT_VERIFY HTTP_SSL_CLIENT_SUBJECT_DN
  15. APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
  16. INSTALL THE WOTT AGENT $ sudo apt-get install -y curl

    && \ sudo mkdir -p /opt/wott && \ echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/ wott/config.ini && \ curl -s https://packagecloud.io/install/repositories/wott/agent/ script.deb.sh | sudo bash && sudo apt install -y wott-agent @vpetersson wott.io
  17. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access denied! @vpetersson wott.io
  18. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access granted! @vpetersson wott.io
  19. CONCLUSION mTLS doesn't have to be scary Easier and more

    secure than passwords The foundation of Zero Trust Networking @vpetersson wott.io