Living on the Edge @ Kubernetes London

Living on the Edge @ Kubernetes London

How to connect edge devices using Mutual TLS (mTLS) to a Kubernetes cluster.

C770b64f01d6b9360b59e8470c2754f4?s=128

Viktor Petersson

September 24, 2019
Tweet

Transcript

  1. LIVING ON THE EDGE When edge devices meet Kubernetes wott.io

    @vpetersson
  2. @vpetersson $ whoami wott.io

  3. WHAT'S WOTT? @vpetersson wott.io Security tool for developers Cryptographic identity

    (x509) Ongoing security audit of fleet A security dashboard
  4. THE SCENARIO Kubernetes cluster
 (mittenetes?) @vpetersson Smart oven mittens
 (mittenet?)

    wott.io
  5. THE SCENARIO We got linux edge devices (i.e. our "smart

    oven mitten") That needs to talk to our Kubernetes cluster ...securely ...live on stage @vpetersson wott.io
  6. THE SCENARIO Nginx Ingress Controller Pod Pod Smart Mitten Smart

    Mitten @vpetersson wott.io
  7. CONCEPTS OVERVIEW TLS vs Mutual TLS Zero Trust Networking vs

    Perimeter based security Public Key Infrastructure (PKI) and root of trust @vpetersson wott.io
  8. HOW TLS WORKS @vpetersson wott.io

  9. HOW MUTUAL TLS WORKS @vpetersson wott.io

  10. PERIMETER SECURITY VS ZERO TRUST Perimeter based security Defend the

    moat with firewalls Example: Infra from the 90s Zero Trust Networking Assume everything is hostile (even internal traffic) Encrypt and verify everything Example: Istio and BeyondCorp Read "Zero Trust Networks" by Doug Barth and Evan Gilman for more details @vpetersson wott.io
  11. PUBLIC KEY INFRASTRUCTURE (PKI) Where you get your SSL certificates

    Let's Encrypt, Commodo, Verisign Browsers trust "public CAs"* The CA is your "root of trust" Don't use a "public CA" for zero trust networking** @vpetersson wott.io * This is a bit complicated and out of scope ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman
  12. TODAY'S GOAL Use Mutual TLS (mTLS) for transport layer Rotate

    the keys periodically in an automated fashion* Access control using cryptographic identity No use of API keys or username+password @vpetersson wott.io * Performed automatically by the WoTT Agent
  13. KUBERNETES SETUP Preparation before the talk Spun up a GKE

    k8s cluster Nginx Ingress Controller SSL cert from Let's Encrypt with Certbot (from Jetstack) @vpetersson wott.io
  14. APP OVERVIEW Python Flask app Uses headers from Nginx for

    access control This is a PoC with known attack vectors, so don't blindly use this in production as-is @vpetersson wott.io
  15. APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
 DON'T USE IN PRODUCTION
  16. ACCESS CONTROL OVERVIEW @vpetersson wott.io 1 3

  17. DEPLOYING OUR APP $ kubectl create -f k8s/deployment.yaml $ kubectl

    create -f k8s/service.yaml $ kubectl create -f k8s/ingress.yaml @vpetersson wott.io
  18. CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com No client certificate

    provided. Access denied. @vpetersson wott.io
  19. ACCESS CONTROL OVERVIEW @vpetersson wott.io

  20. PREPARING NGINX $ curl -s https://api.wott.io/v0.2/ca-bundle | \ jq -r

    .ca_bundle > wott-ca.crt $ kubectl create secret generic wott-ca \ -n k8slon \ --from-file=ca.crt=wott-ca.crt $ kubectl get secrets -n k8slon NAME TYPE DATA AGE [...] wott-ca Opaque 1 27h @vpetersson wott.io
  21. INGRESS MAGIC apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: [...] nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream:

    "true" nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" [...] spec: [...] @vpetersson wott.io
  22. DEPLOYING OUR APP $ kubectl apply -f k8s/ingress-mtls.yaml @vpetersson wott.io

  23. CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com <html> <head><title>400 No

    required SSL certificate was sent</title></ head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certificate was sent</center> <hr><center>openresty/1.15.8.1</center> </body> </html> @vpetersson wott.io
  24. ACCESS CONTROL OVERVIEW @vpetersson wott.io

  25. NGINX CONFIGURATION @vpetersson wott.io That gives us these HTTP headers

    to consume in the appserver: HTTP_SSL_CLIENT_VERIFY HTTP_SSL_CLIENT_SUBJECT_DN
  26. APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is

    that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT
 DON'T USE IN PRODUCTION
  27. ONTO THE EDGE @vpetersson wott.io

  28. INSTALL THE WOTT AGENT $ sudo apt-get install -y curl

    && \ sudo mkdir -p /opt/wott && \ echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/ wott/config.ini && \ curl -s https://packagecloud.io/install/repositories/wott/agent/ script.deb.sh | sudo bash && sudo apt install -y wott-agent @vpetersson wott.io
  29. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access denied! @vpetersson wott.io
  30. ACCESS CONTROL OVERVIEW @vpetersson wott.io

  31. WHITELIST THE DEVICE @vpetersson wott.io

  32. CURL ALL DA THINGZ $ sudo curl \
 --key /opt/wott/certs/client.key

    \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access granted! @vpetersson wott.io
  33. ACCESS CONTROL OVERVIEW @vpetersson wott.io

  34. CONCLUSION mTLS doesn't have to be scary Easier and more

    secure than passwords The foundation of Zero Trust Networking @vpetersson wott.io
  35. GET IN TOUCH Check out our agent Ping us at

    hello@wott.io or visit wott.io @vpetersson wott.io
  36. REFERENCES https://github.com/vpetersson/k8slon https://wott.io/blog/tutorials/2019/07/18/edge-to-kubernetes https://wott.io/blog/tutorials/2019/07/15/mtls-with-nginx @vpetersson wott.io