Living on the Edge @ Kubernetes London

LIVING ON THE EDGE When edge devices meet Kubernetes wott.io
@vpetersson

@vpetersson $ whoami wott.io

WHAT'S WOTT? @vpetersson wott.io Security tool for developers Cryptographic identity
(x509) Ongoing security audit of ﬂeet A security dashboard

THE SCENARIO Kubernetes cluster  (mittenetes?) @vpetersson Smart oven mittens  (mittenet?)
wott.io

THE SCENARIO We got linux edge devices (i.e. our "smart
oven mitten") That needs to talk to our Kubernetes cluster ...securely ...live on stage @vpetersson wott.io

THE SCENARIO Nginx Ingress Controller Pod Pod Smart Mitten Smart
Mitten @vpetersson wott.io

CONCEPTS OVERVIEW TLS vs Mutual TLS Zero Trust Networking vs
Perimeter based security Public Key Infrastructure (PKI) and root of trust @vpetersson wott.io

HOW TLS WORKS @vpetersson wott.io

HOW MUTUAL TLS WORKS @vpetersson wott.io

PERIMETER SECURITY VS ZERO TRUST Perimeter based security Defend the
moat with ﬁrewalls Example: Infra from the 90s Zero Trust Networking Assume everything is hostile (even internal trafﬁc) Encrypt and verify everything Example: Istio and BeyondCorp Read "Zero Trust Networks" by Doug Barth and Evan Gilman for more details @vpetersson wott.io

PUBLIC KEY INFRASTRUCTURE (PKI) Where you get your SSL certiﬁcates
Let's Encrypt, Commodo, Verisign Browsers trust "public CAs"* The CA is your "root of trust" Don't use a "public CA" for zero trust networking** @vpetersson wott.io * This is a bit complicated and out of scope ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman

TODAY'S GOAL Use Mutual TLS (mTLS) for transport layer Rotate
the keys periodically in an automated fashion* Access control using cryptographic identity No use of API keys or username+password @vpetersson wott.io * Performed automatically by the WoTT Agent

KUBERNETES SETUP Preparation before the talk Spun up a GKE
k8s cluster Nginx Ingress Controller SSL cert from Let's Encrypt with Certbot (from Jetstack) @vpetersson wott.io

APP OVERVIEW Python Flask app Uses headers from Nginx for
access control This is a PoC with known attack vectors, so don't blindly use this in production as-is @vpetersson wott.io

APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is
that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT  DON'T USE IN PRODUCTION

ACCESS CONTROL OVERVIEW @vpetersson wott.io 1 3

DEPLOYING OUR APP $ kubectl create -f k8s/deployment.yaml $ kubectl
create -f k8s/service.yaml $ kubectl create -f k8s/ingress.yaml @vpetersson wott.io

CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com No client certiﬁcate
provided. Access denied. @vpetersson wott.io

ACCESS CONTROL OVERVIEW @vpetersson wott.io

PREPARING NGINX $ curl -s https://api.wott.io/v0.2/ca-bundle | \ jq -r
.ca_bundle > wott-ca.crt $ kubectl create secret generic wott-ca \ -n k8slon \ --from-ﬁle=ca.crt=wott-ca.crt $ kubectl get secrets -n k8slon NAME TYPE DATA AGE [...] wott-ca Opaque 1 27h @vpetersson wott.io

INGRESS MAGIC apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: [...] nginx.ingress.kubernetes.io/auth-tls-pass-certiﬁcate-to-upstream:
"true" nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" [...] spec: [...] @vpetersson wott.io

DEPLOYING OUR APP $ kubectl apply -f k8s/ingress-mtls.yaml @vpetersson wott.io

CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com <html> <head><title>400 No
required SSL certiﬁcate was sent</title></ head> <body> <center><h1>400 Bad Request</h1></center> <center>No required SSL certiﬁcate was sent</center> <hr><center>openresty/1.15.8.1</center> </body> </html> @vpetersson wott.io

NGINX CONFIGURATION @vpetersson wott.io That gives us these HTTP headers
to consume in the appserver: HTTP_SSL_CLIENT_VERIFY HTTP_SSL_CLIENT_SUBJECT_DN

APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is
that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT  DON'T USE IN PRODUCTION

ONTO THE EDGE @vpetersson wott.io

INSTALL THE WOTT AGENT $ sudo apt-get install -y curl
&& \ sudo mkdir -p /opt/wott && \ echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/ wott/conﬁg.ini && \ curl -s https://packagecloud.io/install/repositories/wott/agent/ script.deb.sh | sudo bash && sudo apt install -y wott-agent @vpetersson wott.io

CURL ALL DA THINGZ $ sudo curl \  --key /opt/wott/certs/client.key
\ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access denied! @vpetersson wott.io

WHITELIST THE DEVICE @vpetersson wott.io

CURL ALL DA THINGZ $ sudo curl \  --key /opt/wott/certs/client.key
\ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access granted! @vpetersson wott.io

CONCLUSION mTLS doesn't have to be scary Easier and more
secure than passwords The foundation of Zero Trust Networking @vpetersson wott.io

GET IN TOUCH Check out our agent Ping us at
[email protected] or visit wott.io @vpetersson wott.io

REFERENCES https://github.com/vpetersson/k8slon https://wott.io/blog/tutorials/2019/07/18/edge-to-kubernetes https://wott.io/blog/tutorials/2019/07/15/mtls-with-nginx @vpetersson wott.io

Living on the Edge @ Kubernetes London

Living on the Edge @ Kubernetes London

More Decks by Viktor Petersson

Other Decks in Technology

Featured

Transcript