Slide 1

Slide 1 text

LIVING ON THE EDGE When edge devices meet Kubernetes wott.io @vpetersson

Slide 2

Slide 2 text

@vpetersson $ whoami wott.io

Slide 3

Slide 3 text

WHAT'S WOTT? @vpetersson wott.io Security tool for developers Cryptographic identity (x509) Ongoing security audit of ﬂeet A security dashboard

Slide 4

Slide 4 text

THE SCENARIO Kubernetes cluster  (mittenetes?) @vpetersson Smart oven mittens  (mittenet?) wott.io

Slide 5

Slide 5 text

THE SCENARIO We got linux edge devices (i.e. our "smart oven mitten") That needs to talk to our Kubernetes cluster ...securely ...live on stage @vpetersson wott.io

Slide 6

Slide 6 text

THE SCENARIO Nginx Ingress Controller Pod Pod Smart Mitten Smart Mitten @vpetersson wott.io

Slide 7

Slide 7 text

CONCEPTS OVERVIEW TLS vs Mutual TLS Zero Trust Networking vs Perimeter based security Public Key Infrastructure (PKI) and root of trust @vpetersson wott.io

Slide 8

Slide 8 text

HOW TLS WORKS @vpetersson wott.io

Slide 9

Slide 9 text

HOW MUTUAL TLS WORKS @vpetersson wott.io

Slide 10

Slide 10 text

PERIMETER SECURITY VS ZERO TRUST Perimeter based security Defend the moat with ﬁrewalls Example: Infra from the 90s Zero Trust Networking Assume everything is hostile (even internal trafﬁc) Encrypt and verify everything Example: Istio and BeyondCorp Read "Zero Trust Networks" by Doug Barth and Evan Gilman for more details @vpetersson wott.io

Slide 11

Slide 11 text

PUBLIC KEY INFRASTRUCTURE (PKI) Where you get your SSL certiﬁcates Let's Encrypt, Commodo, Verisign Browsers trust "public CAs"* The CA is your "root of trust" Don't use a "public CA" for zero trust networking** @vpetersson wott.io * This is a bit complicated and out of scope ** Recommendation from "Zero trust Network" by Doug Barth and Evan Gilman

Slide 12

Slide 12 text

TODAY'S GOAL Use Mutual TLS (mTLS) for transport layer Rotate the keys periodically in an automated fashion* Access control using cryptographic identity No use of API keys or username+password @vpetersson wott.io * Performed automatically by the WoTT Agent

Slide 13

Slide 13 text

KUBERNETES SETUP Preparation before the talk Spun up a GKE k8s cluster Nginx Ingress Controller SSL cert from Let's Encrypt with Certbot (from Jetstack) @vpetersson wott.io

Slide 14

Slide 14 text

APP OVERVIEW Python Flask app Uses headers from Nginx for access control This is a PoC with known attack vectors, so don't blindly use this in production as-is @vpetersson wott.io

Slide 15

Slide 15 text

APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT  DON'T USE IN PRODUCTION

Slide 16

Slide 16 text

ACCESS CONTROL OVERVIEW @vpetersson wott.io 1 3

Slide 17

Slide 17 text

DEPLOYING OUR APP $ kubectl create -f k8s/deployment.yaml $ kubectl create -f k8s/service.yaml $ kubectl create -f k8s/ingress.yaml @vpetersson wott.io

Slide 18

Slide 18 text

CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com No client certiﬁcate provided. Access denied. @vpetersson wott.io

Slide 19

Slide 19 text

ACCESS CONTROL OVERVIEW @vpetersson wott.io

Slide 20

Slide 20 text

PREPARING NGINX $ curl -s https://api.wott.io/v0.2/ca-bundle | \ jq -r .ca_bundle > wott-ca.crt $ kubectl create secret generic wott-ca \ -n k8slon \ --from-ﬁle=ca.crt=wott-ca.crt $ kubectl get secrets -n k8slon NAME TYPE DATA AGE [...] wott-ca Opaque 1 27h @vpetersson wott.io

Slide 21

Slide 21 text

INGRESS MAGIC apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: [...] nginx.ingress.kubernetes.io/auth-tls-pass-certiﬁcate-to-upstream: "true" nginx.ingress.kubernetes.io/auth-tls-secret: k8slon/wott-ca nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" [...] spec: [...] @vpetersson wott.io

Slide 22

Slide 22 text

DEPLOYING OUR APP $ kubectl apply -f k8s/ingress-mtls.yaml @vpetersson wott.io

Slide 23

Slide 23 text

CURL ALL DA THINGZ $ curl https://k8slon.vpetersson.com 400 No required SSL certiﬁcate was sent

400 Bad Request

No required SSL certiﬁcate was sent

openresty/1.15.8.1 @vpetersson wott.io

Slide 24

Slide 24 text

ACCESS CONTROL OVERVIEW @vpetersson wott.io

Slide 25

Slide 25 text

NGINX CONFIGURATION @vpetersson wott.io That gives us these HTTP headers to consume in the appserver: HTTP_SSL_CLIENT_VERIFY HTTP_SSL_CLIENT_SUBJECT_DN

Slide 26

Slide 26 text

APP OVERVIEW @vpetersson wott.io Did we receive a cert? Is that cert in the whitelist? Generate whitelist 1 2 3 THIS IS A PROOF OF CONCEPT  DON'T USE IN PRODUCTION

Slide 27

Slide 27 text

ONTO THE EDGE @vpetersson wott.io

Slide 28

Slide 28 text

INSTALL THE WOTT AGENT $ sudo apt-get install -y curl && \ sudo mkdir -p /opt/wott && \ echo -e "[DEFAULT]\\nenroll_token = abc123" | sudo tee -a /opt/ wott/conﬁg.ini && \ curl -s https://packagecloud.io/install/repositories/wott/agent/ script.deb.sh | sudo bash && sudo apt install -y wott-agent @vpetersson wott.io

Slide 29

Slide 29 text

CURL ALL DA THINGZ $ sudo curl \  --key /opt/wott/certs/client.key \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access denied! @vpetersson wott.io

Slide 30

Slide 30 text

ACCESS CONTROL OVERVIEW @vpetersson wott.io

Slide 31

Slide 31 text

WHITELIST THE DEVICE @vpetersson wott.io

Slide 32

Slide 32 text

CURL ALL DA THINGZ $ sudo curl \  --key /opt/wott/certs/client.key \ --cert /opt/wott/certs/client.crt \ https://k8slon.vpetersson.com Access granted! @vpetersson wott.io

Slide 33

Slide 33 text

ACCESS CONTROL OVERVIEW @vpetersson wott.io

Slide 34

Slide 34 text

CONCLUSION mTLS doesn't have to be scary Easier and more secure than passwords The foundation of Zero Trust Networking @vpetersson wott.io

Slide 35

Slide 35 text

GET IN TOUCH Check out our agent Ping us at [email protected] or visit wott.io @vpetersson wott.io

Slide 36

Slide 36 text

REFERENCES https://github.com/vpetersson/k8slon https://wott.io/blog/tutorials/2019/07/18/edge-to-kubernetes https://wott.io/blog/tutorials/2019/07/15/mtls-with-nginx @vpetersson wott.io