Kubernetes TLS/Let’s Encrypt

This talk is about the state of Let's Encrypt on Kubernetes and Kubernetes’ Cluster Root CA as of June 2017. It is presented at the Seattle Kubernetes Meetup.

tl;dr: LE solutions on k8s are not mature yet, you should probably still buy a TLS certificate. For internal TLS, look at using a service mesh like Istio or Linkerd.

Transcript

1. Google Cloud Platform logo Kubernetes TLS/Let’s Encrypt Seattle Kubernetes Meetup 2017-06-01 Ahmet Alp Balkan <ahmetb@google.com> Software Engineer, Developer Programs @ Google @ahmetb 1
2. Google Cloud Platform What is TLS Transport-layer* security * OSI Layer 4 (TCP/IP) SSL v2.0 in Netscape Navigator (1995) SSL v3.0 redesign of the protocol (1996) TLS v1.0 IETF standard published (1999) TLS v1.3 still draft as of 2016. Required by HTTP/2 in most implementations. Photo credit: 24hourwebhostingsupport.com 2
3. Google Cloud Platform What is HTTPS TLS TLS TLS TLS TLS HTTP HTTP Photo credit: windchillfactory.com 3
4. Google Cloud Platform Public-key cryptography 101 4
5. Google Cloud Platform Public-key cryptography 101 Alice Bob ALICE_PRIV ALICE_PUB Photo credit: indelwebastomarine.com 5
6. Google Cloud Platform Public-key cryptography 101 Alice Bob Photo credit: indelwebastomarine.com 6
7. Google Cloud Platform Public-key cryptography 101 Alice Bob secret text Photo credit: indelwebastomarine.com 7
8. Google Cloud Platform Public-key cryptography 101 Alice Bob Photo credit: indelwebastomarine.com 8
9. Google Cloud Platform Public-key cryptography 101 Alice Bob Photo credit: indelwebastomarine.com secret text 9
10. Google Cloud Platform Why TLS? Photo credit: Laura Poitras/Praxis Film y u no TLS? 10
11. Google Cloud Platform Why TLS? Users want Privacy Developers want Privacy of users, integrity of the transmitted data Services need Identity/Authentication (with mutual TLS)
12. Google Cloud Platform Where TLS? tl;dr: everywhere service HTTPs service HTTPs HTTPs users apps service HTTPs Load Balancer HTTPs gRPC+TLS 12
13. Google Cloud Platform Challenges with TLS for external traffic • Find a Certificate Authority (CA) • Prepare a Certificate Signing Request (learn how to create a CSR) • Manual verification process (fax, …) • Pay for the certificate • Download/store/install certs safely • Make sure the certs do not expire • Hope CA doesn’t get compromised • Hope you don’t need to revoke your cert 13
14. Google Cloud Platform ...Challenges with TLS for internal traffic • Host a Certificate Authority (CA) and PKI • Create a trust chain • Create certs for each service • develop in-house software to automate • Make sure certs do not expire • Create and store keys safely, set up the server • Hope you don’t get compromised 14
15. Google Cloud Platform Internal TLS 15
16. Google Cloud Platform Kubernetes TLS Certificate Authority Every Kubernetes cluster has a root CA. • Used by cluster components to validate trust • You can have cluster root CA sign your certificates too. 16
17. Google Cloud Platform Requesting a certificate from Kubernetes root CA 1. Create a CertificateSigningRequest 2. Approve the request (manual) 3. Server: download the cert from k8s API & use 4. Client: trust the k8s cluster root CA • k8s CA bundle is available on every Pod: /var/run/secrets/kubernetes.io/serviceaccount/ca.tls More info: https://kubernetes.io/docs/tasks/tls/ managing-tls-in-a-cluster/ 17
18. Google Cloud Platform Current issues with Kubernetes cluster root CA • Write code to automate process • check out kelseyhightower/certificate-init-container • Write code to • refresh the certificates • signal pods to reload the cert or restart the pods • handle cluster root CA certificate rotation • CA used for cluster components is the same as user workloads. • Suboptimal developer experience for now. ¯\_(ツ)_/¯ 18
19. Google Cloud Platform A better alternative A service mesh can run its own CA/PKI, and encrypt all communication between microservices automatically: • Istio: http://istio.io • Linkerd: http://linkerd.io app sidecar app sidecar localhost localhost POD A POD B TLS 19
20. Google Cloud Platform External TLS 20
21. Google Cloud Platform Let’s Encrypt Started in 2014, launched in 2016. Problem: • Getting a TLS certificate is too messy and expensive. • Only ~50% of the web traffic is encrypted today (2017) Solution: • Create a free CA that is fully automated. 21
22. Google Cloud Platform Let’s Encrypt – How it works Uses open IETF ACME protocol for issuing certificates. ACME client ACME server request certificate challenges complete challenge verify challenge issue 90-day certificate (or denial) 22
23. Google Cloud Platform ACME – Challenges • HTTP: place a file on your server (easy) e.g. example.com/.well-known/j4kw5n23 • DNS: add TXT record to your domain (easy) TXT _acme-challenge.example.com j4kw5n23 • TLS-SNI: serve a self-signed certificate with the challenge token with two subjectAltNames (difficult) 23
24. Google Cloud Platform Your apps can request Let’s Encrypt certificates. Challenges: • How to survive pod restarts • Coordinating between multiple replicas Example: • Caddy is a HTTP/2 server with automatic TLS (written in Go). It uses the github.com/xenolf/lego ACME implementation. (caddyserver.com) • Many languages have ACME libraries. You can build ACME in your application 24
25. Google Cloud Platform State of Let’s Encrypt in Kubernetes Fragmented. Popular Fork of below Merged solution, to be incubated 25
26. Google Cloud Platform Before you use these • You probably should not use either of these yet. • until the merged cert-manager moves to kubernetes-incubator and has stable releases. • Suboptimal user experience. :( • Make sure you have monitoring in place for expirations. Set alerting for certs close to expiration and refresh certificates periodically. 26
27. Google Cloud Platform Most practical option today Actually go buy a TLS certificate from one of the CAs to terminate external TLS. Use service mesh (linkerd, istio etc) to encrypt your internal traffic. 27
28. Google Cloud Platform Try it out – kube-lego on Google Container Engine https://github.com/jetstack/kube-lego/tree/master/examples/gce Took me 20 minutes to get a HTTPs website running and I had to troubleshoot constantly (and I used it before). 28
29. Google Cloud Platform Thank you @ahmetb 29
30. Google Cloud Platform Appendix/Backup slides 30
31. Google Cloud Platform Step 1: Create CSR server-csr.json: { "hosts": [ "web.default.svc.cluster.local" ], "CN": "web.default.svc.cluster.local", "key": { "algo": "ecdsa", "size": 256 } } Turns this into a server.csr and server-key.pem: $ cfssl genkey server-csr.json | cfssljson -bare server 31
32. Google Cloud Platform $ cat <<EOF | kubectl create -f - apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: web.default spec: groups: - system:authenticated request: $(cat server.csr | base64 | tr -d '\n') EOF $ kubectl get csr NAME AGE REQUESTOR CONDITION web.default 3s ahmetalpbalkan@gmail.com Pending Step 2: Submit CSR 32
33. Google Cloud Platform $ kubectl certificate approve web.default $ kubectl get csr NAME AGE REQUESTOR CONDITION web.default 45s ahmetalpbalkan@gmail.com Approved,Issued Step 3: Manually approve CSR 33
34. Google Cloud Platform Secret is stored on status.certificate field: $ kubectl get csr web.default -o jsonpath='{.status.certificate}' \ | base64 -d > server.crt Create a TLS secret with the key/cert pair: $ kubectl create secret tls web-tls --cert server.crt --key server-key.pem Then mount the web-tls secret as volume and use it. Step 4: Extract the certificate 34