Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes TLS/Let’s Encrypt

Kubernetes TLS/Let’s Encrypt

This talk is about the state of Let's Encrypt on Kubernetes and Kubernetes’ Cluster Root CA as of June 2017. It is presented at the Seattle Kubernetes Meetup.

tl;dr: LE solutions on k8s are not mature yet, you should probably still buy a TLS certificate. For internal TLS, look at using a service mesh like Istio or Linkerd.

Ahmet Alp Balkan

June 01, 2017

More Decks by Ahmet Alp Balkan

Other Decks in Technology


  1. Google Cloud Platform logo Kubernetes TLS/Let’s Encrypt Seattle Kubernetes Meetup

    2017-06-01 Ahmet Alp Balkan <[email protected]> Software Engineer, Developer Programs @ Google @ahmetb 1
  2. Google Cloud Platform What is TLS Transport-layer* security * OSI

    Layer 4 (TCP/IP) SSL v2.0 in Netscape Navigator (1995) SSL v3.0 redesign of the protocol (1996) TLS v1.0 IETF standard published (1999) TLS v1.3 still draft as of 2016. Required by HTTP/2 in most implementations. Photo credit: 24hourwebhostingsupport.com 2
  3. Google Cloud Platform What is HTTPS TLS TLS TLS TLS

    TLS HTTP HTTP Photo credit: windchillfactory.com 3
  4. Google Cloud Platform Why TLS? Users want Privacy Developers want

    Privacy of users, integrity of the transmitted data Services need Identity/Authentication (with mutual TLS)
  5. Google Cloud Platform Where TLS? tl;dr: everywhere service HTTPs service

    HTTPs HTTPs users apps service HTTPs Load Balancer HTTPs gRPC+TLS 12
  6. Google Cloud Platform Challenges with TLS for external traffic •

    Find a Certificate Authority (CA) • Prepare a Certificate Signing Request (learn how to create a CSR) • Manual verification process (fax, …) • Pay for the certificate • Download/store/install certs safely • Make sure the certs do not expire • Hope CA doesn’t get compromised • Hope you don’t need to revoke your cert 13
  7. Google Cloud Platform ...Challenges with TLS for internal traffic •

    Host a Certificate Authority (CA) and PKI • Create a trust chain • Create certs for each service • develop in-house software to automate • Make sure certs do not expire • Create and store keys safely, set up the server • Hope you don’t get compromised 14
  8. Google Cloud Platform Kubernetes TLS Certificate Authority Every Kubernetes cluster

    has a root CA. • Used by cluster components to validate trust • You can have cluster root CA sign your certificates too. 16
  9. Google Cloud Platform Requesting a certificate from Kubernetes root CA

    1. Create a CertificateSigningRequest 2. Approve the request (manual) 3. Server: download the cert from k8s API & use 4. Client: trust the k8s cluster root CA • k8s CA bundle is available on every Pod: /var/run/secrets/kubernetes.io/serviceaccount/ca.tls More info: https://kubernetes.io/docs/tasks/tls/ managing-tls-in-a-cluster/ 17
  10. Google Cloud Platform Current issues with Kubernetes cluster root CA

    • Write code to automate process • check out kelseyhightower/certificate-init-container • Write code to • refresh the certificates • signal pods to reload the cert or restart the pods • handle cluster root CA certificate rotation • CA used for cluster components is the same as user workloads. • Suboptimal developer experience for now. ¯\_(ツ)_/¯ 18
  11. Google Cloud Platform A better alternative A service mesh can

    run its own CA/PKI, and encrypt all communication between microservices automatically: • Istio: http://istio.io • Linkerd: http://linkerd.io app sidecar app sidecar localhost localhost POD A POD B TLS 19
  12. Google Cloud Platform Let’s Encrypt Started in 2014, launched in

    2016. Problem: • Getting a TLS certificate is too messy and expensive. • Only ~50% of the web traffic is encrypted today (2017) Solution: • Create a free CA that is fully automated. 21
  13. Google Cloud Platform Let’s Encrypt – How it works Uses

    open IETF ACME protocol for issuing certificates. ACME client ACME server request certificate challenges complete challenge verify challenge issue 90-day certificate (or denial) 22
  14. Google Cloud Platform ACME – Challenges • HTTP: place a

    file on your server (easy) e.g. example.com/.well-known/j4kw5n23 • DNS: add TXT record to your domain (easy) TXT _acme-challenge.example.com j4kw5n23 • TLS-SNI: serve a self-signed certificate with the challenge token with two subjectAltNames (difficult) 23
  15. Google Cloud Platform Your apps can request Let’s Encrypt certificates.

    Challenges: • How to survive pod restarts • Coordinating between multiple replicas Example: • Caddy is a HTTP/2 server with automatic TLS (written in Go). It uses the github.com/xenolf/lego ACME implementation. (caddyserver.com) • Many languages have ACME libraries. You can build ACME in your application 24
  16. Google Cloud Platform State of Let’s Encrypt in Kubernetes Fragmented.

    Popular Fork of below Merged solution, to be incubated 25
  17. Google Cloud Platform Before you use these • You probably

    should not use either of these yet. • until the merged cert-manager moves to kubernetes-incubator and has stable releases. • Suboptimal user experience. :( • Make sure you have monitoring in place for expirations. Set alerting for certs close to expiration and refresh certificates periodically. 26
  18. Google Cloud Platform Most practical option today Actually go buy

    a TLS certificate from one of the CAs to terminate external TLS. Use service mesh (linkerd, istio etc) to encrypt your internal traffic. 27
  19. Google Cloud Platform Try it out – kube-lego on Google

    Container Engine https://github.com/jetstack/kube-lego/tree/master/examples/gce Took me 20 minutes to get a HTTPs website running and I had to troubleshoot constantly (and I used it before). 28
  20. Google Cloud Platform Step 1: Create CSR server-csr.json: { "hosts":

    [ "web.default.svc.cluster.local" ], "CN": "web.default.svc.cluster.local", "key": { "algo": "ecdsa", "size": 256 } } Turns this into a server.csr and server-key.pem: $ cfssl genkey server-csr.json | cfssljson -bare server 31
  21. Google Cloud Platform $ cat <<EOF | kubectl create -f

    - apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: web.default spec: groups: - system:authenticated request: $(cat server.csr | base64 | tr -d '\n') EOF $ kubectl get csr NAME AGE REQUESTOR CONDITION web.default 3s [email protected] Pending Step 2: Submit CSR 32
  22. Google Cloud Platform $ kubectl certificate approve web.default $ kubectl

    get csr NAME AGE REQUESTOR CONDITION web.default 45s [email protected] Approved,Issued Step 3: Manually approve CSR 33
  23. Google Cloud Platform Secret is stored on status.certificate field: $

    kubectl get csr web.default -o jsonpath='{.status.certificate}' \ | base64 -d > server.crt Create a TLS secret with the key/cert pair: $ kubectl create secret tls web-tls --cert server.crt --key server-key.pem Then mount the web-tls secret as volume and use it. Step 4: Extract the certificate 34