Traefik on Kubernetes at MySocialApp (CNCF Paris Meetup)

How Traefik works in HA mode on Kubernetes with Let's Encrypt

Transcript

1. Traefik on Kubernetes at MySocialApp CNCF Meetup Paris - 04/26/2018
2. Who am I? Pierre Mavro @deimosfr MySocialApp Co-Founder / Criteo Senior SRE Lead - NoSQL
3. Build your own social networking app now with great features and messaging *social network app builder mysocialapp.io
4. Our clients Per app: • ~100k users • ~190k notifications/day • ~2.4M API call per day • ~200ms API latency at 99per
5. About Traefik and MySocialApp infrastructure
6. Ingress Controllers Back in 2017, Open Source Load Balancers (LB) solutions for Kubernetes Requested features Traefik Nginx HAProxy Kubernetes native integration ✔ ✔ ❌ SSL native management ✔ ❌ ❌ Advanced LB features ✔ ❌ ✔ Overall good performances ✔ ✔ ✔
7. Traefik • Simple and powerful LB • Official docker image • Rest API • Hot-reloading of configuration • Circuit breakers, retry • Round Robin, rebalancer load-balancers • Metrics (Prometheus...) • Websocket, HTTP/2, GRPC ready • Let’s Encrypt support (Automatic certs renewal) • Proxy Protocol support • HA cluster, Raft based • OpenTracing Support • ...
8. MySocialApp Infra • Kubernetes on premise • Traefik Load Balancer • Let’s Encrypt SSL • Cloudflare DNS
9. Traefik integration
10. Traefik on Kubernetes • Annotation configuration ◦ Rate limiting ◦ Redirects ◦ Whitelist ◦ Sticky sessions ◦ Custom Headers ◦ Basic Auth ◦ … apiVersion: extensions/v1beta1 kind: Ingress metadata: name: mysocialapp-website annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/affinity: "true" traefik.ingress.kubernetes.io/load-balancer-method: drr traefik.ingress.kubernetes.io/max-conn-amount: 10 ... spec: rules: - host: www.mysocialapp.io http: paths: - path: / backend: serviceName: mysocialapp-website servicePort: http
11. Traefik on Kubernetes • HA Mode supported (beta) • Backend requested • Watch Kubernetes API changes
12. Traefik on Kubernetes • Backend requested for: ◦ Storing configuration ◦ Storing SSL certs ◦ Distributed lock • Supported backends: ◦ Consul (recommended) ◦ Etcd ◦ BoltDB ◦ ZooKeeper [kubernetes] endpoint = "https://<your_kube_api_ip>:443" [consul] endpoint = "consul:8500" watch = true prefix = "traefik" 1. Consul Helm Chart: https://github.com/MySocialApp/kubernetes-helm-chart-consul 2. Traefik Helm Chart with HA support: https://github.com/MySocialApp/kubernetes-helm-chart-traefik Traefik configuration
13. Traefik supporting Let’s Encrypt (LE)
14. Let’s Encrypt support • Challenges: ◦ HTTP ◦ DNS (recommended) • Supported DNS providers: ◦ Cloudflare ◦ Gandi ◦ Google Cloud DNS ◦ PowerDNS ◦ Route 53 ◦ ... [acme] email = "your@email.here" storage = "traefik/acme/account" entryPoint = "https" OnHostRule = true acmeLogging = true [acme.dnsChallenge] provider = "cloudflare" delayBeforeCheck = 20 [[acme.domains]] main = "yourdomain.here"
15. External DNS • Automate DNS records • Supported DNS providers: ◦ Google Cloud DNS ◦ Cloudflare ◦ Azure ◦ … External DNS Helm Chart: https://github.com/MySocialApp/kubernetes-helm-chart-externaldns apiVersion: extensions/v1beta1 kind: Ingress metadata: name: mysocialapp-website annotations: kubernetes.io/ingress.class: traefik external-dns.alpha.kubernetes.io/target: your.cname.here ... spec: rules: - host: www.mysocialapp.io http: paths: - path: / backend: serviceName: mysocialapp-website servicePort: http
16. Feedbacks
17. Let’s Encrypt Rate Limiting Samples extracted from: https://letsencrypt.org/docs/rate-limits/ • If you’ve hit a rate limit, we don’t have a way to temporarily reset it. You’ll need to wait until the rate limit expires after a week • You can create a maximum of 10 Accounts per IP Address per 3 hours • There is a Failed Validation limit of 5 failures per account, per hostname, per hour Many other limitations ! • Take seriously rate limiting into account
18. Reduce LE Rate Limit issue If you want to go live production with Let’s Encrypt, you better: • Test using staging env • Get backups • Use latest Lego lib (latest Traefik version) • B plan ! (next slide) • Use wildcards if possible Wildcards are as easy as: [[acme.domains]] main = "*.yourdomain.here" • Test before live prod • Consul is a distributed system ◦ Storage ◦ Rolling restart ◦ Backups
19. Overcome LE Rate Limit You can deal with a CDN. Cloudflare to the rescue ! Enable it on your records: And configure how you want to deal with end to end encryption. This can helps to mitigate the issue. • Useful for: ◦ DNS API ◦ SSL certs ◦ Caching ◦ HA support ◦ ...
20. Traefik status @traefikproxy • Production ready • Can be complex in HA mode • Let’s Encrypt can be a pain • Amazing integration with Kubernetes • Wonderful support and community • Bug fix are quickly merged • Huge thanks to Emile Vauge (Creator) traefik.io
21. Questions? Pierre Mavro / @deimosfr