Ingress at Swatmobile

What is Kubernetes Ingress? What flavours are available? how do these make our life easier? What are some of the caveats we have to watch out for? As we've been experimenting with several different set ups both at Swat and at previous startup(s), mostly from a functionality and operational perspective, we share our experience and also our thoughts on how we see this evolving. In this talk there is less focus on performance benchmarks and cover more conceptually the topic of ingress.

March 2019 - Kubernetes Singapore meetup

Transcript

1. Kubernetes Ingress What, How & Caveats
2. Who am I? Vincent De Smet vincentdesmet so0k I LIKE KUBERNETES
3. What is ingress? • Revisit Kubernetes service Service: - clusterIP: 10.100.x.y - selector: - app: my-app - version: 1.0
4. • Revisit Kubernetes service 10.100.x.y What is ingress? Service: - nodePort: 3200Z - selector: - app: my-app - version: 1.0 3200Z 3200Z 3200Z
5. • Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80
6. DNS • Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80 foo.com
7. DNS • Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80 foo.com bar.com 10.100.a.b lb.cloud.com 80 3200C 3200C 3200C
8. What is Ingress? • Services of type Load Balancer Reduced failure domain per service Cost • Ingress - added in kube 1.1 A way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint. - Jay Gorrel
9. Web 101 - Simple times bar foo DNS foo.com bar.com 80 80
10. Web 201 - Containers bar foo DNS foo.com bar.com ??
11. Reverse Proxies - Containers - Reverse Proxy - nginx - haproxy - ... vhost: bar vhost: foo DNS foo.com bar.com 80 nginx
12. Reverse Proxies with Kubernetes - Static Configuration? vhost: bar vhost: foo DNS foo.com bar.com 80 nginx vhost: baz vhost: qux vhost: ...
13. - Dynamic configuration ← INGRESS CONTROLLERS ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Kubernetes Ingress vhost: bar vhost: foo DNS foo.com bar.com nginx vhost: baz vhost: qux vhost: ... nginx-ingress-controller 80
14. Reverse Proxies with Kubernetes - Dynamic configuration ← INGRESS CONTROLLERS vhost: bar vhost: foo DNS foo.com bar.com nginx vhost: baz vhost: qux vhost: ... ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 nginx-ingress-controller 80
15. Reverse Proxies with Kubernetes - Dynamic configuration ← INGRESS CONTROLLERS vhost: bar vhost: foo DNS foo.com bar.com haproxy vhost: baz vhost: qux vhost: ... ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 haproxy-ingress-controller 80
16. Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 proxy-controller
17. Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller
18. Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller cert-controller
19. Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller cert-controller lb-controller
20. Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 heptio/contour external-dns DNS bar.com jetstack/cert-manager kube-ingress-aws-controller
21. Tips & Caveats - Run multiple ingress controllers - ingress.class - Ingress status - Failure domains - Reverse proxy config - Failed Let's Encrypt requests - Annotation overload - Cloud Native proxies - SIGHUP - Observability
22. Tips & Caveats - Run multiple ingress controllers - ingress.class Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80
23. Tips & Caveats - Run multiple ingress controllers - ingress.class - custom annotations Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour-int zalando.org/aws-load-balancer-scheme: internal spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour-ext spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80
24. Tips & Caveats - Ingress status Ingress: kind: Ingress metadata: annotations: ... spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … } DNS bar.com
25. Tips & Caveats - Ingress status Ingress: kind: Ingress metadata: annotations: external-dns.alpha.kubernetes.io/target: lb.cloud.com spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { } DNS bar.com
26. Tips & Caveats - External DNS: Custom filter Ingress: kind: Ingress metadata: annotations: external-dns.swatmobile.io/enable: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … }
27. Tips & Caveats - Failure domain - Reverse proxy config Ingress: kind: Ingress metadata: annotations: external-dns.swatmobile.io/enable: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … }
28. Tips & Caveats - Failure domain - Reverse proxy config - Watch out: Failed Let's Encrypt requests Ingress: kind: Ingress metadata: annotations: kubernetes.io/tls-acme: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 tls: - secretName: bar-com-tls hosts: - bar.com status: loadBalancer: { … }
29. Tips & Caveats - Plan for failed Let's Encrypt request: - Run multiple ingress to reduce failure domain - Decouple Cert request from ingress Ingress: kind: Ingress metadata: annotations: kubernetes.io/tls-acme: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 tls: - secretName: bar-com-tls hosts: - bar.com status: loadBalancer: { … }
30. Tips & Caveats - Annotation overload - https://github.com/heptio/contour/blob/master/docs/ingressroute.md - Ambassador - Istio - ... - Balance CRD vs Annotation Ingress Annotations more mature (for now)
31. Tips & Caveats - Cloud Native proxies - SIGHUP - Observability
32. Kops - Edge Nodes Why? - Compliance requirements (WAF) How? - Dedicated nodePool / instanceGroup - AWS LB limited to edge nodes
33. Kops - Edge Nodes Dedicated nodePool / instanceGroup:
34. Kops - Edge Nodes AWS LB limited to edge nodes: Problem: - Kubernetes service type LoadBalancer (targets all worker nodes) Solution: - https://github.com/zalando-incubator/kube-ingress-aws-controller CUSTOM_FILTERS
35. Kops - Edge Nodes AWS LB limited to edge nodes:
36. Kops - Edge Nodes AWS LB limited to edge nodes:
37. Thank you Questions?