Kubernetes Ingress What, How & Caveats

Who am I? Vincent De Smet vincentdesmet so0k I LIKE KUBERNETES

What is ingress? ● Revisit Kubernetes service Service: - clusterIP: 10.100.x.y - selector: - app: my-app - version: 1.0

● Revisit Kubernetes service 10.100.x.y What is ingress? Service: - nodePort: 3200Z - selector: - app: my-app - version: 1.0 3200Z 3200Z 3200Z

● Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80

DNS ● Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80 foo.com

DNS ● Revisit Kubernetes service What is ingress? Service: - loadBalancer - selector: - app: my-app - version: 1.0 10.100.x.y 3200Z 3200Z 3200Z lb.cloud.com 80 foo.com bar.com 10.100.a.b lb.cloud.com 80 3200C 3200C 3200C

What is Ingress? ● Services of type Load Balancer Reduced failure domain per service Cost ● Ingress - added in kube 1.1 A way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint. - Jay Gorrel

Web 101 - Simple times bar foo DNS foo.com bar.com 80 80

Web 201 - Containers bar foo DNS foo.com bar.com ??

Reverse Proxies - Containers - Reverse Proxy - nginx - haproxy - ... vhost: bar vhost: foo DNS foo.com bar.com 80 nginx

Reverse Proxies with Kubernetes - Static Configuration? vhost: bar vhost: foo DNS foo.com bar.com 80 nginx vhost: baz vhost: qux vhost: ...

- Dynamic configuration ← INGRESS CONTROLLERS ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Kubernetes Ingress vhost: bar vhost: foo DNS foo.com bar.com nginx vhost: baz vhost: qux vhost: ... nginx-ingress-controller 80

Reverse Proxies with Kubernetes - Dynamic configuration ← INGRESS CONTROLLERS vhost: bar vhost: foo DNS foo.com bar.com nginx vhost: baz vhost: qux vhost: ... ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 nginx-ingress-controller 80

Reverse Proxies with Kubernetes - Dynamic configuration ← INGRESS CONTROLLERS vhost: bar vhost: foo DNS foo.com bar.com haproxy vhost: baz vhost: qux vhost: ... ingress: rules: - host: foo.com http: paths: - path: / backend: serviceName: foo servicePort: 80 ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 haproxy-ingress-controller 80

Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 proxy-controller

Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller

Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller cert-controller

Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 DNS bar.com proxy-controller dns-controller cert-controller lb-controller

Ingress controllers ... ingress: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 heptio/contour external-dns DNS bar.com jetstack/cert-manager kube-ingress-aws-controller

Tips & Caveats - Run multiple ingress controllers - ingress.class - Ingress status - Failure domains - Reverse proxy config - Failed Let's Encrypt requests - Annotation overload - Cloud Native proxies - SIGHUP - Observability

Tips & Caveats - Run multiple ingress controllers - ingress.class Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80

Tips & Caveats - Run multiple ingress controllers - ingress.class - custom annotations Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour-int zalando.org/aws-load-balancer-scheme: internal spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 Ingress: kind: Ingress metadata: annotations: kubernetes.io/ingress.class: contour-ext spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80

Tips & Caveats - Ingress status Ingress: kind: Ingress metadata: annotations: ... spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … } DNS bar.com

Tips & Caveats - Ingress status Ingress: kind: Ingress metadata: annotations: external-dns.alpha.kubernetes.io/target: lb.cloud.com spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { } DNS bar.com

Tips & Caveats - External DNS: Custom filter Ingress: kind: Ingress metadata: annotations: external-dns.swatmobile.io/enable: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … }

Tips & Caveats - Failure domain - Reverse proxy config Ingress: kind: Ingress metadata: annotations: external-dns.swatmobile.io/enable: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 status: loadBalancer: { … }

Tips & Caveats - Failure domain - Reverse proxy config - Watch out: Failed Let's Encrypt requests Ingress: kind: Ingress metadata: annotations: kubernetes.io/tls-acme: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 tls: - secretName: bar-com-tls hosts: - bar.com status: loadBalancer: { … }

Tips & Caveats - Plan for failed Let's Encrypt request: - Run multiple ingress to reduce failure domain - Decouple Cert request from ingress Ingress: kind: Ingress metadata: annotations: kubernetes.io/tls-acme: "true" spec: rules: - host: bar.com http: paths: - path: / backend: serviceName: bar servicePort: 80 tls: - secretName: bar-com-tls hosts: - bar.com status: loadBalancer: { … }

Tips & Caveats - Annotation overload - https://github.com/heptio/contour/blob/master/docs/ingressroute.md - Ambassador - Istio - ... - Balance CRD vs Annotation Ingress Annotations more mature (for now)

Tips & Caveats - Cloud Native proxies - SIGHUP - Observability

Kops - Edge Nodes Why? - Compliance requirements (WAF) How? - Dedicated nodePool / instanceGroup - AWS LB limited to edge nodes

Kops - Edge Nodes Dedicated nodePool / instanceGroup:

Kops - Edge Nodes AWS LB limited to edge nodes: Problem: - Kubernetes service type LoadBalancer (targets all worker nodes) Solution: - https://github.com/zalando-incubator/kube-ingress-aws-controller CUSTOM_FILTERS

Kops - Edge Nodes AWS LB limited to edge nodes:

Kops - Edge Nodes AWS LB limited to edge nodes:

Thank you Questions?