Moving to Kubernetes: the Bad and the Ugly

Over those past two years we’ve deployed a new platform at Xing based on Kubernetes on which to develop, test, and deploy XING's applications.
Today most of our workloads run on this platform but the road sometimes felt a bit bumpy.

From misconfiguration, to software bugs in the Kubernetes ecosystem or even Kernel race conditions, we encountered a variety of problems.
In this talk we will discuss some of those issues, how we fixed, mitigated, or worked around them.

Transcript

1. MOVING TO KUBERNETES THE GOOD THE BAD AND THE UGLY
3. Plan - Introduction 1. NGINX Ingress Controller 2. Container Networking 3. CronJobs 4. Docker - Q&A
4. Introduction
5. Kubernetes setup • Kubernetes 1.11 • Docker 18.06 • Ubuntu Bionic • VMWare on-premise
6. Some numbers • 11 clusters • 500 worker nodes • 30k Pods • 230 productive applications
7. NGINX Ingress Controller
8. Requirements • No request lost normally • Requests can last 60s without being interrupted
9. Bad results • Major bug in NGINX • One major and a few minors in NGINX Ingress Controller
10. Network interrupts Install irqbalance
11. —— apiVersion: v1 kind: Deployment metadata: name: nginx-ingress-controller spec: template: spec: terminationGracePeriodSeconds: “60” Ingress Controller shutdown Allow more than the default 30 seconds
12. —— apiVersion: v1 kind: ConfigMap metadata: name: nginx-ingress-controller-config data: worker-shutdown-timeout: “60” NGINX Worker shutdown Allow more than the default 10 seconds
13. Client Pod deletion
14. Client API Server Asks for Pod deletion Pod deletion
15. Client API Server Kubelet Notifies Pod deletion Pod deletion
16. Client API Server Kubelet Pods Sends SIGTERM Pod deletion
17. Client API Server Kubelet Pods Pod deletion
18. Client API Server Kubelet Pods Notifies Pod deletion Endpoint Controller Pod deletion
19. Client API Server Kubelet Pods Updates Endpoints Endpoint Controller Pod deletion
20. Client API Server Kubelet Pods Endpoint Controller Pod deletion
21. Client API Server Kubelet Pods Endpoint Controller NGINX Ingress Controller Notifies about Endpoints update Pod deletion
22. Client API Server Kubelet Pods Endpoint Controller NGINX Ingress Controller NGINX Generates config & sends “reload” signal Pod deletion
23. Client API Server Kubelet Pods Endpoint Controller NGINX Ingress Controller NGINX Pod deletion RACE CONDITION
24. —— apiVersion: v1 kind: Pod spec: containers: - name: lifecycle-demo-container lifecycle: preStop: exec: command: [“/bin/sh","-c","sleep 4"] preStop hook Give ingress—nginx time to remove the upstream
25. A few additional words • Recent releases do less configuration reloads • At XING we still run 0.10 • Taking a step-back to look for alternatives
26. Networking
27. Flannel 172.16.0.2 172.16.0.2 Pod A Node-1 172.16.0.2 172.16.0.2 Pod B Node-2 Network Interface Network Interface
28. SNAT 172.16.0.2 172.16.0.2 172.16.0.2 Node-1, 10.0.0.1 External server 192.168.0.100 Before (1) SRC 172.16.0.2 DST 192.168.0.100 After (2) SRC 10.0.0.1 DST 192.168.0.100 Network Interface 1 2
29. Timeouts • Impacts only Scala applications • Lot of HTTP calls to two specific IP addresses
30. Collecting clues • Response time jumps from 1s or 3s • TCP SYN packets disappearing within the host • NAT statistics show insert_failed > 0
31. Conntrack • Kernel tracks connections in a (conntrack) table • Race leads to some insertion to conflict
32. Race condition When: • multiple containers on the same host • at the same moment • connect to the same address outside the cluster then one connection might be delayed by 1s or more
33. Mitigation 1 Use —random-fully on iptables nat rules
34. Mitigation 2 Reuse connections
35. Workaround Avoid SNAT by using routable Pod IPs
36. That’s not all Similar race condition affects parallel DNS resolution (see #56903)
37. CronJobs
38. Timezones • Not supported • SIG Apps+Architecture against it for now • “Write your own controller” (see #47202)
39. —— apiVersion: batch/v1beta1 kind: CronJob metadata: name: example-app-cron-with-timezone annotations: com.xing.cronjob.timezone: CET spec: schedule: "0 8 * * *" Custom controller
40. Frozen CronJob After a CronJob missed 100 runs in a row, it’s ignored by Kubernetes, forever
41. “Fix” Remove the status.lastScheduleTime field when stuck
42. Avoid minute CronJobs Prefer always running Pods that start a subprocess periodically
43. Prevent out-of-schedule runs “CronJobs might be executed twice or not at all “ Set startDeadlineSeconds on your CronJobs
44. Docker
45. Bugs • Daemon randomly freezing • Container inspection hanging • Container stuck, can't be stopped, killed • Docker shims crashing
46. Symptoms • Nodes flapping between Ready and NotReady • Kubelets complaining that “PLEG is not healthy”
47. Initial solution Patched dumb-init delaying container’s exit for 5s
48. Pod Lifecycle Event Generator • Kubelet lists the containers every seconds • When a container’s state changes, inspects it and generates an event • Not healthy if this operation takes > 3 minutes
49. Origin of flappiness • Permanently broken containers are inspected twice • Inspection timing out = 2x 2m delay = NotReady • Successful container listing = Ready
50. Increase responsiveness Reduce the Kubelet timeout to 5 seconds (—runtime-request-timeout)
51. Origin of timeouts Caused by stuck inspection commands on: • Containers with no processes • Containers with only zombies
52. Container killer • Runs on every node • Searches for containers that can’t be inspected • kill -9 the Docker shims • Provides metrics about suspicious Pods
53. There is more to talk about • cgroup hierarchy and oom killers • side effects of using ipvs • process trees with sharedPIDs • some production horror stories
54. A last word
55. We had this case where Our got stuck CronJob garbage collection
56. We had this case where Our got stuck DaemonSetController CronJob garbage collection
57. We had this case where Our got stuck QuotaController DaemonSetController CronJob garbage collection
58. We had this case where Our got stuck own controller QuotaController DaemonSetController CronJob garbage collection
59. Check that your controllers are actually doing anything One alert to have
60. Thanks any questions ? maxlaverse maxime.lagresle maxime.lagresle@xing.com