Kafka on Kubernetes: Keeping It Simple

I gave a presentation at Kafka Summit San Francisco 2019 that attempts to convince viewers that it is possible to run a stable Kafka cluster on Kubernetes without a complicated setup.

Bonus: I illustrated the slides by hand, and hid the Kafka logo in many of the pictures!

The talk description is below:

If you’ve ever thought that running Kafka on Kubernetes was a terrible idea, welcome to the club: that’s what we thought as well. When we migrated Etsy’s Kafka deployment from bare metal to GCP, we made a surprising discovery: running Kafka on Kubernetes was the best option for us — and it wasn’t half as complicated as we thought it had to be.

I’ll use the story of our cloud migration journey to frame a discussion of how a “simple” Kafka-on-k8s setup can work. You’ll walk away with an example of how to set up a stable production system that uses vanilla Kubernetes workloads, as well as some technical tips and tricks — and pitfalls to avoid — for your own cloud migration or first Kafka-on-k8s deployment.

Transcript

1. KAFKA ON KUBERNETES: Keeping It Simple
8. We’re not doing that…
 right?
9. We’re not doing that…
 right?
10. We’re not doing that…
 right? Definitely not!
11. ONE YEAR LATER
12. ONE YEAR LATER 1. Production Kafka cluster on Kubernetes. 2. Suggesting this idea to other people.
13. ONE YEAR LATER 1. Production Kafka cluster on Kubernetes. 2. Suggesting this idea to other people.
14. We’re not doing that…
 right? Definitely not!
15. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED.
16. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED. * "Complicated"
17. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED. * "Complicated" = custom resource definitions, plugins, operators, etc.
24. WHAT YOU’LL GET OUT OF THIS
25. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life production setup
26. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life production setup ➤ Technical tips and tricks
27. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life production setup ➤ Technical tips and tricks ➤ Advice for migrating production systems
28. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
29. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
30. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
31. WHAT WE BUILT and how it works
33. WHAT WE CONSIDERED ➤ VMs + Chef
34. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure
35. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images
36. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems
37. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups
38. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads
39. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads ➤ …Kubernetes?!
40. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads ➤ …Kubernetes?!
43. CROSS-ZONE TRAFFIC IS $$$.
44. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones
45. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node type
46. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node type - varying workloads, e.g. high CPU requirement
47. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node pools - varying workloads, e.g. high CPU requirement ➤ Workload allocation controlled by: ➤ nodeSelectors ➤ pool: highmem ➤ failure-domain.beta.kubernetes.io/zone: us-central1-a
48. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node pools - varying workloads, e.g. high CPU requirement ➤ Workload allocation controlled by: ➤ nodeSelectors ➤ pool: highmem ➤ failure-domain.beta.kubernetes.io/zone: us-central1-a ➤ taints + tolerations ➤ - key: pool
 operator: Equal
 value: highmem
 effect: NoSchedule
49. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker
51. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer
53. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
54. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
55. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
56. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g. Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners ➤ https://rmoff.net/2018/08/02/kafka-listeners-explained/ * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
57. SOME NOTES ON MANAGEMENT
58. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm charts ➤ https://github.com/confluentinc/cp-helm-charts
59. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm charts ➤ https://github.com/confluentinc/cp-helm-charts ➤ No Tiller, Helm templating only
60. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm charts ➤ https://github.com/confluentinc/cp-helm-charts ➤ No Tiller, Helm templating only ➤ Confluent Docker images, with additions
61. SOME NOTES ON MANAGEMENT ➤ Monitoring:
62. SOME NOTES ON MANAGEMENT ➤ Monitoring: ➤ Kafka exposes JMX metrics by default
63. SOME NOTES ON MANAGEMENT ➤ Monitoring: ➤ Kafka exposes JMX metrics by default ➤ Prometheus JMX Exporter as Java agent (vs. Helm chart sidecar)
64. WHAT WE LEARNED and what we think you should know
65. WHAT WE LEARNED
66. WHAT WE LEARNED
67. WHAT WE LEARNED ➤ Ephemeral resources
68. WHAT WE LEARNED ➤ Ephemeral resources
69. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions
70. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl)
71. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl)
72. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check the Apache Kafka JIRA
73. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Oh, and upgrade Kafka.
74. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check JIRA/Upgrade Kafka ➤ Producers and consumers too!
75. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check JIRA/Upgrade Kafka ➤ Producers and consumers too! ➤ Examples: KAFKA-7755, KAFKA-7890 ➤ See also: cp-helm-charts issue #240
76. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts
77. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties
78. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies
79. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes
80. Health checks are important for self-healing clusters!
81. Health checks are important for self-healing clusters!
82. Health checks are important for self-healing clusters!
83. CONTAINER PROBES
84. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?”
85. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?” ➤ Readiness Probe
 “Should this container accept traffic?”
87. https://github.com/andreas-schroeder/kafka-health-check
88. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?” ➤ Endpoint: is this broker healthy?
89. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?” ➤ Endpoint: is this broker healthy? ➤ Don’t be too strict!
90. https://github.com/andreas-schroeder/kafka-health-check
91. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates
92. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates
93. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget
94. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy
95. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy ➤ Health check overrides ➤ What happens if you deploy a change that breaks the health checks?
96. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy ➤ Health check overrides ➤ What happens if you deploy a change that breaks the health checks? ➤ See Kubernetes issue #62750
97. MIGRATING PRODUCTION ARCHITECTURE
98. MIGRATING PRODUCTION ARCHITECTURE
99. MIGRATING PRODUCTION ARCHITECTURE
100. MIGRATING PRODUCTION ARCHITECTURE
101. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty!
102. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty!
103. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks
104. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test
105. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts…
106. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs
107. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs ➤ Simulate failure
108. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs ➤ Simulate failure
109. Try to keep it simple!
110. WHY IT WORKS FOR US ( for now, at least!)
111. WHY IT WORKS FOR US
112. WHY IT WORKS FOR US ➤ Increased automation
113. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler configuration
114. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler configuration ➤ Efficient resource usage ➤ Bin packing ➤ GKE autoscaling
115. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler configuration ➤ Efficient resource usage ➤ Bin packing ➤ GKE autoscaling ➤ Improved developer workflows for streaming services ➤ e.g. adding new Kafka Streams applications, Kafka Connect workloads
117. THANK YOU! Twitter: @NikkiThean
 Confluent Slack: @nikki
 Email: nikki.thean@gmail.com Thank you to Kamo for drawing inspiration!