Kafka on Kubernetes: Keeping It Simple

Da847f0931c0b9413619128ac31a33e0?s=47 Nikki Thean
October 01, 2019

Kafka on Kubernetes: Keeping It Simple

I gave a presentation at Kafka Summit San Francisco 2019 that attempts to convince viewers that it is possible to run a stable Kafka cluster on Kubernetes without a complicated setup.

Bonus: I illustrated the slides by hand, and hid the Kafka logo in many of the pictures!

The talk description is below:

If you’ve ever thought that running Kafka on Kubernetes was a terrible idea, welcome to the club: that’s what we thought as well. When we migrated Etsy’s Kafka deployment from bare metal to GCP, we made a surprising discovery: running Kafka on Kubernetes was the best option for us — and it wasn’t half as complicated as we thought it had to be.

I’ll use the story of our cloud migration journey to frame a discussion of how a “simple” Kafka-on-k8s setup can work. You’ll walk away with an example of how to set up a stable production system that uses vanilla Kubernetes workloads, as well as some technical tips and tricks — and pitfalls to avoid — for your own cloud migration or first Kafka-on-k8s deployment.

Da847f0931c0b9413619128ac31a33e0?s=128

Nikki Thean

October 01, 2019
Tweet

Transcript

  1. KAFKA ON KUBERNETES: Keeping It Simple

  2. None
  3. None
  4. None
  5. None
  6. None
  7. None
  8. We’re not doing that…
 right?

  9. We’re not doing that…
 right?

  10. We’re not doing that…
 right? Definitely not!

  11. ONE YEAR LATER

  12. ONE YEAR LATER 1. Production Kafka cluster on Kubernetes. 2.

    Suggesting this idea to other people.
  13. ONE YEAR LATER 1. Production Kafka cluster on Kubernetes. 2.

    Suggesting this idea to other people.
  14. We’re not doing that…
 right? Definitely not!

  15. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED.

  16. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED. *

    "Complicated"
  17. RUNNING KAFKA ON KUBERNETES DOESN'T HAVE TO BE COMPLICATED. *

    "Complicated" = custom resource definitions, plugins, operators, etc.
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. WHAT YOU’LL GET OUT OF THIS

  25. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life

    production setup
  26. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life

    production setup ➤ Technical tips and tricks
  27. WHAT YOU’LL GET OUT OF THIS ➤ Example of real-life

    production setup ➤ Technical tips and tricks ➤ Advice for migrating production systems
  28. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform

    ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
  29. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform

    ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
  30. SYSTEMS MENTIONED ➤ Kafka ➤ Kubernetes ➤ Chef ➤ Terraform

    ➤ Helm ➤ Prometheus ➤ Google Cloud Platform
  31. WHAT WE BUILT and how it works

  32. None
  33. WHAT WE CONSIDERED ➤ VMs + Chef

  34. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

  35. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images
  36. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems
  37. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups
  38. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads
  39. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads ➤ …Kubernetes?!
  40. WHAT WE CONSIDERED ➤ VMs + Chef ➤ Mutable infrastructure

    ➤ VMs + Machine/Docker Images ➤ Same amount of toil, new cloud infra problems ➤ Managed Instance Groups ➤ Not good for stateful workloads ➤ …Kubernetes?!
  41. None
  42. None
  43. CROSS-ZONE TRAFFIC IS $$$.

  44. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones

  45. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node

    type
  46. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node

    type - varying workloads, e.g. high CPU requirement
  47. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node

    pools - varying workloads, e.g. high CPU requirement ➤ Workload allocation controlled by: ➤ nodeSelectors ➤ pool: highmem ➤ failure-domain.beta.kubernetes.io/zone: us-central1-a
  48. RESOURCE MANAGEMENT ➤ Examples of separation: ➤ zones ➤ node

    pools - varying workloads, e.g. high CPU requirement ➤ Workload allocation controlled by: ➤ nodeSelectors ➤ pool: highmem ➤ failure-domain.beta.kubernetes.io/zone: us-central1-a ➤ taints + tolerations ➤ - key: pool
 operator: Equal
 value: highmem
 effect: NoSchedule
  49. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker
  50. None
  51. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer
  52. None
  53. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
  54. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
  55. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
  56. SERVICE DISCOVERY ➤ Within the cluster: ClusterIP Service ➤ e.g.

    Kafka broker to Kafka broker, Kafka Connect worker to Kafka broker ➤ External services to Kafka: ➤ Bootstrapping: Cloud DNS to LoadBalancer ➤ Direct to broker: NodePort* ➤ Configuring your Kafka listeners ➤ https://rmoff.net/2018/08/02/kafka-listeners-explained/ * In our specific case, ClusterIP due to VPC-native IP aliasing on GCP. Also possible: dedicated LoadBalancers.
  57. SOME NOTES ON MANAGEMENT

  58. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm

    charts ➤ https://github.com/confluentinc/cp-helm-charts
  59. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm

    charts ➤ https://github.com/confluentinc/cp-helm-charts ➤ No Tiller, Helm templating only
  60. SOME NOTES ON MANAGEMENT ➤ Installation, deploy: ➤ Confluent Helm

    charts ➤ https://github.com/confluentinc/cp-helm-charts ➤ No Tiller, Helm templating only ➤ Confluent Docker images, with additions
  61. SOME NOTES ON MANAGEMENT ➤ Monitoring:

  62. SOME NOTES ON MANAGEMENT ➤ Monitoring: ➤ Kafka exposes JMX

    metrics by default
  63. SOME NOTES ON MANAGEMENT ➤ Monitoring: ➤ Kafka exposes JMX

    metrics by default ➤ Prometheus JMX Exporter as Java agent (vs. Helm chart sidecar)
  64. WHAT WE LEARNED and what we think you should know

  65. WHAT WE LEARNED

  66. WHAT WE LEARNED

  67. WHAT WE LEARNED ➤ Ephemeral resources

  68. WHAT WE LEARNED ➤ Ephemeral resources

  69. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions
  70. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl)
  71. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl)
  72. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check the Apache Kafka JIRA
  73. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Oh, and upgrade Kafka.
  74. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check JIRA/Upgrade Kafka ➤ Producers and consumers too!
  75. WHAT WE LEARNED ➤ Ephemeral resources ➤ Don’t assume static

    IPs; make sure Kafka clients can handle pod evictions ➤ Check your JVM DNS cache TTL (networkaddress.cache.ttl) ➤ Check JIRA/Upgrade Kafka ➤ Producers and consumers too! ➤ Examples: KAFKA-7755, KAFKA-7890 ➤ See also: cp-helm-charts issue #240
  76. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts

  77. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤

    Changes to Kafka cluster: versions, broker properties
  78. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤

    Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies
  79. WHAT WE LEARNED ➤ Different kinds of updates/rolling restarts ➤

    Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes
  80. Health checks are important for self-healing clusters!

  81. Health checks are important for self-healing clusters!

  82. Health checks are important for self-healing clusters!

  83. CONTAINER PROBES

  84. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?”

  85. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?”

    ➤ Readiness Probe
 “Should this container accept traffic?”
  86. None
  87. https://github.com/andreas-schroeder/kafka-health-check

  88. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?”

    ➤ Endpoint: is this broker healthy?
  89. CONTAINER PROBES ➤ Liveness Probe
 “Should I restart this container?”

    ➤ Endpoint: is this broker healthy? ➤ Don’t be too strict!
  90. https://github.com/andreas-schroeder/kafka-health-check

  91. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates
  92. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates
  93. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget
  94. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy
  95. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy ➤ Health check overrides ➤ What happens if you deploy a change that breaks the health checks?
  96. WHAT WE HAD TO FIGURE OUT ➤ Different kinds of

    updates/rolling restarts ➤ Changes to Kafka cluster: versions, broker properties ➤ Workload configuration: resources, security policies ➤ Upgrading Kubernetes nodes ➤ How health checks affect updates ➤ podDisruptionBudget, podManagementPolicy ➤ Health check overrides ➤ What happens if you deploy a change that breaks the health checks? ➤ See Kubernetes issue #62750
  97. MIGRATING PRODUCTION ARCHITECTURE

  98. MIGRATING PRODUCTION ARCHITECTURE

  99. MIGRATING PRODUCTION ARCHITECTURE

  100. MIGRATING PRODUCTION ARCHITECTURE

  101. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty!

  102. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty!

  103. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks
  104. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test
  105. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts…
  106. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs
  107. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs ➤ Simulate failure
  108. MIGRATING PRODUCTION ARCHITECTURE ➤ Get your hands dirty! ➤ Simulate

    common maintenance tasks ➤ Benchmark for performance ➤ kafka-producer-perf-test, kafka-consumer-perf-test ➤ Variables: disk type, CPU count, producer record size, producer batch size, Java opts… ➤ We were able to use Compute Engine persistent disks (shared storage) rather than local SSDs ➤ Simulate failure
  109. Try to keep it simple!

  110. WHY IT WORKS FOR US ( for now, at least!)

  111. WHY IT WORKS FOR US

  112. WHY IT WORKS FOR US ➤ Increased automation

  113. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler

    configuration
  114. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler

    configuration ➤ Efficient resource usage ➤ Bin packing ➤ GKE autoscaling
  115. WHY IT WORKS FOR US ➤ Increased automation ➤ Simpler

    configuration ➤ Efficient resource usage ➤ Bin packing ➤ GKE autoscaling ➤ Improved developer workflows for streaming services ➤ e.g. adding new Kafka Streams applications, Kafka Connect workloads
  116. None
  117. THANK YOU! Twitter: @NikkiThean
 Confluent Slack: @nikki
 Email: nikki.thean@gmail.com Thank

    you to Kamo for drawing inspiration!