Kafka and Zookeeper on AWS

How to use Elastic Network Interfaces to smoothly recover from Zookeeper failures. Link to gist https://gist.github.com/rcillo/1a64d757bf3ebaffcb3c71eb95607f1f

Transcript

1. How to smoothly recover from Zookeeper failures Ricardo de Cillo Data engineer at Zalando @rcillo Kafka on AWS
3. A typical setup Kafka Cluster Broker 1 Broker 2 Broker N . . . Zookeeper Ensemble Node 1 Node 2 Node 3
4. A typical setup… in real life Kafka Cluster Broker 1 Broker 2 Broker N . . . Zookeeper Ensemble Node 1 Node 2 Node 3
5. A typical setup… in real life Kafka Cluster Broker 1 Broker 2 Broker N . . . Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13
6. A typical setup… in real life Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . .
7. A typical setup… in real life Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties
8. A typical setup… in real life Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect
9. A typical setup… in real life Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
10. A failure scenario Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
11. 1. Everything keeps working (Quorum satisfied) What happens
12. 1. Everything keeps working (Quorum satisfied) 2. Maybe a Zookeeper leadership election What happens
13. 1. Everything keeps working (Quorum satisfied) 2. Maybe a Zookeeper leadership election 3. Kafka-Zookeeper client gracefully reconnects to the remaining nodes What happens
14. What happens 1. Everything keeps working (Quorum satisfied) 2. Maybe a Zookeeper leadership election 3. Kafka-Zookeeper client gracefully reconnects to the remaining nodes 4. Users probably don't even notice
15. Recovering
16. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
17. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.130
18. Nops
19. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
20. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
21. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
22. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
23. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
24. All good?
25. Noticed something?
26. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
27. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.130
28. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.130
29. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.130
30. Recovering Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.130 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.130 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.130
31. It takes hours
32. My personal opinion
33. My personal opinion - Kafka was not designed to run on disposable hardware like EC2 instances
34. My personal opinion - Kafka was not designed to run on disposable hardware like EC2 instances - It works better with static IPs
35. My personal opinion - Kafka was not designed to run on disposable hardware like EC2 instances - It works better with static IPs - It does not work with zookeeper.connect=domain-name
37. My personal opinion - Kafka was not designed to run on disposable hardware like EC2 instances - It works best with static IPs - It does not work with zookeeper.connect=domain-name - Kafka reloads config
39. My personal opinion - Kafka was not designed to run on disposable hardware like EC2 instances - It works best with static IPs - It does not work with zookeeper.connect=domain-name - Kafka reloads config
40. How we improved it
41. How we improved it Zookeeper + Exhibitor Node 1: 10.0.0.11 Node 2: 10.0.0.12 Node 3: 10.0.0.13 S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13
42. How we improved it Zookeeper + Exhibitor S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13 Node 1: 10.0.0.1 Node 2: 10.0.0.2 Node 3: 10.0.0.3 10.0.0.11 10.0.0.12 10.0.0.13
43. Let's test it
44. How we improved it Zookeeper + Exhibitor S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13 Node 1: 10.0.0.1 Node 2: 10.0.0.2 Node 3: 10.0.0.3 10.0.0.11 10.0.0.12 10.0.0.13
45. How we improved it Zookeeper + Exhibitor S3 configs 1:10.0.0.11 2:10.0.0.12 3:10.0.0.13 Kafka Cluster Broker 1 Broker 2 Broker N . . . server properties zookeeper connect 10.0.0.11 10.0.0.12 10.0.0.13 Node 1: 10.0.0.1 Node 2: 10.0.0.2 Node 3: 10.0.0.4 10.0.0.11 10.0.0.12 10.0.0.13
46. =)
47. =) Check this gist
48. Thank you