Reattaching Kafka EBS in AWS

Worried about what happens when your Kafka brokers go down in AWS? Here's a solution to allow quick recovery while keeping your EBS volumes.
https://github.com/zalando-nakadi/bubuku/tree/master/instance_control
https://dzone.com/articles/reattaching-kafka-ebs-in-aws

Transcript

1. Kafka on EBS Zalando Kafka Meetup Andrey Dyachkov 07.11.2017
2. whoami Software Engineer @ Zalando Nakadi (Messaging Bus) Twitter: @a_dyachkov Github: @adyach
3. Cluster the beginning 5 Kafka m3.medium 2TB gp2 3 Zookeeper m3.medium Ingest 250GB per day
4. Cluster nowadays 15 Kafka m4.2xlarge 8TB st1 3 Zookeeper i3.large Ingest 5TB per day Egress 30TB per day
5. Problems • Loss of instance • Changing instance type • Upgrading docker image • Kafka cluster upgrade
6. Problems 7 hours of copying data for 1 node PANIC PICTURE PLACEHOLDER
7. Amazon EBS
8. Problems
9. Problems
10. Solution
11. Solution
12. Solution 0 minutes of copying data PROBLEM SOLVED PICTURE PLACEHOLDER
13. AWS Auto Recovery
14. AWS Auto Recovery • Attaches EBS • Retains instance IP
15. What are those scripts
16. Create Kafka node bubuku_cluster.py create --instance-count 1 --cluster-config $CONFIG • Create IAM profile • Create Security Groups • Allocate IP address • Wait for instance to be started • Start Kafka
17. Terminate Kafka node bubuku_cluster.py terminate --cluster-config $CONFIG \ --ip 10.246.2.11 \ --user adyachkov \ --odd odd-eu-central-1.aruha-test.zalan.do • Deattach EBS • Stop Kafka gracefully • Terminate instance
18. Attach Kafka node bubuku_cluster.py attach --volume-id vol-1 --cluster-config $CONFIG • Create Kafka node • Attach provided EBS to it • Start Kafka
19. Upgrade Kafka node bubuku_cluster.py upgrade --cluster-config $CONFIG \ --ip 10.246.2.11 \ --user adyachkov \ --odd odd-eu-central-1.aruha-test.zalan.do • Deattach EBS • Stop Kafka gracefully • Terminate instance • Run new instance • Attach EBS • Start Kafka
20. Get Kafka Cluster bubuku_cluster.py get --cluster-config $CONFIG 1 Ip 2 Id 3 Volume 4 Availability zone 5 Image Version 172.31.144.11 i-05cf16fc8a4c7467d vol-0ed90a4b5239d73fb eu-central-1b kafka10.1.1 172.31.160.11 i-02405444a936b3e26 vol-0035d9c0a5cc64e2e eu-central-1c kafka10.1.1 172.31.128.12 i-078034bfde0e7d8a3 vol-0e13f305f678a07a8 eu-central-1a kafka10.1.1
21. Impact No worries to lose a node during upgrade 10 mins to upgrade a node 2 hours to upgrade a cluster x42 faster to upgrade
22. Thank you for attention Your questions Article: https://dzone.com/articles/reattaching-kafka-ebs-in-aws https://jobs.zalando.com/tech/blog/reattaching-kafka-ebs-in-aws Slides: https://speakerdeck.com/adyach/reattaching-kafka-ebs-in-aws Scripts: https://github.com/zalando-nakadi/bubuku/tree/master/instance_control