Kafka Producers
• Producers are responsible to chose
what topic to publish data to
• The producer is responsible for
choosing a partition to write to
• Can be handled round robin or
partition functions
Slide 15
Slide 15 text
Kafka Consumers
• Consumption can be done via:
• queuing
• pub-sub
Slide 16
Slide 16 text
Kafka Consumers
• Kafka consumer group
• Strong ordering
Slide 17
Slide 17 text
Kafka Consumers
• Strong ordering
Slide 18
Slide 18 text
https://github.com/opentable/puppet-exhibitor
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
Iteration 2
Introduction of Kafka
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
Iteration 3
Further ‘Improvements’ to the cluster layout
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
The Numbers
• Logs kept in ES for 30 days then archived
• 12 billion documents active in ES
• ES space was about 25 - 30TB in EBS volumes
• Average Doc Size ~ 1.2KB
• V-Day 2015: ~750M docs collected without failure
Slide 26
Slide 26 text
What about metrics and
monitoring?
Slide 27
Slide 27 text
Monitoring - Nagios
• Alerts on
• ES Cluster
• zK and Kafka Nodes
• Logstash / Redis nodes