How ScalingData uses Kafka for Event-Oriented Machine Data

Full post here: https://www.hakkalabs.co/articles/scalingdata-uses-kafka-event-oriented-machine-data

Transcript

1. How ScalingData Uses Kafka …for event-oriented machine data. e. sammer - cto @scalingdata esammer@scalingdata.com - @esammer
2. i’m not going to talk about how kafka works.
3. “…try to fill the spaces in between.”* (*obligatory htda reference.)
4. 1. producers -> kafka 2. kafka -> consumers 3. consumers -> storage/processing
5. 1. producers -> kafka 2. kafka -> consumers 3. consumers -> storage/processing message schema? apis? legacy sources? keys? dupes? partitioning? offset management? durability? dupes? dead letter handling? where does transformation go? data center replication? …well, with filtering and stuff?
6. these things aren’t kafka’s responsibility. they depend on… lots of stuff. but you need to think about them. nothing is free.
7. Broker Cluster “events” topic Agent Log4J Appender Java API REST API Syslog Statsd Clients Files / Directories Other Stuff Event Event Event
8. event (avro): { ts: long, event_type_id: int, source: string, body: byte[], attributes: map<string> }
9. Broker Cluster “events” topic HDFS Consumer Solr Consumer HDFS Solr Event Record Doc Custom Stuff Magic ScalingData Application Impala/Hive Spark
10. HDFS Consumer (process) Thread Consumer Kite Writer (Dataset A) Transform Engine Transform Transform Event Avro Object Thread Consumer Kite Writer (Dataset B) Transform Engine Transform Transform Event Avro Object HDFS Consumer (process) Thread Consumer Kite Writer (Dataset A) Transform Engine Transform Transform Event Avro Object Thread Consumer Kite Writer (Dataset B) Transform Engine Transform Transform Event Avro Object Topic A Topic B HDFS Dataset A Data File Data File Dataset B Data File Data File
11. i’m probably out of time by now. …but maybe time for some questions. (we’re hiring.)