Solid Data Infrastructure with Logs

How to build data infrastructure based on logs.

Talk originally published in PyCon Finland 2015

Transcript

1. Solid Data Infrastructure With Logs
2. Talk overview ‣ How we do things today ‣ Why are there problems ‣ How logs help ‣ What is Kafka? ‣ Kafka with Python
3. Me @nailor / / jyrki@spotify.com
4. Stockholm, May 9th to 10th 2016
5. So, let’s talk about a program written in Scala that runs on JVM at a Python conference. Keynoter, just before the mob with pitchforks
6. Your Average* App *For some definition of average
7. Internet Application DB Search Cache Analytics Data dumps Notifications Web 3.0 Moar stack
9. Complex Nightmare System ‣ Complicated data flows ‣ Inconsistencies creeping all over the place ‣ Price to introduce new systems
10. Price to Include New System ‣ Data bootstrapping? ‣ Modifications to existing systems?
11. OK! OK! A A Is the Data Consistent? Database Cache A A OK! A A B B OK! B B OK! A A OK!
12. How to fix?
13. How about a log based data structure? “wat?”
14. Think the Database You Have ‣ Data gets inserted into a transaction log first ‣ Data structures get mutated after ‣ If a crash happens, it can recover from the log ‣ The log is the source of truth ‣ Your database is a log based data structure!
15. Enter Kafka!
16. “But I already got a database?” - The Lonely Developer
17. What’s Kafka? ‣ Developed by LinkedIn ‣ Written in Scala, runs on JVM ‣ Distributed, partitioned, replicated commit log service ‣ Maintains feeds of messages in categories (topics)
18. Terminology: Broker ‣ A single server in a Kafka cluster ‣ Broker metadata is stored in ZooKeeper ‣ Automatic leader election and failover
19. Terminology: Topics and Logs ‣ Topic is a single logical binder for logs, a feed of messages ‣ Topic maintains a partitioned log ‣ Log size is controlled by time based retention policy ‣ Logs can be compacted by key
20. Terminology: Partition ‣ Allows scaling a topic past a single server ‣ Kafka guarantees order within the partition ‣ Allows parallel consumption of a log
21. Terminology: Message ‣ A single binary entry in the log ‣ Free format, though usually some encoding (JSON, Avro) is used ‣ Belongs to a partition ‣ Is keyed for compaction
22. Terminology: Producer ‣ Creates messages into the queue ‣ Controls topic, key and partition of the message
23. Terminology: Consumer Group ‣ Consumes a single topic ‣ Multiple groups can consume same topic ‣ Kafka assigns partitions within the topic to consumer within the consumer group ‣ Guarantees consuming of partitions in order within the consumer group - something most other MQs don’t!
24. Terminology: Consumer ‣ Consumes a single partition in a topic ‣ Can consume in any order ‣ Can reset the offset ‣ Are grouped in consumer groups
25. Life of a Single Topic in Kafka Old New #0 #0 #0 #1 #1 #1 #2 #2 #2 #3 #3 #4 Partition 0 Partition 1 Partition2 Producer Writes
26. Nice stuff ‣ Easy to scale horizontally ‣ Guaranteed order of processing ‣ Producer has full control over the partitioning within a topic ‣ Data is retained forever if wanted ‣ …or limited by time ‣ …or compacted ‣ … or all combined!
27. Why Kafka? Why not <another MQ>? ‣ Because I like it. ‣ Seriously, it really fits this use case. ‣ Super good at scaling up (if you need to care about that)
28. So, How to Build Around Kafka?
29. Internet Application DB Search Cache Analytics Data dumps Notifications Web 3.0 Moar stack
30. Internet Application DB Search Cache Analytics Data dumps Notifications Web 3.0 Moar stack K
 a
 f
 k
 a
32. Why is this design soooooo nice?
33. Introducing new data sources Kafka Producer Search v1 Search v2 1. Consume from the beginning 2. Start streaming 3. Switch over
34. Introducing new data sources becomes a cheap operation! ‣ Cache reload / warm up? No prob! ‣ New DB schema? No prob! ‣ New component? No prob!
35. Ok, how About Python and Kafka?
36. DEMO TIME!
37. That’s it! Thanks :) @nailor / jyrki@spotify.com
38. Useful links! ‣ kafka.apache.org ‣ martin.kleppmann.com/2015/05/27/ logs-for-data-infrastructure.html ‣ github.com/nailor/kafka-python-demo ‣ kafka-python.readthedocs.org/ ‣ Slides are here: speakerdeck.com/nailor