Lightweight and Fault-tolerant: Using compacted topics as a source of truth

Deck presented at the June 2018 Toronto Kafka meetup

Transcript

1. Lightweight and Fault-tolerant: Using compacted topics as a source of truth Elizabeth Giles · PagerDuty
2. Context The Problem of Backpressure Compacted Topics Our Solution
4. Ingestion Service Processing Service Ingestion Service Ingestion Service Processing Service Processing Service Event Event Event Event Event
5. Context The Problem of Backpressure Compacted Topics Our Solution
6. Ingestion Service Processing Service Ingestion Service Ingestion Service Processing Service Processing Service Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event
7. Ingestion Service Processing Service Ingestion Service Ingestion Service Processing Service Processing Service Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Processing Service Ingestion Service HELP!!!
8. Ingestion Service Processing Service Ingestion Service Ingestion Service Processing Service Processing Service Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Processing Service Ingestion Service Customer: Acme Age: 5 minutes Customer: Acme Age: 0 minutes Throttle: Customer: Acme For: 10 minutes To: 20 events/minute
9. Context The Problem of Backpressure Compacted Topics Our Solution
10. What happens to Kafka data?
11. --config cleanup.policy=delete
12. Topic > Partition > Log Segment > Message
13. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10 1 byte
14. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 4 bytes segment.bytes=4, roll.ms=10
15. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 10 ms segment.bytes=4, roll.ms=10 4 bytes
16. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 10 ms segment.bytes=4, roll.ms=10 10 ms 4 bytes
17. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 Current Segment segment.bytes=4, roll.ms=10 10 ms 10 ms 4 bytes
18. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10 S4 S3 S2 S1
19. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.bytes=4 S4 S3 S2 S1
20. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.bytes=4 S4 S3 S2 S1
21. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.bytes=4 S4 S3 S2
22. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.ms=10 S4 S3 S2 S1
23. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.ms=10 S4 S3 S2 S1
24. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=delete, retention.ms=10 S4
25. --config cleanup.policy=compact
26. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1
27. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Clean Dirty Uncleanable Cleanable
28. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Clean Key Last Offset “a” 7 “d” 6
29. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
30. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
31. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
32. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
33. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
34. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S3 S2 S1 Key Last Offset “a” 7 “d” 6
35. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S1
36. Offset 1 2 3 6 7 Key “b” “f” “c” “d” “a” Value 2 3 4 7 8 Time 1 2 3 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact S4 S1
37. segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.cleanable.ratio=0.5 Clean Dirty Clean Dirty Clean Dirty Clean Dirty
38. segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.cleanable.ratio=0.5 Clean Dirty Clean Dirty Clean Dirty Clean Dirty
39. segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.cleanable.ratio=0.5 Clean Dirty Clean Dirty Clean Dirty Clean Dirty
40. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.compaction.lag.ms=20 S4 S3 S2 S1
41. Offset 0 1 2 3 4 5 6 7 Key “a” “b” “f” “c” “d” “a” “d” “a” Value 1 2 3 4 5 6 7 8 Time 0 1 2 3 4 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.compaction.lag.ms=20 S4 S3 S2 S1
42. Offset 1 2 3 5 6 7 Key “b” “f” “c” “a” “d” “a” Value 2 3 4 6 7 8 Time 1 2 3 25 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.min.compaction.lag.ms=20 S4 S3 S1
43. Offset 0 1 2 3 4 5 6 7 8 9 Key “a” “b” “f” “c” “f” “d” “a” “c” “d” “a” Value 1 2 3 4 null 5 6 null 7 8 Time 0 1 2 3 4 5 25 6 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.delete.retention.ms=20 S4 S3 S2 S1
44. Offset 0 1 2 3 4 5 6 7 8 9 Key “a” “b” “f” “c” “f” “d” “a” “c” “d” “a” Value 1 2 3 4 null 5 6 null 7 8 Time 0 1 2 3 4 5 25 6 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.delete.retention.ms=20 S4 S3 S2 S1
45. Offset 0 1 2 3 4 5 6 7 8 9 Key “a” “b” “f” “c” “f” “d” “a” “c” “d” “a” Value 1 2 3 4 null 5 6 null 7 8 Time 0 1 2 3 4 5 25 6 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.delete.retention.ms=20 S4 S3 S2 S1
46. Offset 1 7 8 9 Key “b” “c” “d” “a” Value 2 null 7 8 Time 1 6 36 37 segment.bytes=4, roll.ms=10, cleanup.policy=compact, cleaner.delete.retention.ms=20 S4 S3 S1
47. So… what’s the point?
48. Context The Problem of Backpressure Compacted Topics Our Solution
49. Ingestion Service Processing Service Ingestion Service Ingestion Service Processing Service Processing Service Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Processing Service Ingestion Service Customer: Acme Age: 5 minutes Customer: Acme Age: 0 minutes Throttle: Customer: Acme For: 10 minutes To: 20 events/minute
50. Requirements: 1. Processing service can create throttles 2. Ingestion service can use throttles 3. Resilience to loss of throttle data
51. Compacted Topics!
52. Requirements: 1. Processing service can create throttles 2. Ingestion service can use throttles 3. Resilience to loss of throttle data
53. What about...
54. Ingestion Service Processing Service Event Event Event Event Event
55. Ingestion Service Processing Service Event Event Event Event Event Throttle
56. Ingestion Service Processing Service Event Event Event Event Event Throttle Throttle Throttle
57. Ingestion Service Processing Service Event Event Event Event Event Throttle Throttle Throttle Throttle
58. Questions?