Introducing Decaton

Transcript

1. Introducing Decaton LINE Corporation
 Haruki Okada
2. • Haruki Okada • Software engineer at LINE Corporation • Developing company-wide Apache Kafka platform • Decaton maintainer Speaker
3. • https://github.com/line/decaton • Asynchronous task processing library built on top of Apache Kafka • Battle tested in various LINE services for several years • Open sourced in March, 2020 • Various features which make task processing development simple • Can process single partition in multiple threads • Contribute to improve consuming throughput especially for I/O intensive tasks What’s Decaton?
4. • One of typical Kafka usage at LINE • I/O intensive background tasks Asynchronous task processing Task processor Web API Web API Kafka Produce tasks
 including external I/O Process asynchronously Storage
5. Why not plain KafkaConsumer? ɾɾɾ | 5 | 4 | 3 | 2 | 1 Partition poll() loop Partition Consumer Thread
6. Sequential processing ɾɾɾ | 5 | 4 | 3 | 2 | 1 doProcess(1) doProcess(2) doProcess(3) ɾ ɾ ɾ Consumer thread Partition Process latency • Processing model in major frameworks: • Kafka Streams • Spring Kafka
7. Consumer throughput is a problem • If a process includes I/O with 10ms latency => 100 tasks / second at max Per partition throughput = 1 process latency per record
8. • It’s difficult to estimate required concurrency from the beginning • However, • Adding partitions often requires contacting cluster administrator • Adding partitions has side effects • Message ordering breaks temporarily • More partitions tends to generate smaller producer batches • Number of open file descriptors / memory-mapped files • Not preferable in LINE circumstance • i.e. Single, multi-tenant shared Kafka cluster Why not adding more partitions?
9. Why not just process async?
10. At least once is broken | 5 | 4 | 3 | 2 | 1 | 5 | 4 | 3 | 2 | 1 => 1. Fetch 1 to 5 by poll() 2. Submit tasks to async executor => 3. Commit latest offset “5”
 as poll() is already completed => 5. Consumer crash !!! => 6. Other instance takes over
 from offset “5” => 4. Process async • => LOST offset 3,4
11. Why not just commit in batch?
12. Head-of-line blocking is a problem | 5 | 4 | 3 | 2 | 1 | 10 | 9 | 8 | 7 | 6 ɾɾɾ Waiting single outlier “4”
 to complete Records returned by poll() (Current batch) Blocked…. Partition
13. Ideal commit management Continue processing without waiting “4" Commit “3" ɾɾɾ | 5 | 4 | 3 | 2 | 1 | 10 | 9 | 8 | 7 | 6 ɾɾɾ Partition Records returned by poll() ɾɾɾ
14. Ideal commit management ɾɾɾ | 5 | 4 | 3 | 2 | 1 | 10 | 9 | 8 | 7 | 6 ɾɾɾ Partition Records returned by poll() ɾɾɾ In Decaton, this offset is called “Watermark” i.e. The highest offset that all preceding offsets are already processed
15. What we need is: • Mechanism to track watermarks • Note that any offset could be completed by several threads in arbitrary order | 5 | 4 | 3 | 2 | 1 Processor thread Processor thread ɾ ɾ ɾ
16. How? • We can calculate watermarks by: • Iterating over completed offsets in ascending order, then check continuity
 by mapping to fetched offsets • i.e. Sort out-of-order offset completion by its offset • => Priority queue ? 4 | 2 | 1 ɾɾɾ | 5 | 4 | 3 | 2 | 1 Completed offsets Fetched offsets Watermark
17. Initial approach: Priority queue 4 | 2 | 1 ɾɾɾ | 5 | 4 | 3 | 2 | 1 Consumer thread 1. Register fetched offsets as “pending” Processor thread Processor thread 2. Submit tasks 5,3,1 4,2 3. Put completed offsets concurrently
 (priority queue) 4. zip offsets
 => watermark = 2
18. Deployed to the production Kafka Web API Decaton processors API servers 1 million / sec tasks HBase Put
19. Watermark tracking
 became bottleneck ObjectMonitor
20. Lock was necessary 4 | 2 | 1 Processor thread Processor thread lock() lock() ɾɾɾ | 5 | 4 | 3 | 2 | 1 • To protect priority queue from concurrent mutation • Appending to the queue • Remove offsets until a watermark • To keep queue length finite
21. How can we improve? 4 | 2 | 1 Processor thread Processor thread lock() lock() • Problem: mutating shared object (priority queue) by multiple processor threads
22. Get rid of mutating shared object • Each processor thread only mutates the state of the offsets it’s responsible for Processor thread Processor thread | 1 | | 2 | | 4 | | 5 | | 3 | markComplete() markComplete() Pending offsets
23. Revised:
 Lock-free approach Consumer thread Processor thread Processor thread Fixed length
 ring buffer 1. Initialize states for fetched offsets (1..5)
 as “not completed” & set watermark pointer 0 1 2 3 4 5 6 7
24. Revised:
 Lock-free approach Consumer thread 0 1 2 3 4 5 6 7 2. Submit tasks 5,3,1 4,2 4. Advance
 watermark pointer 3. Mark offsets as “complete” Processor thread Processor thread
25. No longer a bottleneck
26. • Optimized watermark tracking is the core of Decaton • Enables multi-threaded processing per single partition • With minimizing the impact of head-of-line blocking Summary
27. • In Decaton, process ordering guarantee is relaxed from “per partition” to “per key” Ordering semantics Processor thread Processor thread ɾɾɾ | 5 | 4 | 3 | 2 | 1 | a b a b a key: Partition | 5 | 3 | 1 | a a a Internal queue | 4 | 2 | b b
28. Benchmark comparison 0 500 1000 1500 2000 2500 3000 Kafka Streams Spring Kafka Decaton Decaton (10 threads / partition) https://github.com/ocadaruma/decaton-benchmark-comparison Throughput (msg/sec) • Benchmark condition: • Process latency: 10ms per task • Partition count: 3 • In Decaton, throughput can be scaled
 linearly as increasing threads
29. • 1. Define task protocol • 2. Implement task producer • 3. Implement task processor Getting started with Decaton
30. • Protocol between task producer and processor • Can be defined in arbitrary format • Protobuf as an example 1. Define task protocol
31. 2. Implement task producer
32. 3. Implement task processor • That’s it. Simple enough isn’t it?
33. • Decaton provides various useful features in task processing • From requirements for real-world LINE service development • Rate limiting • Retry queueing • And more Built-in features
34. Rate limiting Decaton processor Web API Web API Kafka Sudden traffic spike • Abusing • Big campaign External web service Call Throttle processing rate Buffering on Kafka
35. • Can be enabled by only few lines of code Rate limiting
36. Retry queueing • Retry until succeeds ? • => Could block subsequent tasks
 (Though the impact is mitigated thanks to
 Decaton’s commit management model) • Just give up the task? • => Not preferable Decaton processor Storage
 ( high load) Fails intermittently
37. Retry queueing Decaton processor Storage
 ( high load) Fails intermittently Kafka Produce retry task Retry after backoff
38. Retry queueing • 1. Prepare retry topic in advance • 2. Enable retry queueing feature in processor
39. • Decaton is a battle tested Kafka consumer framework • Suites for I/O intensive workload the most • Provides various features which suit for many situations in task processing • Give it a try! • Feedbacks and contributions are welcome Conclusion
40. Thank you