Apache_Gearpump_-_Strata_Beijing_2016.pdf

Transcript

1. Apache Gearpump next-gen streaming engine Manu Zhang, mauzhang@apache.org Sean Zhong, sean-zhong@apache.org
2. What is Gearpump ? • An Akka[2] based real-time streaming engine • An Apache Incubator[1] project since Mar.8th, 2016 • A super simple pump that consists of only two gears but very powerful at streaming water
3. Gearpump Architecture • Actor concurrency • Message passing communication • error handling and isolation with supervision hierarchy • Master HA with Akka Cluster
4. Why Gearpump ?
5. Stream processing is hard but no system meets all requirements • Infinite data • Out-of-order data • Low latency assurance (e.g real-time recommendation) • Correctness requirement (e.g. charge advertisers for ads) • Cheap to update user logics
6. How Gearpump solves the hard parts • User interface • Flow control • Out-of-order processing • Exactly once • Dynamic DAG
7. Use case - Detect anomaly words 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink
8. How Gearpump solves the hard parts • User Interface • Flow control • Out-of-order processing • Exactly Once • Dynamic DAG
9. User Interface - DAG API Graph( KafkaSource ~ ShufflePartitioner ~> WindowCounter, WindowCounter ~ HashPartitioner ~> ModelCalculator, WindowCounter ~ HashPartitioner ~> AnomalyDetector, ModelCalculator ~> HashPartitioner ~> AnomalyDetector, AnomalyDetector ~> ShufflePartitioner ~> KafkaSink)
10. partitioner User Interface - Partitioner Graph( KafkaSource ~ ShufflePartitioner ~> WindowCounter, WindowCounter ~ HashPartitioner ~> ModelCalculator, WindowCounter ~ HashPartitioner ~> AnomalyDetector, ModelCalculator ~ HashPartitioner ~> AnomalyDetector, AnomalyDetector ~ ShufflePartitioner ~> KafkaSink) WindowCounter (parallelism = 3) ModelCalculator (parallelism = 2) Task Task Task Task Task
11. User Interface - Compose DAG
12. How Gearpump solves the hard parts • User interface • Flow control • Out-of-order processing • Exactly once • Dynamic DAG
13. Flow Control 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink
14. Without Flow Control - messages queue up KafkaSource WindowCounter ModelCalculator fast fast very slow
15. Without Flow Control - OOM KafkaSource WindowCounter ModelCalculator fast fast very slow
16. With Flow Control - Backpressure KafkaSource WindowCounter ModelCalculator fast slow down very slow backpressure
17. With Flow Control - Backpressure KafkaSource WindowCounter ModelCalculator Slow down Slow down Very Slow backpressure backpressure pull slower
18. How Gearpump solves the hard parts • User Interface • Flow control • Out-of-order processing • Exactly Once • Dynamic DAG
19. Out-of-order data Event time 3 2 1 Processing time • Event time - when data generated • Processing time - when data processed 1 2 3 4 5 6
20. Window Count 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink
21. When can window counts be emitted ? window messages count [0, 2) 1 [2, 4) 2 [4, 6) 2 (“gearpump”, 1) WindowCounter In-memory Table • No window can be emitted since message as early as time 1 has not arrived (“gearpump”, 1) (“gearpump”, 3) (“gearpump”, 5) (“gearpump”, 4) (“gearpump”, 2) WindowCounter
22. On Watermark[4] Event time 3 2 1 Processing time 1 2 3 4 5 6 watermark • No timestamp earlier than watermark will be seen
23. Out-of-order processing with watermark window messages count [0, 2) 1 [2, 4) 2 [4, 6) 2 (“gearpump”, 1) WindowCounter In-memory Table (“gearpump”, 1) (“gearpump”, 3) (“gearpump”, 5) (“gearpump”, 4) (“gearpump”, 2) WindowCounter watermark = 0， No window can be emitted
24. Out-of-order processing with watermark window messages count [0, 2) 1 [2, 4) 2 [4, 6) 2 (“gearpum p”, [0, 2), 1) WindowCounter In-memory Table (“gearpump”, 1) (“gearpump”, 3) (“gearpump”, 5) (“gearpump”, 4) (“gearpump”, 2) WindowCounter watermark = 2， Window [0, 2) can be emitted (“gearpump”, [0, 2), 1)
25. How to get watermark ? 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink
26. From upstream 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink watermark
27. More on Watermark • Source watermark defined by user • Usually heuristic based • Users decide whether to drop data arriving after watermark
28. How Gearpump solves the hard parts • User Interface • Flow control • Out-of-order processing • Exactly once • Dynamic DAG
29. Exactly Once - no Lost or Duplicated States • Lost states can cause anomaly words skipped • Duplicated states can cause false alarm States 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink kafka_offset window_counts models
30. Exactly Once with asynchronous checkpointing (2, kafka_offset) KafkaSource WindowCounter ModelCalculator Watermark = 2 Watermark = 0 Watermark = 0 (2, kafka_offset)
31. (2, kafka_offset) (2, window_counts) Exactly Once with asynchronous checkpointing KafkaSource WindowCounter ModelCalculator Watermark = 2 Watermark = 2 Watermark = 0 (2, window_counts)
32. (2, kafka_offset) (2, window_counts) (2, models) Exactly Once with asynchronous checkpointing KafkaSource WindowCounter ModelCalculator Watermark = 2 Watermark = 2 Watermark = 2 (2, models)
33. (2, kafka_offset) (2, window_counts) (2, models) Exactly Once with asynchronous checkpointing KafkaSource WindowCounter ModelCalculator Watermark = 2 Watermark = 2 Watermark = 2 (2, models)
34. (2, kafka_offset) (2, window_counts) (2, models) Crash KafkaSource WindowCounter ModelCalculator Watermark = 2 Watermark = 2 Watermark = 2 (2, models)
35. (2, kafka_offset) (2, window_counts) (2, models) Recover to latest checkpoint at 2 KafkaSource WindowCounter ModelCalculator models window_counts kafka_offset Replay from kafka Get state at 2
36. More on Exactly Once • User extends PersistentState interface and system checkpoints state for users automatically • Watermark is persisted on master
37. How Gearpump solves the hard parts • User Interface • Flow control • Out-of-order processing • Exactly Once • Dynamic DAG
38. Can we update model algorithm in a cheap way ? 1. Words 3. W indow Counts 2. Window Counts 4. Models 5. Anom aly words KafkaSource WindowCounter ModelCalculator AnomalyDetector KafkaSink The model algorithm doesn’t perform well
39. Dynamic DAG - modify application at runtime
40. Performance
41. Yahoo! Streaming Benchmark[5]
42. Yahoo! Streaming Benchmark • 4 node cluster • Each node has 32-core, 64 GB memory • 10GB Ethernet
43. Advanced features
44. DAG Visualization • Watermark • Node size reflects throughput • Edge width represents flow rate
45. DAG Visualization • Data skew analysis
46. Apache Beam[6] Gearpump Runner Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other Languages Beam Java Beam Python Execution Execution Cloud Dataflow Execution Apache Gearpump
47. Experimental features • Web UI Authorization / OAuth2 Authentication • CGroup Resource Isolation • Binary Storm compatibility • Akka Streams integration
48. Summary • Gearpump is good at streaming infinite out-of-order data and guarantees correctness • Gearpump helps users to easily program streaming applications, get runtime information and update dynamically
49. References 1. gearpump.apache.org 2. akka.io 3. http://www.slideshare.net/SeanZhong/strata-singapore-gearpumpreal-ti me-dagprocessing-with-akka-at-scale 4. https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p734-akid au.pdf 5. https://yahooeng.tumblr.com/post/135321837876/benchmarking-streami ng-computation-engines-at 6. Apache Beam [project overview]
50. Backup Slides
51. What is Akka • Message driven • Shared nothing • Location transparent • Supervision based failure management • High performance Actor Actor Actor Actor Actor spawn