Apache Beam and Dataflow

Transcript

1. Apache Beam and Google Cloud Dataflow Mete Atamel Developer Advocate at Google @meteatamel @meteatamel
2. Apache Beam: A single unified model for batch and stream data processing @meteatamel
3. MapReduce: Batch Processing (Prepare) Map (Shuffle) (Produce) Reduce @meteatamel
4. Confidential & Proprietary Google Cloud Platform 4 MapReduce BigTable Dremel Colossus Flume Megastore Spanner PubSub Millwheel Cloud Dataflow Cloud Dataproc @meteatamel
5. 2012 2002 2004 2006 2008 2010 MapReduce GFS Big Table Dremel Pregel FlumeJava Colossus Spanner 2014 MillWheel Dataflow 2016 Data Processing @ Google @meteatamel
6. FlumeJava: Easy and Efficient MapReduce Pipelines • Higher-level API with simple data processing abstractions. ◦ Focus on what you want to do to your data, not what the underlying system supports. • A graph of transformations is automatically transformed into an optimized series of MapReduces. @meteatamel
7. // Collection of raw events PCollection<SensorEvent> raw = ...; Example: Computing mean temperature @meteatamel // Element-wise extract location/temperature pairs PCollection<KV<String, Double>> input = raw.apply(ParDo.of(new ParseFn())) // Composite transformation containing an aggregation PCollection<KV<String, Double>> output = input .apply(Mean.<Double>perKey()); // Write output output.apply(BigtableIO.Write.to(...));
8. @meteatamel
9. ...big data... @meteatamel
10. ...really, really big... Tuesday Wednesday Thursday @meteatamel
11. Batch failure mode #1 Latency @meteatamel
12. MapReduce Tuesday Wednesday Batch failure mode #2: Sessions Jose Lisa Ingo Asha Cheryl Ari Wednesday Tuesday @meteatamel
13. Continuous & Unbounded 9:00 8:00 14:00 13:00 12:00 11:00 10:00 2:00 1:00 7:00 6:00 5:00 4:00 3:00 @meteatamel
14. Historical events Exact historical model Periodic batch processing Approximate real-time model Stream processing system Continuous updates
15. 2012 2002 2004 2006 2008 2010 MapReduce GFS Big Table Dremel Pregel FlumeJava Colossus Spanner 2014 MillWheel Dataflow 2016 Data Processing @ Google @meteatamel
16. MillWheel: Streaming Computations • Framework for building low-latency data-processing applications • User provides a DAG of computations to be performed • System manages state and persistent flow of elements @meteatamel
17. Streaming Patterns: Element-wise transformations 13:00 14:00 8:00 9:00 10:00 11:00 12:00 Processing Time
18. Streaming Patterns: Aggregating Time Based Windows 13:00 14:00 8:00 9:00 10:00 11:00 12:00 Processing Time
19. Streaming Patterns: Event-Time Based Windows Event Time Processing Time 11:00 10:00 15:00 14:00 13:00 12:00 11:00 10:00 15:00 14:00 13:00 12:00 Input Output
20. Streaming Patterns: Session Windows Event Time Processing Time 11:00 10:00 15:00 14:00 13:00 12:00 11:00 10:00 15:00 14:00 13:00 12:00 Input Output
21. Formalizing Event-Time Skew Watermarks describe event time progress. "No timestamp earlier than the watermark will be seen" Often heuristic-based. Too Slow? Results are delayed. Too Fast? Some data is late.
22. 1 + 1 = 2 Completeness $$$ Cost Latency Streaming or Batch? Tradeoffs
23. Dataflow Model One model unifying batch and streaming @meteatamel
24. What are you computing? Where in event time results are calculated? When in processing time are results materialized? How do refinements relate?
25. What are you computing? What Where When How Element-Wise Aggregating Composite ParDo GroupByKey, Combine ParDo + Count + ParDo
26. What: Computing Integer Sums // Collection of raw log lines PCollection<String> raw = IO.read(...); What Where When How // Element-wise transformation into team/score pairs PCollection<KV<String, Integer>> input = raw.apply(ParDo.of(new ParseFn()); // Composite transformation containing an aggregation PCollection<KV<String, Integer>> scores = input.apply(Sum.integersPerKey());
27. What: Computing Integer Sums What Where When How
28. What: Computing Integer Sums What Where When How
29. Windowing divides data into event-time-based finite chunks. Often required when doing aggregations over unbounded data. Where in event time? What Where When How Fixed Sliding 1 2 3 5 4 Sessions 2 4 3 1 Key 2 Key 1 Key 3 Time 1 2 3 4
30. Where: Fixed 2-minute Windows What Where When How PCollection<KV<String, Integer>> scores = input .apply(Window .into(FixedWindows.of(Duration.standardMinutes(2))) .apply(Sum.integersPerKey());
31. Where: Fixed 2-minute Windows What Where When How
32. When in processing time? What Where When How • Triggers control when results are emitted. • Triggers are often relative to the watermark.
33. When: Triggering at the Watermark What Where When How PCollection<KV<String, Integer>> scores = input .apply(Window .into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark())) .apply(Sum.integersPerKey());
34. When: Triggering at the Watermark What Where When How
35. When: Early and Late Firings What Where When How PCollection<KV<String, Integer>> scores = input .apply(Window .into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1)))) .apply(Sum.integersPerKey());
36. When: Early and Late Firings What Where When How
37. How do refinements relate? What Where When How • How should multiple outputs per window accumulate? • Should we emit the running sum, or only the values that have come in since the last result? (Accumulating & Retracting not yet implemented in Apache Beam.)
38. How: Add Newest, Remove Previous What Where When How PCollection<KV<String, Integer>> scores = input .apply(Window .into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1))) .accumulatingFiredPanes()) .apply(Sum.integersPerKey());
39. 1.Classic Batch 2. Batch with Fixed Windows 3. Streaming 5. Streaming With Accumulations 4. Streaming with Speculative + Late Data Customizing What When Where How What Where When How
40. Dataflow to Apache Beam Evolution of Dataflow into Apache Beam @meteatamel
41. The Dataflow Model & Cloud Dataflow Dataflow Model & SDKs A unified model for batch and stream processing No-ops, fully managed service Google Cloud Dataflow
42. a unified model for batch and stream processing supporting multiple runtimes A great place to run Beam Apache Beam Google Cloud Dataflow The Dataflow Model & Cloud Dataflow Beam
43. 1. The Beam Model: What / Where / When / How 2. SDKs for writing Beam pipelines -- starting with Java 3. Runners for Existing Distributed Processing Backends • Apache Flink (thanks to data Artisans) • Apache Spark (thanks to Cloudera) • Google Cloud Dataflow (fully managed service) • Local (in-process) runner for testing What is Part of Apache Beam?
44. Apache Beam Vision: Mix/Match SDKs & runtimes • The Beam Model: the abstractions at the core of Apache Beam Language B SDK Language A SDK Language C SDK Runner 1 Runner 3 Runner 2 • Choice of SDK: Users write their pipelines in a language that’s familiar and integrated with their other tooling • Choice of Runners: Users choose the right runtime for their current needs -- on-prem / cloud, open source / not, fully managed / not • Scalability for Developers: Clean APIs allow developers to contribute modules independently The Beam Model Language A Language C Language B The Beam Model
45. Apache Beam Vision: as of March 2017 • Beam’s Java SDK runs on multiple runtime environments, including: ◦ Apache Apex ◦ Apache Spark ◦ Apache Flink ◦ Google Cloud Dataflow ◦ [in development] Apache Gearpump • Cross-language infrastructure is in progress. ◦ Beam’s Python SDK currently runs on Google Cloud Dataflow Beam Model: Fn Runners Apache Spark Cloud Dataflow Beam Model: Pipeline Construction Apache Flink Java Java Python Python Apache Apex Apache Gearpump
46. How do you build an abstraction layer? Apache Spark Cloud Dataflow Apache Flink ???????? ????????
47. Beam: the intersection of runner functionality?
48. Beam: the union of runner functionality?
49. Beam: the future!
50. Categorizing Runner capabilities http://beam.incubator.apache.org/ documentation/runners/capability-matrix/
51. Confidential & Proprietary Google Cloud Platform 51 @meteatamel Data Processing with Apache Beam MapReduce BigTable Dremel Colossus Flume Megastore Spanner PubSub Millwheel Cloud Dataflow Cloud Dataproc Apache Beam
52. Confidential & Proprietary Google Cloud Platform 52 52 cloud.google.com/dataflow beam.apache.org @ApacheBeam Mete Atamel @meteatamel Thank You @meteatamel