Big Data Computation Architecture

It talks about the evolution of big data computation architecture

Transcript

1. Evolution of Big Data Architecture Gang Tao
2. Computing Trend
4. Non Functional Requirements • Latency • Throughput • Fault Tolerant • Scalable • Exactly Once Semantic
5. Hadoop
6. Hadoop Eco System
7. Map Reduce Move computation to Data
8. Map Reduce
9. Hadoop the Limitation • Map/Reduce is hard to use • Latency is high • inevitable data movement
10. Lambda
11. Lambda Architecture query = function(all data)
12. Design Principle human fault-tolerance – the system is unsusceptible to data loss or data corruption because at scale it could be irreparable. data immutability – store data in it’s rawest form immutable and for perpetuity. (INSERT/ SELECT/DELETE but no UPDATE !) recomputation – with the two principles above it is always possible to (re)-compute results by running a function on the raw data.
13. Lambda Architecture
14. Batch Processing
15. Batch Processing Using only batch processing, leaves you always with a portion of non- processed data.
16. Realtime Stream Processing
17. query = function(all data) + function(new data)
18. Lambda Architecture
19. A Reference Case
20. Lambda the Bad? What is Good? What is Bad? Inmutable Data , Reprocessing Keeping code written in two different systems perfectly in sync
21. Over complicated Lambda
22. Batch VS. Stream
23. Stream & Realtime
24. Storm Logic View Bolts Spouts Move data to Computation
25. Storm Deployment View
26. Code Compare
27. DAG • DAG : Directed Acyclic Graph • Used in Spark, Storm, Flink etc.
28. Out of Control Build Complexity with Simplicity
29. Stream Processing Model One at a time Micro batch
30. Stream Processing Model One at a time Micro Batch Low Latency Y N High Throughput N Y at least once Y Y excatly once Sometimes Y simple programing model Y N
31. Stream Computing the Limitation • Queries must be written before data • There should be another way to query past data • Queries cannot be run twice • All results will be lost when any error occurs All data have gone when bugs found • Disorders of events break results • Recorded time based queries? Or arrival time based queries?
32. Batch/Stream Unification
33. Stream with Spark
34. Apache Flink
35. Flink Distributed Snapshot
36. Fault Tolerance in Stream • At Least Once : ensure all operators see all events • Stream -> Replay on failure • Exactly Once : • Flink : distributed Snapshot • Spark : Micro Batch
37. Jay Kreps
38. Kappa Architecture