Apache Storm Concepts

A 2015 concepts overview for the Apache Storm project and the challenges for a Streaming Analysis solution.

Transcript

1. Apache Storm Concepts André Dias andre.dias@summa.com.br
2. Highlights • Background • Storm’s History • Concepts • Integrations • Real Cases • Q&A
3. Background
4. Background Big Data V’s
5. Background Big Data V’s Volume Petabytes / Terabytes Velocity Real-time / Near Real-time Variety Sensors, Blog Posts, Logs, Social Networks...
6. Background Data Streaming
7. Background Data Streaming
8. Background Data Streaming Now Please… I’m in traffic…
9. • Discovery • Ingest • Process • Persist • Analyze • Expose Background Data Value Chain
10. • Discovery • Ingest • Process • Persist • Analyze • Expose Background Data Value Chain (Storm Focus)
11. Background Data Value Chain • Discovery • Ingest • Process • Persist • Analyze • Expose
12. Background Data Value Chain (Ingest) Stream Processing Data in Motion Batch Processing
13. • Data processing architecture ◦ Generic ◦ Scalable ◦ Fault-tolerant (human/hardware) • Low latency Background Lambda Architecture (LA)
14. Background Lambda Architecture (LA)
15. Storm’s History Nathan Marz • Backtype (2008), acquired by Twitter (2011) • Lambda Architecture Creator • September, 2011. Storm Creation • September, 2013. Storm entered to ASF
16. Why Use It? • Scalable, as Hadoop • No data loss, reliable • Fault tolerant • Language agnostic • Real-time, real fast
17. Why Use It? Storm is TLP !!
18. Concepts
19. Concepts • Tuple • Streams • Spouts • Bolts • Topologies / Trident API • Stream Groupings
20. Concepts Spouts • Source of Streams • Data Consumers (Ingestion) • Emits Tuples
21. Concepts Bolts • Units of Work to tuples • Data streaming logic • Can emit tuples as well • Data store integration
22. Concepts Topology • Data Streaming Flow Representation • DAG (Direct Acyclic Graph) of Spouts and Bolts • Streaming computation • Each node as individual task (parallel execution) • Stateless
23. Concepts Trident API • Abstraction Layer over low-level Storm API • More Complex Topologies • Stateful • Micro-batch • High-level API (similar to Pig / Cascading - Hadoop) • Message processed at least once (guaranteed)
24. Concepts Trident API - Micro Batch • Trident Batches ◦ are Ordered
25. Concepts Trident API - Micro Batch • Trident Batches ◦ can be Partitioned
26. Concepts Streaming Groups • Data Flow Control over Topologies
27. Architecture
28. Architecture Components - Nimbus • Master node (similar to JobTracker) • Monitor and distribute the processing workload across worker nodes • Stores all its data into Zookeeper
29. Architecture Components - Supervisor • Worker node (similar to TaskTracker) • Monitor and distribute the processing workload across worker nodes • Stores all its data into Zookeeper
30. Architecture Overview • Master-slave approach • Cluster coordination (Zookeeper) • Nimbus HA
31. Integrations
32. Real Cases Collector sensor information to a Data Lake Micro-batch user contents, content feeds and application logs Real-time user music recommendations
33. Q&A
34. THANK YOU! Use Storm