Storm + Trident and The Holistic Architecture: Using Hadoop for batch and Storm for real time

Yodit Stanton, Freelance Data Scientist, Developer & Systems Architect talk at Data Science London @ds-ldn, 24/0413

Transcript

1. The Holistic Architecture Storm + Hadoop @yoditstanton AGILE Holistic Fool Proof AGILE When you don't know what you are doing ™
2. “The hardest part of data science is understanding a problem,” Hilary Mason chief scientist at bitly
4. DATA SCIENCE Software DARK ARTS Hardware
5. Clever But .... What if you want to be AS clever in Real Time too?
6. Think of the Complexity!!!!
7. Three Layered Architecture brain child of Nathan Marz
8. Batch Layer Records Everything Then Aggregate Append New Data
9. Serving Layer Mock up Experiment Pivot
10. REAL TIME LAYER Previously Complex Trident + Storm Abstracts Complexity
11. Example TridentTopology topology = new TridentTopology(); TridentState wordCounts = topology.newStream("spout1", spout) .each(new Fields("sentence"), new Split(), new Fields ("word")) .groupBy(new Fields("word")) .persistentAggregate(new MemoryMapState. Factory(), new Count(), new Fields("count")) .parallelismHint(6);
12. Simple right? What about the Dark Arts?
13. VowPal Wabbit One One Model to Rule them all
14. Things we are playing with. Online Learning VowPal Wabbit STORM HADOOP BATCH
15. HELLO WORLD
16. Thank you Questions? @yoditstanton