Agile Lab - Massive real time analytics with Spark

Transcript

1. Mattia Bertorello
2. Who is Agile Lab? GO BIG (data) or GO HOME
3. Summary •Why streaming matters •Why prediction? •Streaming architecture •Spark streaming •Demo time
4. Why streaming matters ⡥2015 Agile Lab S.r.l.
5. Why streaming matters Data Big Data Business Reaction Batch Analysis Typical BigData Workflow
6. Why streaming matters Data Real Time Processing FASTER REACTIONS ! MORE PROFITS Business Reaction Streaming BigData Workflow
7. Why streaming matters • Fleet Management • Insurance • Recommendation • Etc...
8. Why prediction? ⡥2015 Agile Lab S.r.l.
9. Why prediction? • Rule based categorization and clustering is obsolete • Pattern discovery • Adaptation to fast changing data • Smart thinking: no dummies • Prediction is more valuable
10. Streaming architecture ⡥2015 Agile Lab S.r.l.
11. Streaming architecture Ingestion Layer Processing Layer Serving Layer
12. Spark Streaming
13. Spark Notebook Use Apache Spark straight from the Browser
14. Demo time... ⡥2015 Agile Lab S.r.l.
15. Card transaction analysis PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP Classificazione delle transazioni online/offline PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP | ISONLINE fraud detection algorithm SQL aggregation Generazione di allarmi in tempo reale
16. We are hiring... ⡥2015 Agile Lab S.r.l. paolo.platter@agilelab.it