Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Agile Lab - Massive real time analytics with Spark
Search
Mattia Bertorello
September 22, 2015
Technology
0
120
Agile Lab - Massive real time analytics with Spark
Mattia Bertorello
September 22, 2015
Tweet
Share
More Decks by Mattia Bertorello
See All by Mattia Bertorello
Big Data: a brief introduction using Hadoop
mattiabertorello
1
160
Other Decks in Technology
See All in Technology
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
1
360
Vertex AI を中心に 生成AIのアップデートを共有します
kaz1437
0
310
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
Cloud Native Java with Spring Boot (CNCF Aarhus, April 2024)
thomasvitale
1
170
KubeCon EU 2024 Recap “Kubernetes Policy Time Machine: Where to Next?”
ryysud
0
220
FrontDoorとWebAppsを組み合わせた際のリダイレクト処理の注意点
kenichirokimura
1
530
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
200
VS CodeでAWSを操作しよう
smt7174
8
1.7k
一生覚えておきたい「システム開発=コミュニケーション」〜初めての実務案件振り返りLT〜
maimyyym
1
150
ChatGPT for IT Service Management (IT Pro)
dahatake
7
1.6k
GrafanaMeetup_AmazonManagedGrafanaのアクセス制御機能とマルチテナント環境下でのアクセス制御について
daitak
0
240
自己改善からチームを動かす! 「セルフエンジニアリングマネージャー」のすゝめ
shoota
6
750
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
8
2.4k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
2
1.3k
A designer walks into a library…
pauljervisheath
200
23k
Visualization
eitanlees
136
14k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Building a Scalable Design System with Sketch
lauravandoore
456
32k
How GitHub (no longer) Works
holman
304
140k
Creatively Recalculating Your Daily Design Routine
revolveconf
210
11k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
244
20k
GraphQLとの向き合い方2022年版
quramy
32
12k
The World Runs on Bad Software
bkeepers
PRO
61
6.7k
Transcript
Mattia Bertorello
Who is Agile Lab? GO BIG (data) or GO HOME
Summary •Why streaming matters •Why prediction? •Streaming architecture •Spark streaming
•Demo time
Why streaming matters ⡥2015 Agile Lab S.r.l.
Why streaming matters Data Big Data Business Reaction Batch Analysis
Typical BigData Workflow
Why streaming matters Data Real Time Processing FASTER REACTIONS !
MORE PROFITS Business Reaction Streaming BigData Workflow
Why streaming matters • Fleet Management • Insurance • Recommendation
• Etc...
Why prediction? ⡥2015 Agile Lab S.r.l.
Why prediction? • Rule based categorization and clustering is obsolete
• Pattern discovery • Adaptation to fast changing data • Smart thinking: no dummies • Prediction is more valuable
Streaming architecture ⡥2015 Agile Lab S.r.l.
Streaming architecture Ingestion Layer Processing Layer Serving Layer
Spark Streaming
Spark Notebook Use Apache Spark straight from the Browser
Demo time... ⡥2015 Agile Lab S.r.l.
Card transaction analysis PAN CIFRATO | AMOUNT | DESCRIPTION |
TIMESTAMP Classificazione delle transazioni online/offline PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP | ISONLINE fraud detection algorithm SQL aggregation Generazione di allarmi in tempo reale
We are hiring... ⡥2015 Agile Lab S.r.l.
[email protected]