Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
To Stream or Not To Stream? The Landscape of Online Analytics
Search
Gianmarco De Francisci Morales
May 28, 2015
Research
1
300
To Stream or Not To Stream? The Landscape of Online Analytics
EVAM Solution Day 2015, Istanbul.
Gianmarco De Francisci Morales
May 28, 2015
Tweet
Share
More Decks by Gianmarco De Francisci Morales
See All by Gianmarco De Francisci Morales
Repurpose, Reuse, Recycle the building blocks of Machine Learning
gdfm
0
50
How I Learned to Stop Worrying and Love the Risk
gdfm
0
110
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
110
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
220
Quantifying and Reducing Controversy in Social Media
gdfm
0
130
Big Data Streams: The Next Frontier
gdfm
2
290
Mining Big Data Streams: Better Algorithms or Faster Systems?
gdfm
0
450
SAMOA @Strata Barcelona 2014
gdfm
2
630
Distributed Adaptive Model Rules for Mining Big Data Streams
gdfm
1
150
Other Decks in Research
See All in Research
仮説検定とP値
shuntaros
6
7.3k
LayerXにおけるAI・機械学習技術の活用と展望 / layerx-ai-jsai2024
shimacos
2
2.5k
論文紹介 AST: Audio Spectrogram Transformer
kazu07
0
190
SSII2024 [PD] 画像センシングの未来
ssii
PRO
0
290
RCEへの近道
kawakatz
1
620
Engineering LaCAM∗: Towards Real-Time, Large-Scale, and Near-Optimal Multi-Agent Pathfinding
kei18
0
460
CARA MEMBUKA VIDEO DEWASA DI INDONESIA
bloglangit
0
320
一人称視点映像解析の基礎と応用(CVIMチュートリアル)
takumayagi
0
800
Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction(日本語版)
aiueola
0
120
機械学習を用いたポケモン対戦選出予測
fufufukakaka
1
560
機械学習と最適化の融合動的ロットサイズ決定問題を例として
mickey_kubo
2
360
1on1ガイドへの想い(chachaki編)
chachakix
0
150
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
224
21k
Rails Girls Zürich Keynote
gr2m
93
13k
Become a Pro
speakerdeck
PRO
15
4.8k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
35
6.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.5k
Designing for Performance
lara
604
67k
How STYLIGHT went responsive
nonsquared
93
5k
Agile that works and the tools we love
rasmusluckow
325
20k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
19k
Clear Off the Table
cherdarchuk
89
320k
Happy Clients
brianwarren
94
6.5k
Writing Fast Ruby
sferik
623
60k
Transcript
To Stream or Not To Stream? The Landscape of Online
Analytics Gianmarco De Francisci Morales
[email protected]
@gdfm7
None
5 Questions What? Why? How? Where? When?
What? Data Stream
Text Big Data Too big to handle
Text Big Data Streams Drinking from a firehose
Stream Analytics Batch data = snapshot of streaming data Descriptive
Predictive Prescriptive
Value of Data
Online vs Real-Time
Why? Motivation and Goal
–Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside
a company is some new information comes in and the company reacts to that asynchronously.” Asynchronous Processing
Nervous System vs Silos
Perishable Insights Great instantaneous value Ephemeral Opportunity cost
Hype Cycle
Hype Cycle
How? Stream Processing Architecture
Architecture Overview
Ingestion Plethora of solutions Still ad-hoc (read: messy) Schema evolution:
Avro Column-store: Parquet Log collection: Flume
Brokerage
Processing PE PE Input Stream PEI PEI PEI PEI PEI
Output Stream Event routing
Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive
callbacks
Example: Reactive Web App
Lambda vs Kappa
Where? Applications
Application Domains Industrial applications Telecommunications and networks Web applications Internet
of Things
Predictive Maintenance
Text Search
Machine Learning SA SAMOA%
Anomaly Detection
When? Adoption Risks
– Gartner, 2015 “Despite considerable hype and reported successes for
early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early
Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,
MSFT Azure Stream Analytics Open source
Ease Inherently harder Ops best practices not ironed out (yet)
Lack of skills, training, and support Rethink applications from scratch
Actionable Insights? Define what you want Moving target Garbage in,
garbage out
Conclusions Who?
5 Answers What? Why? How? Where? When? Stream analytics Perishable
insights Asynchronous processing Everywhere In 5 years
Text Slow Fish or Fast Fish Which fish will you
be?
Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7
[email protected]