Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
To Stream or Not To Stream? The Landscape of On...
Search
Gianmarco De Francisci Morales
May 28, 2015
Research
1
360
To Stream or Not To Stream? The Landscape of Online Analytics
EVAM Solution Day 2015, Istanbul.
Gianmarco De Francisci Morales
May 28, 2015
Tweet
Share
More Decks by Gianmarco De Francisci Morales
See All by Gianmarco De Francisci Morales
Echo Chambers on Social Media
gdfm
0
43
Learning Agent-Based Models from Data
gdfm
0
29
Repurpose, Reuse, Recycle the building blocks of Machine Learning
gdfm
0
91
How I Learned to Stop Worrying and Love the Risk
gdfm
0
140
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
150
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
330
Quantifying and Reducing Controversy in Social Media
gdfm
0
170
Big Data Streams: The Next Frontier
gdfm
2
330
Mining Big Data Streams: Better Algorithms or Faster Systems?
gdfm
0
480
Other Decks in Research
See All in Research
【輪講資料】Moshi: a speech-text foundation model for real-time dialogue
hpprc
3
510
学生向けアンケート<データサイエンティストについて>
datascientistsociety
PRO
0
4.4k
A scalable, annual aboveground biomass product for monitoring carbon impacts of ecosystem restoration projects
satai
3
110
Vision and LanguageからのEmbodied AIとAI for Science
yushiku
PRO
1
260
20250502_ABEJA_論文読み会_スライド
flatton
0
180
EarthSynth: Generating Informative Earth Observation with Diffusion Models
satai
3
130
問いを起点に、社会と共鳴する知を育む場へ
matsumoto_r
PRO
0
470
カスタマーサクセスの視点からAWS Summitの展示を考える~製品開発で活用できる勘所~
masakiokuda
2
140
データサイエンティストの採用に関するアンケート
datascientistsociety
PRO
0
1.1k
【緊急警告】日本の未来設計図 ~沈没か、再生か。国民と断行するラストチャンス~
yuutakasan
0
140
とあるSREの博士「過程」 / A Certain SRE’s Ph.D. Journey
yuukit
6
3.5k
[輪講] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
nk35jk
2
660
Featured
See All Featured
Bash Introduction
62gerente
613
210k
Producing Creativity
orderedlist
PRO
346
40k
Side Projects
sachag
455
43k
Designing Experiences People Love
moore
142
24k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Navigating Team Friction
lara
187
15k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
The Cult of Friendly URLs
andyhume
79
6.5k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Thoughts on Productivity
jonyablonski
69
4.7k
Transcript
To Stream or Not To Stream? The Landscape of Online
Analytics Gianmarco De Francisci Morales
[email protected]
@gdfm7
None
5 Questions What? Why? How? Where? When?
What? Data Stream
Text Big Data Too big to handle
Text Big Data Streams Drinking from a firehose
Stream Analytics Batch data = snapshot of streaming data Descriptive
Predictive Prescriptive
Value of Data
Online vs Real-Time
Why? Motivation and Goal
–Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside
a company is some new information comes in and the company reacts to that asynchronously.” Asynchronous Processing
Nervous System vs Silos
Perishable Insights Great instantaneous value Ephemeral Opportunity cost
Hype Cycle
Hype Cycle
How? Stream Processing Architecture
Architecture Overview
Ingestion Plethora of solutions Still ad-hoc (read: messy) Schema evolution:
Avro Column-store: Parquet Log collection: Flume
Brokerage
Processing PE PE Input Stream PEI PEI PEI PEI PEI
Output Stream Event routing
Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive
callbacks
Example: Reactive Web App
Lambda vs Kappa
Where? Applications
Application Domains Industrial applications Telecommunications and networks Web applications Internet
of Things
Predictive Maintenance
Text Search
Machine Learning SA SAMOA%
Anomaly Detection
When? Adoption Risks
– Gartner, 2015 “Despite considerable hype and reported successes for
early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early
Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,
MSFT Azure Stream Analytics Open source
Ease Inherently harder Ops best practices not ironed out (yet)
Lack of skills, training, and support Rethink applications from scratch
Actionable Insights? Define what you want Moving target Garbage in,
garbage out
Conclusions Who?
5 Answers What? Why? How? Where? When? Stream analytics Perishable
insights Asynchronous processing Everywhere In 5 years
Text Slow Fish or Fast Fish Which fish will you
be?
Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7
[email protected]