Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
To Stream or Not To Stream? The Landscape of On...
Search
Gianmarco De Francisci Morales
May 28, 2015
Research
1
350
To Stream or Not To Stream? The Landscape of Online Analytics
EVAM Solution Day 2015, Istanbul.
Gianmarco De Francisci Morales
May 28, 2015
Tweet
Share
More Decks by Gianmarco De Francisci Morales
See All by Gianmarco De Francisci Morales
Echo Chambers on Social Media
gdfm
0
35
Learning Agent-Based Models from Data
gdfm
0
20
Repurpose, Reuse, Recycle the building blocks of Machine Learning
gdfm
0
81
How I Learned to Stop Worrying and Love the Risk
gdfm
0
140
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
130
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
290
Quantifying and Reducing Controversy in Social Media
gdfm
0
160
Big Data Streams: The Next Frontier
gdfm
2
320
Mining Big Data Streams: Better Algorithms or Faster Systems?
gdfm
0
480
Other Decks in Research
See All in Research
AIトップカンファレンスからみるData-Centric AIの研究動向 / Research Trends in Data-Centric AI: Insights from Top AI Conferences
tsurubee
3
2.3k
The Economics of Platforms 輪読会 第1章
tomonatu8
0
170
GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization
satai
3
120
定性データ、どう活かす? 〜定性データのための分析基盤、はじめました〜 / How to utilize qualitative data? ~We have launched an analysis platform for qualitative data~
kaminashi
6
830
Satellite Sunroof: High-res Digital Surface Models and Roof Segmentation for Global Solar Mapping
satai
3
280
RapidPen: AIエージェントによるペネトレーションテスト 初期侵入全自動化の研究
laysakura
0
330
実行環境に中立なWebAssemblyライブマイグレーション機構/techtalk-2025spring
chikuwait
0
140
20250226 NLP colloquium: "SoftMatcha: 10億単語規模コーパス検索のための柔らかくも高速なパターンマッチャー"
de9uch1
0
330
SpectralMamba: Efficient Mamba for Hyperspectral Image Classification
satai
3
300
言語モデルの内部機序:解析と解釈
eumesy
PRO
35
15k
Security, Privacy, and Trust in Generative AI
tsubasashi
0
110
LLM 시대의 Compliance: Safety & Security
huffon
0
640
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Building Adaptive Systems
keathley
41
2.5k
Practical Orchestrator
shlominoach
186
11k
Raft: Consensus for Rubyists
vanstee
137
6.9k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Done Done
chrislema
183
16k
The Cost Of JavaScript in 2023
addyosmani
49
7.7k
Being A Developer After 40
akosma
91
590k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
119
51k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.5k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.8k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Transcript
To Stream or Not To Stream? The Landscape of Online
Analytics Gianmarco De Francisci Morales
[email protected]
@gdfm7
None
5 Questions What? Why? How? Where? When?
What? Data Stream
Text Big Data Too big to handle
Text Big Data Streams Drinking from a firehose
Stream Analytics Batch data = snapshot of streaming data Descriptive
Predictive Prescriptive
Value of Data
Online vs Real-Time
Why? Motivation and Goal
–Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside
a company is some new information comes in and the company reacts to that asynchronously.” Asynchronous Processing
Nervous System vs Silos
Perishable Insights Great instantaneous value Ephemeral Opportunity cost
Hype Cycle
Hype Cycle
How? Stream Processing Architecture
Architecture Overview
Ingestion Plethora of solutions Still ad-hoc (read: messy) Schema evolution:
Avro Column-store: Parquet Log collection: Flume
Brokerage
Processing PE PE Input Stream PEI PEI PEI PEI PEI
Output Stream Event routing
Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive
callbacks
Example: Reactive Web App
Lambda vs Kappa
Where? Applications
Application Domains Industrial applications Telecommunications and networks Web applications Internet
of Things
Predictive Maintenance
Text Search
Machine Learning SA SAMOA%
Anomaly Detection
When? Adoption Risks
– Gartner, 2015 “Despite considerable hype and reported successes for
early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early
Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,
MSFT Azure Stream Analytics Open source
Ease Inherently harder Ops best practices not ironed out (yet)
Lack of skills, training, and support Rethink applications from scratch
Actionable Insights? Define what you want Moving target Garbage in,
garbage out
Conclusions Who?
5 Answers What? Why? How? Where? When? Stream analytics Perishable
insights Asynchronous processing Everywhere In 5 years
Text Slow Fish or Fast Fish Which fish will you
be?
Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7
[email protected]