Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
To Stream or Not To Stream? The Landscape of Online Analytics
Gianmarco De Francisci Morales
May 28, 2015
Research
1
190
To Stream or Not To Stream? The Landscape of Online Analytics
EVAM Solution Day 2015, Istanbul.
Gianmarco De Francisci Morales
May 28, 2015
Tweet
Share
More Decks by Gianmarco De Francisci Morales
See All by Gianmarco De Francisci Morales
gdfm
0
26
gdfm
0
89
gdfm
0
62
gdfm
2
230
gdfm
0
400
gdfm
2
500
gdfm
1
91
gdfm
1
180
gdfm
4
3.9k
Other Decks in Research
See All in Research
racterub
0
150
sei88888
4
300
corgies
0
210
tarugoconf
0
1.7k
yuukit
0
1.2k
hakubishin3
1
900
takuyaa
0
140
yururoi
3
220
mot_ai_tech
1
1k
ysekky
3
1.5k
yuukit
4
1.6k
yumu19
1
2.5k
Featured
See All Featured
keavy
107
14k
yeseniaperezcruz
302
31k
qrush
285
19k
carmenhchung
31
1.5k
hatefulcrawdad
257
17k
samanthasiow
56
6.4k
chriscoyier
145
20k
jacobian
255
20k
erikaheidi
14
4.3k
jonrohan
1021
380k
hannesfritz
28
950
colly
66
3k
Transcript
To Stream or Not To Stream? The Landscape of Online
Analytics Gianmarco De Francisci Morales gdfm@acm.org @gdfm7
None
5 Questions What? Why? How? Where? When?
What? Data Stream
Text Big Data Too big to handle
Text Big Data Streams Drinking from a firehose
Stream Analytics Batch data = snapshot of streaming data Descriptive
Predictive Prescriptive
Value of Data
Online vs Real-Time
Why? Motivation and Goal
–Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside
a company is some new information comes in and the company reacts to that asynchronously.” Asynchronous Processing
Nervous System vs Silos
Perishable Insights Great instantaneous value Ephemeral Opportunity cost
Hype Cycle
Hype Cycle
How? Stream Processing Architecture
Architecture Overview
Ingestion Plethora of solutions Still ad-hoc (read: messy) Schema evolution:
Avro Column-store: Parquet Log collection: Flume
Brokerage
Processing PE PE Input Stream PEI PEI PEI PEI PEI
Output Stream Event routing
Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive
callbacks
Example: Reactive Web App
Lambda vs Kappa
Where? Applications
Application Domains Industrial applications Telecommunications and networks Web applications Internet
of Things
Predictive Maintenance
Text Search
Machine Learning SA SAMOA%
Anomaly Detection
When? Adoption Risks
– Gartner, 2015 “Despite considerable hype and reported successes for
early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early
Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,
MSFT Azure Stream Analytics Open source
Ease Inherently harder Ops best practices not ironed out (yet)
Lack of skills, training, and support Rethink applications from scratch
Actionable Insights? Define what you want Moving target Garbage in,
garbage out
Conclusions Who?
5 Answers What? Why? How? Where? When? Stream analytics Perishable
insights Asynchronous processing Everywhere In 5 years
Text Slow Fish or Fast Fish Which fish will you
be?
Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7 gdfm@acm.org