Upgrade to Pro — share decks privately, control downloads, hide ads and more …

To Stream or Not To Stream? The Landscape of Online Analytics

Gianmarco De Francisci Morales
May 28, 2015
Research
1
210

To Stream or Not To Stream? The Landscape of Online Analytics

EVAM Solution Day 2015, Istanbul.

Gianmarco De Francisci Morales

May 28, 2015
Tweet

More Decks by Gianmarco De Francisci Morales

See All by Gianmarco De Francisci Morales
How I Learned to Stop Worrying and Love the Risk
gdfm
0
50
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
54
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
120
Quantifying and Reducing Controversy in Social Media
gdfm
0
81
Big Data Streams: The Next Frontier
gdfm
2
250
Mining Big Data Streams: Better Algorithms or Faster Systems?
gdfm
0
420
SAMOA @Strata Barcelona 2014
gdfm
2
530
Distributed Adaptive Model Rules for Mining Big Data Streams
gdfm
1
120
Big Data Stream Mining Tutorial
gdfm
1
210

Other Decks in Research

See All in Research
Privacy & Semantic Technologies
kirrane
0
100
Stack-chanで始めるROS音声対話ロボット
yoshipon
1
350
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
5
880
PLDI '21論文読み会: Quantum Abstract Interpretation
ideininc
0
870
note UXリサーチチームの取り組み
sendamasato
0
140
第15回チャンピオンズミーティング・キャンサー杯ラウンド2集計 / Umamusume Cancer 2022 Round2
kitachan_black
0
580
文章のなかの地理空間 － 地理空間情報科学(GIS)と自然言語処理(NLP)の融合へ向けて －
hiroki13
3
2.7k
まちづくりのデジタル_トランスフォーメーション実現ビジョン_ver1.0.pdf
toshiseisaku
0
7k
[JSAI2022] 広告文自動生成に関する最近の研究動向
ichiroex
5
5.8k
論文紹介:Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions
ryu1104
0
170
Navigation for a team of agents
kei18
0
420
BigQueryとPythonではじめるプロ野球選手の成績予測（もしくは成績占い） / Baseball Player Performance Prediction using BigQuery and Python
shinyorke
0
1.6k

Featured

See All Featured
A better future with KSS
kneath
226
16k
The Language of Interfaces
destraynor
148
21k
The Pragmatic Product Professional
lauravandoore
19
3.2k
Designing the Hi-DPI Web
ddemaree
273
32k
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
127
8.6k
The Brand Is Dead. Long Live the Brand.
mthomps
47
2.8k
Fireside Chat
paigeccino
14
1.5k
Embracing the Ebb and Flow
colly
73
3.4k
Stop Working from a Prison Cell
hatefulcrawdad
262
18k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
223
49k
Adopting Sorbet at Scale
ufuk
65
7.6k
How to train your dragon (web standard)
notwaldorf
62
4k

Transcript

  1. To Stream or Not To Stream? The Landscape of Online

    Analytics Gianmarco De Francisci Morales
 gdfm@acm.org
 @gdfm7
  2. None

  3. 5 Questions What? Why? How? Where? When?

  4. What? Data Stream

  5. Text Big Data Too big to handle

  6. Text Big Data Streams Drinking from a firehose

  7. Stream Analytics Batch data = snapshot of streaming data Descriptive

    Predictive Prescriptive

  8. Value of Data

  9. Online vs Real-Time

  10. Why? Motivation and Goal

  11. –Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside

    a company is some 
 new information comes in and the company reacts 
 to that asynchronously.” Asynchronous Processing

  12. Nervous System vs Silos

  13. Perishable Insights Great instantaneous value Ephemeral Opportunity cost

  14. Hype Cycle

  15. Hype Cycle

  16. How? Stream Processing Architecture

  17. Architecture Overview

  18. Ingestion Plethora of solutions Still ad-hoc
 (read: messy) Schema evolution:

    Avro Column-store: Parquet Log collection: Flume

  19. Brokerage

  20. Processing PE PE Input Stream PEI PEI PEI PEI PEI

    Output Stream Event routing

  21. Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive

    callbacks

  22. Example: Reactive Web App

  23. Lambda vs Kappa

  24. Where? Applications

  25. Application Domains Industrial applications Telecommunications and networks Web applications Internet

    of Things

  26. Predictive Maintenance

  27. Text Search

  28. Machine Learning SA SAMOA%

  29. Anomaly Detection

  30. When? Adoption Risks

  31. – Gartner, 2015 “Despite considerable hype and reported successes for

    early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early

  32. Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,

    MSFT Azure Stream Analytics Open source

  33. Ease Inherently harder Ops best practices not ironed out (yet)

    Lack of skills, training, and support Rethink applications from scratch

  34. Actionable Insights? Define what you want Moving target Garbage in,

    garbage out

  35. Conclusions Who?

  36. 5 Answers What? Why? How? Where? When? Stream analytics Perishable

    insights Asynchronous processing Everywhere In 5 years

  37. Text Slow Fish or Fast Fish Which fish will you

    be?

  38. Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7 gdfm@acm.org