Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
250
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
Dylib Hijacking on macOS: Dead or Alive?
Avatar for Patrick Wardle patrickwardle
0
440
Implementing and Evaluating a High-Level Language with WasmGC and the Wasm Component Model: Scala’s Case
Avatar for Rikito Taniguchi tanishiking
0
170
AI AgentをLangflowでサクッと作って、1日働かせてみた！
Avatar for Yano-13 yano13
1
120
Bill One 開発エンジニア 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
4
14k
旅で応援する✈️ NEWTが目指すコミュニティ支援とあたらしい旅行 / New Travel: Supporting by NEWT on Your Journey
Avatar for miisan mii3king
0
140
Sansan Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
3k
React19.2のuseEffectEventを追う
Avatar for マグロ隊長kinTV maguroalternative
2
600
「タコピーの原罪」から学ぶ間違った”支援” / the bad support of Takopii
Avatar for piyonakajima piyonakajima
0
120
初めてのDatabricks Apps開発
Avatar for Takaaki Yayoi taka_aki
1
240
serverless team topology
Avatar for kensh _kensh
2
130
会社を支える Pythonという言語戦略 ~なぜPythonを主要言語にしているのか？~
Avatar for curekoshimizu curekoshimizu
2
520
NLPコロキウム20251022_超効率化への挑戦: LLM 1bit量子化のロードマップ
Avatar for Yuma Ichikawa yumaichikawa
1
190

Featured

See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
23
1.5k
Scaling GitHub
Avatar for Zach Holman holman
463
140k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
7
290
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
470k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
431
66k
Faster Mobile Websites
Avatar for Dean Hume deanohume
310
31k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
33k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Fireside Chat
Avatar for paigeccino paigeccino
40
3.7k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
130k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com