Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
Django's GeneratedField by example - DjangoCon US 2025
Avatar for Paolo Melchiorre pauloxnet
0
110
これでもう迷わない！Jetpack Composeの書き方実践ガイド
Avatar for ZOZO Developers zozotech
PRO
0
240
COVESA VSSによる車両データモデルの標準化とAWS IoT FleetWiseの活用
Avatar for Yuto Osawa osawa
1
250
機械学習を扱うプラットフォーム開発と運用事例
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
220
【実演版】カンファレンス登壇者・スタッフにこそ知ってほしいマイクの使い方 / 大吉祥寺.pm 2025
Avatar for Arthur arthur1
1
580
Automating Web Accessibility Testing with AI Agents
Avatar for South maminami373
0
1.2k
Firestore → Spanner 移行 を成功させた段階的移行プロセス
Avatar for a-thug athug
1
420
職種の壁を溶かして開発サイクルを高速に回す~情報透明性と職種越境から考えるAIフレンドリーな職種間連携~
Avatar for daitasu daitasu
0
140
下手な強制、ダメ！絶対！ 「ガードレール」を「檻」にさせない"ガバナンス"の取り方とは？
Avatar for Masataka Tsukamoto tsukaman
2
420
react-callを使ってダイヤログをいろんなとこで再利用しよう！
Avatar for shinaps shinaps
1
220
ガチな登山用デバイスからこんにちは
Avatar for halka halka
1
230
OCI Oracle Database Services新機能アップデート(2025/06-2025/08)
Avatar for oracle4engineer oracle4engineer
PRO
0
100

Featured

See All Featured
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
273
40k
Gamification - CAS2011
Avatar for David Bonilla davidbonilla
81
5.4k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
70
4.8k
Scaling GitHub
Avatar for Zach Holman holman
463
140k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
131
19k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
36
6.9k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
64
7.9k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
271
21k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.5k
Visualization
Avatar for Eitan Lees eitanlees
148
16k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com