Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Base Lab Base Lab
May 07, 2015
Technology
120
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
160
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
140
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
320
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
260
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
160
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
500
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
260

Other Decks in Technology

See All in Technology
型は壁、Rustでもバグを直すな、表現できなくせよ
Avatar for nwiizo nwiizo
13
2k
プロダクトだけじゃない、社内プロセスにおける自動化・省力化ノススメ
Avatar for KAKEHASHI kakehashi
PRO
1
3.7k
DatabricksにおけるMCPソリューション
Avatar for Takaaki Yayoi taka_aki
1
250
オブザーバビリティ、本当に活用できてる？ 〜API連携×生成AIで成熟度を自動評価〜
Avatar for dmmsre dmmsre
1
3.2k
End-to-Endで考える信頼性 — LINEアプリにおける クライアント開発×SRE連携の実践
Avatar for maru maruloop
4
4.2k
10年目を迎えた「ABEMA」がどのように AI 活用を推進して、AI 駆動開発にシフトしているのか / How ABEMA, entering its 10th year, is promoting the use of AI and shifting toward AI-driven development
Avatar for Yusei Yamanaka miyukki
0
140
“それは自分の仕事じゃない"を 越えて行け
Avatar for yuukiyo yuukiyo
0
310
Data + AI Summit 2026 イベントレポート： 「AIがビジネスで意思決定するデータ基盤」へ
Avatar for KJ(Kenji Matsuda) nek0128
0
140
CSに"SLO"は要らない、経営層に"99.9%"は伝わらない - SREを全社に"翻訳"する3原則
Avatar for サイバーセキュリティクラウド cscengineer
PRO
1
4.6k
AICoEでAIネイティブ組織への進化
Avatar for YukiOgawa yukiogawa
0
170
「最後に責任を取るのはチーム」— 人間のPRレビューを最小化してアップデートしたメンタルモデル
Avatar for Jotaro Nishime jnishime_dresscode
0
650
Road to SRE NEXTの今までとこれから
Avatar for Hiroya Onoe hiroyaonoe
0
320

Featured

See All Featured
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.5k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
2k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
230
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.3k
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.3k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
230
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
34k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
970
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.2k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com