Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
大規模組織にAIエージェントを迅速に導入するためのセキュリティの勘所 / AI agents for large-scale organizations
Avatar for Masato Ishigaki / 石垣雅人 i35_267
6
360
【2025 Japan AWS Jr. Champions Ignition】点から線、線から面へ 〜僕たちが起こすコラボレーション・ムーブメント〜
Avatar for amixedcolor amixedcolor
1
110
帳票構造化タスクにおけるLLMファインチューニングの性能評価
Avatar for yosukeyoshida yosukeyoshida
1
180
Claude Codeが働くAI中心の業務システム構築の挑戦―AIエージェント中心の働き方を目指して
Avatar for os1ma os1ma
8
930
ecspressoの設計思想に至る道 / sekkeinight2025
Avatar for FUJIWARA Shunichiro fujiwara3
12
2.1k
少人数でも回る！ DevinとPlaybookで支える運用改善
Avatar for ishikawa-pro ishikawa_pro
4
1.9k
20250728 MCP, A2A and Multi-Agents in the future
Avatar for 吉田真吾 yoshidashingo
1
160
[MIRU25] NaiLIA: Multimodal Retrieval of Nail Designs Based on Dense Intent Descriptions
Avatar for Semantic Machine Intelligence Lab., Keio Univ. keio_smilab
PRO
1
150
Bet "Bet AI" - Accelerating Our AI Journey #BetAIDay
Avatar for LayerX layerx
PRO
1
130
公開初日に個人環境で試した Gemini CLI 体験記など / Gemini CLI実験レポート
Avatar for you（@youtoy） you
PRO
3
1.1k
AWS表彰プログラムとキャリアについて
Avatar for Naoki naoki_0531
1
150
株式会社島津製作所_研究開発（集団協業と知的生産）の現場を支える、OSS知識基盤システムの導入
Avatar for Kuniharu, AKAHANE (赤羽根/州晴) akahane92
1
1.3k

Featured

See All Featured
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
126
17k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
51
8.7k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.5k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
26k
A better future with KSS
Avatar for Kyle Neath kneath
238
17k
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
331
22k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
77
5.9k
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
43
7.6k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
Visualization
Avatar for Eitan Lees eitanlees
146
16k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com