Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
120
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
150
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
130
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
310
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
260
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
160
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
490
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
250

Other Decks in Technology

See All in Technology
AI Adaptable なテストを整える工夫 / Ways to Make Your Tests AI-Adaptable
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
2
160
oracle-to-databricks-migration-with-llm-and-dbt
Avatar for case-k-git casek
1
360
Platform Engineering as a Product: Criteria for Improvement and Multi-Tenant Design
Avatar for Kumo Ishikawa kumorn5s
0
220
Datadog 認定試験の概要と対策
Avatar for Uechi Shingo uechishingo
0
190
JJUG CCC 2026 Spring　AI時代の開発こそ標準化を武器に！ ― 方式・プロセス・プラットフォームの標準化
Avatar for WatanabeShun s27watanabe
2
610
Gradle×GitHub_ActionsでCI時間を約50%短縮 ジョブ分割の設計と落とし穴 / Cutting CI Time by ~50% with Gradle and GitHub Actions: Job-Splitting Design and Pitfalls
Avatar for 冨澤宝斗 takatty
0
520
Kiro CLI v2.0.0がやってきた！
Avatar for Hiroo Katoh kentapapa
0
210
コードレビューを制するチームがソフトウェアデリバリーのフローを制す / Beyond Code Review: Distributing Its Responsibilities Across the SDLC
Avatar for mtx2s mtx2s
1
370
イベントで大活躍する電子ペーパー名札 〜その３〜 / ビジュアルプログラミングIoTLT vol.23
Avatar for you（@youtoy） you
PRO
0
160
AI-DLCを活用した高品質・安全なAI駆動開発実践 / AI Driven Development
Avatar for 吉田真吾 yoshidashingo
1
240
組織の中で自分を経営する技術
Avatar for shoota shoota
0
220
大規模災害時でも高い信頼性を維持するアプリケーション基盤の実現/nikkei-tech-talk46
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
120

Featured

See All Featured
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
2
1.1k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
190
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
57
14k
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
210
Navigating the moral maze — ethical principles for Al-driven product design
Avatar for Skipper Chong Warson skipperchong
2
370
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
920
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
247
13k
What the history of the web can teach us about the future of AI
Avatar for Ines Montani inesmontani
PRO
1
590
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
180
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
380

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com