Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems
Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
120
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
470
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
220

Other Decks in Technology

See All in Technology
KubeCon + CloudNativeCon Japan 2025 Recap
Avatar for Ren ren510dev
1
380
What’s new in Android development tools
Avatar for Yuki Anzai yanzm
0
310
LLM時代の検索
Avatar for shibuiwilliam shibuiwilliam
1
150
スタートアップに選択肢を 〜生成AIを活用したセカンダリー事業への挑戦〜
Avatar for Nstock nstock
0
190
マーケットプレイス版Oracle WebCenter Content For OCI
Avatar for oracle4engineer oracle4engineer
PRO
3
960
20250705 Headlamp: 專注可擴展性的 Kubernetes 用戶界面
Avatar for Phil Huang pichuang
0
270
MobileActOsaka_250704.pdf
Avatar for AKAI Tadaaki akaitadaaki
0
120
AI時代の開発生産性を加速させるアーキテクチャ設計
Avatar for PLAID Tech plaidtech
PRO
3
160
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
Avatar for SHIFT EVOLVE shift_evolve
PRO
2
7.7k
生成AI開発案件におけるClineの業務活用事例とTips
Avatar for Shinya Masadome（東京AI祭） shinya337
0
260
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
Avatar for iwamot iwamot
PRO
2
140
関数型プログラミングで 「脳がバグる」を乗り越える
Avatar for manabeai manabeai
1
190

Featured

See All Featured
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
55
5.7k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
51
3.3k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
20
1.3k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
35
2.4k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
411
22k
A better future with KSS
Avatar for Kyle Neath kneath
238
17k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
35
6.7k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
281
13k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
126
17k
Building Adaptive Systems
Avatar for Chris Keathley keathley
43
2.7k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
140
7k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com