Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems
Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
120
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
470
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
220

Other Decks in Technology

See All in Technology
関数型プログラミングで 「脳がバグる」を乗り越える
Avatar for manabeai manabeai
1
190
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
Avatar for iwamot iwamot
PRO
2
140
OPENLOGI Company Profile
Avatar for OPENLOGI hr01
0
67k
American airlines ®️ USA Contact Numbers: Complete 2025 Support Guide
Avatar for markhor64456 airhelpsupport
0
380
CDKTFについてざっくり理解する！！～CloudFormationからCDKTFへ変換するツールも作ってみた～
Avatar for モブエンジニア masakiokuda
1
140
20250705 Headlamp: 專注可擴展性的 Kubernetes 用戶界面
Avatar for Phil Huang pichuang
0
270
MUITにおける開発プロセスモダナイズの取り組みと開発生産性可視化の取り組みについて / Modernize the Development Process and Visualize Development Productivity at MUIT
Avatar for 三菱UFJインフォメーションテクノロジー株式会社 muit
1
16k
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
Avatar for ape1422 airtravelguide
0
340
無意味な開発生産性の議論から抜け出すための予兆検知とお金とAI
Avatar for Masato Ishigaki / 石垣雅人 i35_267
4
13k
改めてAWS WAFを振り返る～業務で使うためのポイント～
Avatar for モブエンジニア masakiokuda
2
250
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
Avatar for SHIFT EVOLVE shift_evolve
PRO
2
7.7k
OPENLOGI Company Profile for engineer
Avatar for OPENLOGI hr01
1
34k

Featured

See All Featured
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
134
9.4k
Making Projects Easy
Avatar for Brett Harned brettharned
116
6.3k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
695
190k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
30
5.9k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
77
5.9k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
173
14k
Writing Fast Ruby
Avatar for Erik Berlin sferik
628
62k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.7k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
267
13k
Scaling GitHub
Avatar for Zach Holman holman
460
140k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
328
39k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
613
210k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com