Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
OCI Network Firewall 概要
Avatar for oracle4engineer oracle4engineer
PRO
1
7.7k
pprof vs runtime/trace (FlightRecorder)
Avatar for task4233 task4233
0
140
Oracle Cloud Infrastructure：2025年9月度サービス・アップデート
Avatar for oracle4engineer oracle4engineer
PRO
0
310
analysis パッケージの仕組みの上でMulti linter with configを実現する / Go Conference 2025
Avatar for Ken’ichiro Oyama k1low
1
240
kaigi_on_rails_2025_設計.pdf
Avatar for Yasuko Ohba nay3
8
4k
LLMアプリケーション開発におけるセキュリティリスクと対策 / LLM Application Security
Avatar for GMO Flatt Security flatt_security
7
1.5k
SoccerNet GSRの紹介と技術応用：選手視点映像を提供するサッカー作戦盤ツール
Avatar for MIXI ENGINEERS mixi_engineers
PRO
1
120
インサイト情報からどこまで自動化できるか試してみた
Avatar for takas takas0522
0
120
Green Tea Garbage Collector の今
Avatar for zchee zchee
PRO
2
350
生成AIを活用したZennの取り組み事例
Avatar for igarashi ryosukeigarashi
0
170
そのグラフに「魂」は宿っているか？ ～生成AI全盛期におけるデータ可視化手法とライブラリ比較～
Avatar for negi111111 negi111111
2
840
サプライチェーン攻撃に学ぶModuleの仕組みと セキュリティ対策
Avatar for kuro kuro_kurorrr
3
790

Featured

See All Featured
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
237
140k
Making Projects Easy
Avatar for Brett Harned brettharned
118
6.4k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
4
660
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.5k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
127
53k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
303
21k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.2k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.6k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
600
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
791
250k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com