Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
MCP認可の現在地と自律型エージェント対応に向けた課題 / MCP Authorization Today and Challenges to Support Autonomous Agents
Avatar for Yoichi Kawasaki yokawasa
5
2.5k
事業特性から逆算したインフラ設計
Avatar for UPSIDER, Inc. Tech&Product div. upsider_tech
0
230
【OptimizationNight】数理最適化のラストワンマイルとしてのUIUX
Avatar for BrainPad brainpadpr
2
540
いま、あらためて考えてみるアカウント管理 with IaC / Account management with IaC
Avatar for kohbis kohbis
2
390
オブザーバビリティ文化を組織に浸透させるには / install observability culture
Avatar for mackerelio mackerelio
0
160
Amazon Qで2Dゲームを作成してみた
Avatar for Siromi siromi
0
170
LLM 機能を支える Langfuse / ClickHouse のサーバレス化
Avatar for yuu26 yuu26
9
2.6k
Exadata Database Service on Dedicated Infrastructure セキュリティ、ネットワーク、および管理について
Avatar for oracle4engineer oracle4engineer
PRO
1
330
Lambda management with ecspresso and Terraform
Avatar for Michael H. Oshita ijin
2
170
モノレポにおけるエラー管理 ~Runbook自動生成とチームメンションの最適化
Avatar for Shota Iwami biwashi
0
360
20250818_KGX・One Hokkaidoコラボイベント
Avatar for tohge05 tohgeyukihiro
0
100
「AIと一緒にやる」が当たり前になるまでの奮闘記
Avatar for KAKEHASHI kakehashi
PRO
3
180

Featured

See All Featured
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
134
9.5k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
36
2.5k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
51
3.4k
Side Projects
Avatar for Sacha Greif sachag
455
43k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
9
770
Faster Mobile Websites
Avatar for Dean Hume deanohume
309
31k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
229
22k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
49
14k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com