Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Base Lab
May 07, 2015
Technology
0
94

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
baselab
0
110
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
baselab
0
100
Tomasz Nowak - Web Application Testing made easy
baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
baselab
3
470
Jerzy Chałupski - Data model on Android
baselab
4
220

Other Decks in Technology

See All in Technology
[IBM TechXchange Dojo]Watson Discoveryとwatsonx.aiでRAGを実現!事例のご紹介+座学②
siyuanzh09
0
110
Kotlin Multiplatformのポテンシャル
recruitengineers
PRO
2
150
JuliaTokaiとJuliaLangJaの紹介 for NGK2025S
antimon2
1
110
WantedlyでのKotlin Multiplatformの導入と課題 / Kotlin Multiplatform Implementation and Challenges at Wantedly
kubode
0
240
機械学習を「社会実装」するということ 2025年版 / Social Implementation of Machine Learning 2025 Version
moepy_stats
5
1k
Oracle Exadata Database Service(Dedicated Infrastructure)：サービス概要のご紹介
oracle4engineer
PRO
0
12k
コロプラのオンボーディングを採用から語りたい
colopl
5
1.2k
0→1事業こそPMは営業すべし / pmconf #落選お披露目 / PM should do sales in zero to one
roki_n_
PRO
1
1.3k
Alignment and Autonomy in Cybozu - 300人の開発組織でアラインメントと自律性を両立させるアジャイルな組織運営 / RSGT2025
ama_ch
1
2.4k
ドメイン駆動設計の実践により事業の成長スピードと保守性を両立するショッピングクーポン
lycorptech_jp
PRO
12
1.7k
今年一年で頑張ること / What I will do my best this year
pauli
1
220
Evolving Architecture
rainerhahnekamp
3
250

Featured

See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
860
How GitHub (no longer) Works
holman
312
140k
How to Ace a Technical Interview
jacobian
276
23k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
How to Think Like a Performance Engineer
csswizardry
22
1.3k
Automating Front-end Workflow
addyosmani
1366
200k
The Invisible Side of Design
smashingmag
299
50k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
113
50k
The Power of CSS Pseudo Elements
geoffreycrofte
74
5.4k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
3
180

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com