Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
95

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems
Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
120
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
470
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
220

Other Decks in Technology

See All in Technology
Long journey of Continuous Delivery at Mercari
Avatar for Hisaharu Ishii hisaharu
1
200
"SaaS is Dead" は本当か！？ 生成AI時代の医療 Vertical SaaS のリアル
Avatar for KAKEHASHI kakehashi
PRO
3
170
データ戦略部門 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
3.2k
vLLM meetup Tokyo
Avatar for jpishikawa jpishikawa
1
190
AI とペアプロしてわかった 3 つのヒューマンエラー
Avatar for TakahiroIKEGAWA (池川 貴裕) takahiroikegawa
1
660
What's new in OpenShift 4.19
Avatar for Red Hat Livestreaming redhatlivestreaming
1
190
AIにどこまで任せる？実務で使える(かもしれない)AIエージェント設計の考え方
Avatar for Har1101 har1101
3
930
Workflows から Agents へ ~ 生成 AI アプリの成長過程とアプローチ~
Avatar for Belong inc. belongadmin
2
130
産業機械をElixirで制御する
Avatar for kikuyuta kikuyuta
0
150
Tensix Core アーキテクチャ解説
Avatar for Tenstorrent Japan tenstorrent_japan
0
340
ゆるSRE #11 LT
Avatar for Kaoru okaru
1
580
堅牢な認証基盤の実現 TypeScriptで代数的データ型を活用する
Avatar for KAKEHASHI kakehashi
PRO
1
200

Featured

See All Featured
Gamification - CAS2011
Avatar for David Bonilla davidbonilla
81
5.3k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.4k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
299
21k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
20
1.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
47
2.8k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
267
13k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
430
65k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
71
11k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Docker and Python
Avatar for Tania Allard trallard
44
3.4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
252
21k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com