Szymon Sobczak - Hadoop + Storm

Slide 1

Slide 1 text

Szymon Sobczak

Slide 2

Slide 2 text

Hadoop + Storm Combo for realtime big data systems

Slide 3

Slide 3 text

Plan • Hadoop & Storm • Our setup • What projects are we running • Decisions we had to make

Slide 4

Slide 4 text

Hadoop

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”, “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

Slide 7

Slide 7 text

Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura" “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

Slide 8

Slide 8 text

Common traits

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Base infrastructure services

Slide 11

Slide 11 text

Understand how the entire Base system works services

Slide 12

Slide 12 text

Big Data S3 uploader

Slide 13

Slide 13 text

Four example projects • Debugging • Reporting • Email intelligence • Forecasting

Slide 14

Slide 14 text

Debugging S3 uploader

Slide 15

Slide 15 text

Reporting

Slide 16

Slide 16 text

Email analysis S3 uploader

Slide 17

Slide 17 text

Forecasting S3 uploader

Slide 18

Slide 18 text

Decisions we made ☑ Collect *all* data ☑ Put them in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm