Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Transcript

1. Szymon Sobczak
2. Hadoop + Storm Combo for realtime big data systems
3. Plan • Hadoop & Storm • Our setup • What projects are we running • Decisions we had to make
4. Hadoop
6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”, “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”
7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura" “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”
8. Common traits
10. Base infrastructure services
11. Understand how the entire Base system works services
12. Big Data S3 uploader
13. Four example projects • Debugging • Reporting • Email intelligence • Forecasting
14. Debugging S3 uploader
15. Reporting
16. Email analysis S3 uploader
17. Forecasting S3 uploader
18. Decisions we made ☑ Collect *all* data ☑ Put them in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm
19. Summary S3 uploader
20. Questions?
21. Thank you szymon.sobczak@getbase.com bigdata.getbase.com