MapReduce Intro and Data Thoughts

MapReduce disco/hadoop discussion and concluding thoughts and data analysis with humans and machines. Data and IPython notebooks at: https://github.com/ContinuumIO/tutorials

Transcript

1. Disco/Hadoop MapReduce Benjamin Zaitlen Continuum Analytics PyCon 2013 Thursday, March 14, 13
2. What is Large Data? Thursday, March 14, 13
3. What is Large Data? • Can’t fit into Excel Thursday, March 14, 13
4. What is Large Data? • Can’t fit into Excel • Increase Memory Thursday, March 14, 13
5. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R Thursday, March 14, 13
6. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R • Increase Memory Thursday, March 14, 13
7. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R • Increase Memory • Can’t fit into Memory Thursday, March 14, 13
8. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R • Increase Memory • Can’t fit into Memory • Increase Memory Thursday, March 14, 13
9. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R • Increase Memory • Can’t fit into Memory • Increase Memory • Can’t fit on a single disk Thursday, March 14, 13
10. What is Large Data? • Can’t fit into Excel • Increase Memory • Can’t fit into R • Increase Memory • Can’t fit into Memory • Increase Memory • Can’t fit on a single disk • Distributed Filesystem: SAN, HDFS/DDFS, AWS: S3, Redshift, etc. Thursday, March 14, 13
11. MapReduce Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
12. MapReduce • A mass of data: records Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
13. MapReduce • A mass of data: records • Split/Map records into key-values pairs Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
14. MapReduce • A mass of data: records • Split/Map records into key-values pairs • Collect/Partition kv pairs (Optional Sort) Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
15. MapReduce • A mass of data: records • Split/Map records into key-values pairs • Collect/Partition kv pairs (Optional Sort) • Buckets are passed to Reduce function Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
16. MapReduce • A mass of data: records • Split/Map records into key-values pairs • Collect/Partition kv pairs (Optional Sort) • Buckets are passed to Reduce function • Result is returned Framework to help solve the problem of distributed computation for distributed data Thursday, March 14, 13
17. MapReduce Workflow Thursday, March 14, 13
18. MapReduce Workflow • Push code to data Thursday, March 14, 13
19. MapReduce Workflow • Push code to data • Lots of network traffic Thursday, March 14, 13
20. MapReduce it’s a Party Thursday, March 14, 13
21. MapReduce it’s a Party Thursday, March 14, 13
22. MapReduce it’s a Party Thursday, March 14, 13
23. MapReduce it’s a Party • Disco: Python + Erlang Thursday, March 14, 13
24. MapReduce it’s a Party • Disco: Python + Erlang • Hadoop: Java-Dumbo (Python Streaming) Thursday, March 14, 13
25. MapReduce it’s a Party • Disco: Python + Erlang • Hadoop: Java-Dumbo (Python Streaming) • Distributed FileSystem: DDFS Thursday, March 14, 13
26. MapReduce it’s a Party • Disco: Python + Erlang • Hadoop: Java-Dumbo (Python Streaming) • Distributed FileSystem: DDFS • Bring your friends • NumPy • SciPy • pandas • scikits-learn • OpenCV Thursday, March 14, 13
27. Canonical Example Thursday, March 14, 13
28. Canonical Example Thursday, March 14, 13
29. Canonical Example Thursday, March 14, 13
30. Chain Jobs Thursday, March 14, 13
31. Chain Jobs Thursday, March 14, 13
32. Chain Jobs Thursday, March 14, 13
33. Chain Jobs Thursday, March 14, 13
34. MapReduce Thoughts Thursday, March 14, 13
35. MapReduce Thoughts • Data Cleansing • Everyone’s pain point Thursday, March 14, 13
36. MapReduce Thoughts • Data Cleansing • Everyone’s pain point • Task Deconstruction • Good for code management • Hides -- in a good way -- data management Thursday, March 14, 13
37. MapReduce Thoughts • Data Cleansing • Everyone’s pain point • Task Deconstruction • Good for code management • Hides -- in a good way -- data management • Can Be Inefficient • Network traffic • Job organization Thursday, March 14, 13
38. Data Thoughts Thursday, March 14, 13
39. Thursday, March 14, 13
40. 83 percent of the radiologists missed the gorilla Thursday, March 14, 13
41. Google Flu • Data Mining • Faster than CDC Thursday, March 14, 13
42. Google Get’s Wrong • Typically, prediction is great! • This year not so much • Google: No comment! • Feedback mechanism from hype-up media Thursday, March 14, 13
43. Wrapping Up • Invisible Gorillas will stay invisible • Turnkey analytics is dangerous • Good Analysis • Requires iterative exploration • Peer review and collaboration Thursday, March 14, 13