An Introduction to MapReduce (IPC2009 2009-11-15)

Workshop given at International PHP Conference 2009 in Karlsruhe, Germany.

Transcript

1. AN INTRODUCTION TO MAPREDUCE
2. David Zülke
3. David Zuelke
5. http://en.wikipedia.org/wiki/File:München_Panorama.JPG
6. Founder
7. awesome
8. Lead Developer
10. @dzuelke
11. BEFORE WE BEGIN... Installing Prerequisites
12. I brought a pre-configured VM
13. (I stole it from the nice folks over at Cloudera)
14. to save some time
15. • /cloudera-training-0.3.2/ • VMWare for Windows, Linux (i386 or x86_64) or Mac OS from /vmware/ if you don’t have it. • For Fusion, go to vmware.com and get an evaluation key. • /php/ PLEASE COPY FROM THE HD
16. (but be so kind as to pretend to be still listening)
17. FROM 30.000 FEET Distributed And Parallel Computing
18. we want to process data
19. how much data exactly?
20. SOME NUMBERS • Google • Data processed per month: 400 PB (in 2007!) • Average job size: 180 GB • Facebook • New data per day: • 200 GB (March 2008) • 2 TB (April 2009) • 4 TB (October 2009)
21. what if you have that much data?
22. what if you have just 1% of that amount?
23. “no problemo”, you say?
24. reading 180 GB sequentially off a disk will take ~45 minutes
25. but you only have 16 GB or so of RAM per computer
26. data can be processed much faster than it can be read
27. solution: parallelize your I/O
28. but now you need to coordinate what you’re doing
29. and that’s hard
30. what if a node dies?
31. is data lost? will other nodes in the grid have to re-start? how do you coordinate this?
32. ENTER: OUR HERO Introducing MapReduce
33. in the olden days, the workload was distributed across a grid
34. but the data was shipped around between nodes
35. or even stored centrally on something like an SAN
36. I/O bottleneck
37. Google made a publication in 2004
38. MapReduce: Simplified Data Processing on Large Clusters http://labs.google.com/papers/mapreduce.html
39. now the data is distributed
40. computing happens on the nodes where the data already is
41. processes are isolated and don’t communicate (share-nothing)
42. BASIC PRINCIPLE: MAPPER • A Mapper reads records and emits <key, value> pairs • Example: Apache access.log • Each line is a record • Extract client IP address and number of bytes transferred • Emit IP address as key, number of bytes as value • For hourly rotating logs, the job can be split across 24 nodes* * In pratice, it’s a lot smarter than that
43. BASIC PRINCIPLE: REDUCER • A Reducer is given a key and all values for this specific key • Even if there are many Mappers on many computers; the results are aggregated before they are handed to Reducers • Example: Apache access.log • The Reducer is called once for each client IP (that’s our key), with a list of values (transferred bytes) • We simply sum up the bytes to get the total traffic per IP!
44. EXAMPLE OF MAPPED INPUT IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371 212.122.174.13 43
45. REDUCER WILL RECEIVE THIS IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 212.122.174.13 43 74.119.8.111 91272 74.119.8.111 8371
46. AFTER REDUCTION IP Bytes 212.122.174.13 210238 74.119.8.111 99643
47. PSEUDOCODE function map($line_number, $line_text) { $parts = parse_apache_log($linetext); emit($parts['ip'], $parts['bytes']); } function reduce($key, $values) { $bytes = array_sum($values); emit($key, $bytes); } 212.122.174.13 210238 74.119.8.111 99643 212.122.174.13 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /foo HTTP/1.1" 200 18271 212.122.174.13 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /bar HTTP/1.1" 200 191726 212.122.174.13 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /baz HTTP/1.1" 200 198 74.119.8.111 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /egg HTTP/1.1" 200 43 74.119.8.111 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /moo HTTP/1.1" 200 91272 212.122.174.13 ‐ ‐ [30/Oct/2009:18:14:32 +0100] "GET /yay HTTP/1.1" 200 8371
48. FINGER EXERCISE Let’s Try PHP First
49. HANDS-ON Time To Write Some Code!
50. ANOTHER ELEPHANT Introducing Apache Hadoop
52. Hadoop is a MapReduce framework
53. it allows us to focus on writing Mappers, Reducers etc.
54. and it works extremely well
55. how well exactly?
56. HADOOP AT FACEBOOK • Predominantly used in combination with Hive (~95%) • 4800 cores with 12 TB of storage per node • Per day: • 4 TB of new data (compressed) • 135 TB of data scanned (compressed) • 7500+ Hive jobs per day, ~80k compute hours http://www.slideshare.net/cloudera/hw09-rethinking-the-data-warehouse-with-hadoop-and-hive
57. HADOOP AT YAHOO! • Over 25,000 computers with over 100,000 CPUs • Biggest Cluster: • 4000 Nodes • 2x4 CPU cores each • 16 GB RAM each • Over 40% of jobs run using Pig http://wiki.apache.org/hadoop/PoweredBy
58. there’s just one little problem
59. it’s written in Java
60. however, there is hope...
61. STREAMING Hadoop Won’t Force Us To Use Java
62. Hadoop Streaming can use any script as Mapper or Reducer
63. many configuration options (parsers, formats, combining, …)
64. it works using STDIN and STDOUT
65. Mappers are streamed the records (usually by line: <byteoffset>\t<line>\n) and emit key/value pairs: <key>\t<value>\n
66. Reducers are streamed key/value pairs: <keyA>\t<value1>\n <keyA>\t<value2>\n <keyA>\t<value3>\n <keyB>\t<value4>\n
67. Caution: no separate Reducer processes per key (but keys are sorted)
68. HANDS-ON Let’s Say Hello To Our Hadoop VM
69. THE HADOOP ECOSYSTEM A Little Tour
70. APACHE AVRO Efficient Data Serialization System With Schemas (compare: Facebook’s Thrift)
71. APACHE CHUKWA Distributed Data Collection System (compare: Facebook’s Scribe)
72. APACHE HBASE Like Google’s BigTable, Only That You Can Have It, Too!
73. HDFS Your Friendly Distributed File System
74. HIVE Data Warehousing Made Simple With An SQL Interface
75. PIG A High-Level Language For Modelling Data Processing Tasks
76. ZOOKEEPER Your Distributed Applications, Coordinated
77. ISABEL DROST Welcome Our Special Guest, Presenting Apache Mahout
78. !e End
79. Questions?
80. THANK YOU! • http://hadoop.apache.org/ is the Hadoop project website • http://www.cloudera.com/hadoop-training has useful resources • Send me an E-Mail: david.zuelke@bitextender.com • Follow @dzuelke on Twitter • Slides will be on SlideShare