Large-Scale Data Processing with Hadoop and PHP (PHPUK2011 2011-02-25)

Presentation given at PHP UK Conference 2011 in London, England, United Kingdom.

Transcript

1. LARGE-SCALE DATA PROCESSING WITH HADOOP AND PHP
2. David Zülke
3. David Zuelke
6. (no need to be afraid)
7. (we’re all friends now)
8. http://en.wikipedia.org/wiki/File:München_Panorama.JPG
9. Founder
11. Lead Developer
13. @dzuelke
14. FROM 30.000 FEET Distributed And Parallel Computing
15. we want to process data
16. how much data exactly?
17. SOME NUMBERS • Facebook • New data per day: • 200 GB (March 2008) • 2 TB (April 2009) • 4 TB (October 2009) • 12 TB (March 2010) • Google • Data processed per month: 400 PB (in 2007!) • Average job size: 180 GB
18. what if you have that much data?
19. what if you have just 1% of that amount?
20. “no problemo”, you say?
21. reading 180 GB sequentially off a disk will take ~45 minutes
22. but you only have 16 GB or so of RAM per computer
23. data can be processed much faster than it can be read
24. solution: parallelize your I/O
25. but now you need to coordinate what you’re doing
26. and that’s hard
27. what if a node dies?
28. is data lost? will other nodes in the grid have to re-start? how do you coordinate this?
29. ENTER: OUR HERO Introducing MapReduce
30. in the olden days, the workload was distributed across a grid
31. but the data was shipped around between nodes
32. or even stored centrally on something like an SAN
33. I/O bottleneck
34. along came a Google publication in 2004
35. MapReduce: Simplified Data Processing on Large Clusters http://labs.google.com/papers/mapreduce.html
36. now the data is distributed
37. computing happens on the nodes where the data already is
38. processes are isolated and don’t communicate (share-nothing)
39. BASIC PRINCIPLE: MAPPER • A Mapper reads records and emits <key, value> pairs • Example: Apache access.log • Each line is a record • Extract client IP address and number of bytes transferred • Emit IP address as key, number of bytes as value • For hourly rotating logs, the job can be split across 24 nodes* * In pratice, it’s a lot smarter than that
40. BASIC PRINCIPLE: REDUCER • A Reducer is given a key and all values for this specific key • Even if there are many Mappers on many computers; the results are aggregated before they are handed to Reducers • Example: Apache access.log • The Reducer is called once for each client IP (that’s our key), with a list of values (transferred bytes) • We simply sum up the bytes to get the total traffic per IP!
41. EXAMPLE OF MAPPED INPUT IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 74.119.8.111 91272 74.119.8.111 8371 212.122.174.13 43
42. REDUCER WILL RECEIVE THIS IP Bytes 212.122.174.13 18271 212.122.174.13 191726 212.122.174.13 198 212.122.174.13 43 74.119.8.111 91272 74.119.8.111 8371
43. AFTER REDUCTION IP Bytes 212.122.174.13 210238 74.119.8.111 99643
44. PSEUDOCODE function  map($line_number,  $line_text)  {    $parts  =  parse_apache_log($line_text);    emit($parts['ip'],  $parts['bytes']); } function  reduce($key,  $values)  {    $bytes  =  array_sum($values);    emit($key,  $bytes); } 212.122.174.13  210238 74.119.8.111      99643 212.122.174.13  -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /foo  HTTP/1.1"  200  18271 212.122.174.13  -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /bar  HTTP/1.1"  200  191726 212.122.174.13  -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /baz  HTTP/1.1"  200  198 74.119.8.111      -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /egg  HTTP/1.1"  200  43 74.119.8.111      -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /moo  HTTP/1.1"  200  91272 212.122.174.13  -­‐  -­‐  [30/Oct/2009:18:14:32  +0100]  "GET  /yay  HTTP/1.1"  200  8371
45. A YELLOW ELEPHANT Introducing Apache Hadoop
47. Hadoop is a MapReduce framework
48. it allows us to focus on writing Mappers, Reducers etc.
49. and it works extremely well
50. how well exactly?
51. HADOOP AT FACEBOOK (I) • Predominantly used in combination with Hive (~95%) • 8400 cores with ~12.5 PB of total storage • 8 cores, 12 TB storage and 32 GB RAM per node • 1x Gigabit Ethernet for each server in a rack • 4x Gigabit Ethernet from rack switch to core http://www.slideshare.net/royans/facebooks-petabyte-scale-data-warehouse-using-hive-and-hadoop Hadoop is aware of racks and locality of nodes
52. HADOOP AT FACEBOOK (II) • Daily stats: • 25 TB logged by Scribe • 135 TB of compressed data scanned • 7500+ Hive jobs • ~80k compute hours • New data per day: • I/08: 200 GB • II/09: 2 TB (compressed) • III/09: 4 TB (compressed) • I/10: 12 TB (compressed) http://www.slideshare.net/royans/facebooks-petabyte-scale-data-warehouse-using-hive-and-hadoop
53. HADOOP AT YAHOO! • Over 25,000 computers with over 100,000 CPUs • Biggest Cluster: • 4000 Nodes • 2x4 CPU cores each • 16 GB RAM each • Over 40% of jobs run using Pig http://wiki.apache.org/hadoop/PoweredBy
54. OTHER NOTABLE USERS • Twitter (storage, logging, analysis. Heavy users of Pig) • Rackspace (log analysis; data pumped into Lucene/Solr) • LinkedIn (friend suggestions) • Last.fm (charts, log analysis, A/B testing) • The New York Times (converted 4 TB of scans using EC2)
55. there’s just one little problem
56. you need to write Java code
57. however, there is hope...
58. STREAMING Hadoop Won’t Force Us To Use Java
59. Hadoop Streaming can use any script as Mapper or Reducer
60. many configuration options (parsers, formats, combining, …)
61. it works using STDIN and STDOUT
62. Mappers are streamed the records (usually by line: <line>\n) and emit key/value pairs: <key>\t<value>\n
63. Reducers are streamed key/value pairs: <keyA>\t<value1>\n <keyA>\t<value2>\n <keyA>\t<value3>\n <keyB>\t<value4>\n
64. Caution: no separate Reducer processes per key (but keys are sorted)
65. HDFS Hadoop Distributed File System
66. HDFS • Stores data in blocks (default block size: 64 MB) • Designed for very large data sets • Designed for streaming rather than random reads • Write-once, read-many (although appending is possible) • Capable of compression and other cool things
67. HDFS CONCEPTS • Large blocks minimize amount of seeks, maximize throughput • Blocks are stored redundantly (3 replicas as default) • Aware of infrastructure characteristics (nodes, racks, ...) • Datanodes hold blocks • Namenode holds the metadata Critical component for an HDFS cluster (HA, SPOF)
68. JOB PROCESSING How Hadoop Works
69. Just like I already described! It’s MapReduce! \o/
70. BASIC RULES • Uses Input Formats to split up your data into single records • You can optimize using combiners to reduce locally on a node • Only possible in some cases, e.g. for max(), but not avg() • You can control partitioning of map output yourself • Rarely useful, the default partitioner (key hash) is enough • And a million other things that really don’t matter right now ;)
71. oh and, if you’re wondering how Hadoop got its name
72. The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless and not used elsewhere: those are my naming criteria. Kids are good at generating such. Googol is a kid’s term. Doug Cutting
73. STREAMING WITH PHP Introducing HadooPHP
74. HADOOPHP • A little framework to help with writing mapred jobs in PHP • Takes care of input splitting, can do basic decoding et cetera • Next version will provide an iterator for the values in reducers to improve memory efficiency • Packages jobs as one .phar archive to ease deployment • Also creates a ready-to-rock shell script to invoke the job
75. written by
77. (yesterday :p)
78. http://github.com/dzuelke/hadoophp
79. HANDS-ON Hadoop Streaming & PHP in action
80. THE HADOOP ECOSYSTEM A Little Tour
81. APACHE AVRO Efficient Data Serialization System With Schemas (compare: Facebook’s Thrift)
82. CLOUDERA FLUME Distributed Data Collection System (compare: Facebook’s Scribe)
83. APACHE HBASE Like Google’s BigTable, Only That You Can Have It, Too!
84. HDFS Your Friendly Distributed File System
85. HIVE Data Warehousing Made Simple With An SQL Interface
86. PIG A High-Level Language For Modeling Data Processing Tasks (fulfills the same purpose as Hive)
87. ZOOKEEPER Your Distributed Applications, Coordinated
88. !e End
89. RESOURCES • http://www.cloudera.com/developers/learn-hadoop/ • Tom White: Hadoop. The Definitive Guide. O’Reilly, 2009 • http://www.cloudera.com/hadoop/ • Cloudera Distribution for Hadoop is easy to install and has all the stuff included: Hadoop, Hive, Flume, Sqoop, Oozie, …
90. Questions?
91. THANK YOU! This was http://joind.in/2516 by @dzuelke. Contact me or hire us: david.zuelke@bitextender.com