Enchanting PythonS to chrunch data - 2015

Using Python to drive MapReduce

Transcript

1. Enchanting PythonS to crunch data ... JuanJo Ciarlante @xjjo bit.ly/jjo-cv
2. EHLO jjo apasionado del software libre :)) 1993-[...] 1er aporte: ip aliasing: “eth0:0” & friends 1995 modularización/optimización IP masq 1997 profesor adjunto, Univ. Mendoza 1999-[...] open/swancrypto-algo(s) 2003 soporte transporte IPv6 para openvpn 2006-2010 Google: Gmail SRE (Site Reliability Eng) 2007-2012 Canonical: Devops/SA, Cloud Rel. Eng. 2012-[...]
3. MR: what ? •framework for massive data processing oactually: data transformation •based on ‘rows’/records as: o<key,value>
4. count’em all ! •have: http server logs 62.121.107.115 - - [05/Jan/2003:15:36:32 -0700] "GET /archive/2002/09/index.shtml HTTP/1.1" 200 22536 … 163.28.16.1 - - [22/Jan/2003:09:59:12 -0700] "GET /projects/domains/diff_domains_2002-07-20.txt HTTP/1.0" 200 57326 … •want: how many hits per page ('urlpath') ?
5. e.g.: hitcount by urlpath 1.2.3.4 /foo 2.3.4.5 /bar 1.2.3.4 /baz 1.2.3.4 /pip 3.4.5.6 /pip 5.6.7.8 /foo 4.5.6.7 /foo 2.3.3.3 /bar /foo | 1 /bar | 1 /baz | 1 /pip | 1 /pip | 1 /foo | 1 /foo | 1 /bar | 1 /foo | 1 | 1 | 1 /bar | 1 | 1 /baz | 1 /pip | 1 | 1 /bar | 2 /baz | 1 /foo | 3 /pip | 2 web1.log web2.log web3.log map reduce (shuffle+sort)
6. MR: how ? •map: picks data from input rows ▪record ---> key, data •(shuffle, sort) classifies by key to build: ▪ ... ---> key, [data1, data2, …] •reduce: aggregates, transforms - eg: ▪key, […] ---> key, sum([...])
7. MR: why is cool ? Kiss: really simple model Scalable: parallel-friendly by design Data-locality aware: distributed FS Sync-free: no explicitly required IPC/sync between tasks Resilient: handles workers' crashes
8. grep'em all ! •have: corpus of documents •want: to search them by word (“grep”)
9. todos giran y giran ↩ todos bajo el sol quién eeeen ↩ se ha tomado todo el vino me ha tomado el tiempo ↩ para verlos otra vez tomado | ? | ? map reduce sol | ? vino | ? e.g.: grep - filename by word idx /idx/HH/tomado.idx /idx/HH/sol.idx /idx/HH/vino.idx todos| giran | giran | todos| bajo | sol | ? eeeen | se | ha | tomado | ? todo | vino | ? me | ha | tomado | ? el | … | f1.txt f2.txt f3.txt
10. todos giran y giran ↩ todos bajo el sol quién eeeen ↩ se ha tomado todo el vino me ha tomado el tiempo ↩ para verlos otra vez todos| giran | giran | todos| bajo | sol | f1.txt f1.txt f2.txt f3.txt map me | ha | tomado | f2.txt el | … | eeeen | se | ha | tomado | f3.txt todo | vino | f3.txt tomado | f2.txt | f3.txt reduce sol | f1.txt vino | f3.txt idx /idx/HH/tomado.idx /idx/HH/sol.idx /idx/HH/vino.idx e.g.: grep - filename by word
11. MR: Hadoop ✔ FLOSS \o/ ✖ in Java :(… for Java :( ✔ => hadoop “streaming”: arbitrary commands with pipelined data locality: input | python mr.py map | shuffle | python mr.py reduce ✔ => and … Spark
12. MR: some python libs •MRJob ✔ local, hadoop, Elastic MR (AWS) ✖ not hadoop ‘native’ •Hadoopy ✔ optim. for hadoop, supports HDFS bin formats ✖ only hadoop •Spark ✔ local, hadoop, not only MR ✔ python, scala, R ✖ JavaJVM driven
13. speaking of diversity ... have: http server logs 62.121.107.115 - - [05/Jan/2003:15:36:32 -0700] "GET /archive/2002/09/index.shtml HTTP/1.1" 200 22536 … 163.28.16.1 - - [22/Jan/2003:09:59:12 -0700] "GET /projects/domains/diff_domains_2002-07-20.txt HTTP/1.0" 200 57326 … want: the diversity of client IPs by path oshamelessly use entropy(concatenated_IPs_bits) as a proxy value for relative diversity
14. e.g.: urlpath diversity 1.2.3.4 /foo 2.3.4.5 /bar 1.2.3.4 /baz 1.2.3.4 /pip 3.4.5.6 /pip 5.6.7.8 /foo 4.5.6.7 /foo 2.3.3.3 /bar /foo | 1.2.3.4 /bar | 2.3.4.5 /baz | 1.2.3.4 /pip | 1.2.3.4 /pip | 3.4.5.6 /foo | 5.6.7.8 /foo | 4.5.6.7 /bar | 2.3.3.3 /foo | 1.2.3.4 | 4.5.6.7 | 5.6.7.8 /bar | 1.2.3.4 | 2.3.3.3 /baz | 1.2.3.4 /pip | 1.2.3.4 | 3.4.5.6 /bar | 1.75 /baz | 2.0 /foo | 2.91 /pip | 2.5 web1.log web2.log web3.log map reduce: entropy([ips]) (shuffle+sort) 2nd map reduce to aggregate url path by entropy
15. from mrjob.job import MRJob class MRHitCount(MRJob): def mapper(self, _, line): ip, path =line.split() yield path, 1 def reducer(self, key, values): yield key, sum(values) if __name__ == '__main__': MRHitCount.run() MRjob: hitcount.py
16. from mrjob.job import MRJob from mrjob.compat import get_jobconf_value class MRGrep(MRJob): def mapper(self, _, line): for word in line.split(): yield word, get_jobconf_value('map.input. file') def reducer(self, key, values): yield key, str(values) if __name__ == '__main__': MRGrep.run() MRjob: grep.py
17. class MREntropyPerURL(MRJob): # 1st MR: urlpath -> entropy([ips]) def input_mapper(self, _, line): ip, path = line.split() yield path, ip def entropy_by_path (self, key, values): yield key, entropy_bits(values) # 2nd MR: aggregate all urlpaths by same entropy_val (omitted) # Pipe-line both MRs: def steps(self): return [self.mr( mapper=self.input_mapper, reducer=self.entropy_by_path ), self.mr( mapper=self.swap_path_entropy, reducer=self.paths_by_entropy )] if __name__ == '__main__': MREntropyPerURL .run() MRjob: urlentropy.py
18. Hacktime \o/ These slides: ▪ http://bit.ly/jjo-mrpy-15/ Source code for this slides: ▪ https://github.com/jjo/mapreduce A Guide to Python Frameworks for Hadoop ▪ http://blog.cloudera.com/blog/2013/01/a-guide-to-python- frameworks-for-hadoop/ Some MR py libs: ▪ https://pythonhosted.org/mrjob/ ▪ http://spark.apache.org/ Interesting datasets: ▪ https://snap.stanford.edu/data/ networks ▪ http://aws.amazon.com/datasets/ diverse data