A Tour of Large-Scale Data Analysis Tools In Python

Transcript

1. Sarah Guido & Sean O’Connor PyCon 2016 May 28th, 2016 A TOUR OF LARGE- SCALE DATA ANALYSIS TOOLS IN PYTHON
2. gph.is/10JrTRx
3. •Download and install Virtual Box from
 http:/ /bit.ly/vboxdl •Download and install Vagrant from
 http:/ /bit.ly/vagrant-dl Env Setup Get pre-reqs installed
4. •Clone the repo from
 http:/ /bit.ly/pycon-vm •git clone https://... Env Setup Clone Repo
5. •Navigate into the root of your checkout in your shell. Env Setup Get VM Running
6. vagrant plugin install vagrant-hostmanager
7. vagrant up
8. •Sean O'Connor • Director of Application Engineering • Has spoken at a number of conferences • @theSeanOC •Sarah Guido • Lead Data Scientist • Has also spoken at a number of conferences • @sarah_guido About us!
9. gph.is/1he9iDA
10. gph.is/1cH3iz6
11. SHORTLINKS? Really?
13. bit.ly/1wYAkjS
17. •Basics of large-scale data processing in Python • Pure Python • Hadoop/MapReduce • Spark • Final exercise •Mostly hands-on! About this tutorial!
18. gph.is/1a7BN2Z
19. gph.is/11wceSz
20. •A "decode" is a click on a Bitly link •Lots of fields: • h: Bitly user hash identifier • g: Bitly global hash identifier • a: browser user agent • u: long URL • t: timestamp (UTC) • c: country (two-letter code) • nk: repeat client • kw: keyword alias for user hash • ckw: custom keyword • cy: city (optional) All about the data! Decode data 1usagov_data 1usagov_data_small 1usagov_data_tiny
21. •Fields • Agency • Ex. Library of Congress • City • Domain Name • Ex. loc.gov • Domain Type • Ex. Federal Agency • Global Hash • Same as in decodes data • Hostname • Ex. www.loc.gov • State All about the data! Agency data agency_map
22. vagrant ssh
23. cd /vagrant
24. ./refresh_data.sh
25. ./refresh_exercises.sh
26. Pure Python Easy but not scalable
27. •Open the small data file. •Parse each line. •Count the clicks per link. •Output a TSV of results. Pure Python Count Clicks EXERCISE 1.1 Try on your own for 5 minutes and then we’ll review a solution
28. •Build on Exercise 1.1 •Count clicks by country in addition to link. •Output results as a three column TSV:
 type, value, count Pure Python Count Clicks & Countries EXERCISE 1.2 Try on your own for 5 minutes and then we’ll review a solution
29. •Build on Exercise 1.2 •Sort your results and return only the top 20 links and 20 countries in descending order. •Output results as a three column TSV:
 type, value, count Pure Python Top Links & Countries EXERCISE 1.3 Try on your own for 5 minutes and then we’ll review a solution
30. •Build on Exercise 1.3 •Merge in the agency data •Calculate the top 20 agencies, in addition to the top countries and links. •Output results as a three column TSV:
 type, value, count Pure Python Join With Agencies EXERCISE 1.4 Try on your own for 5 minutes and then we’ll review a solution
31. •Build on Exercise 1.4 •Filter to clicks on links with a agency of “Department of State”. •Calculate the top 20 links and countries within the filter. •Output results as a three column TSV:
 type, value, count Pure Python Filter By Agency EXERCISE 1.5 Try on your own for 5 minutes and then we’ll review a solution
32. Hadoop Not easy but scalable
33. bit.ly/1wYKtNB
34. bit.ly/1TMlFr9
35. gph.is/1sFS6ic
36. http:/ /bit.ly/pymrjob
37. vagrant ssh
38. cd /vagrant/exercises/hadoop_exercies/2.1/
39. ./mr_count.py ../../../sample_data/1usagov_data
40. ./mr_count.py -r hadoop \
 hdfs:///user/vagrant/sample_data/1usagov_data
41. •Copy the boilerplate.py file into your working file. •Your goal is to count clicks by hash. •Write a mapper, combiner, and reducer to accomplish this. •Run the job on your VM. Hadoop Count Clicks EXERCISE 2.1 Try on your own for 5 minutes and then we’ll review a solution
42. •Build on Exercise 2.1 •Count clicks by country in addition to link. •Update your mapper, combiner, and reducer to accomplish this. •Run the job on your VM. Hadoop EXERCISE 2.2 Try on your own for 5 minutes and then we’ll review a solution Count Clicks & Countries
43. •Build on Exercise 2.2 •Sort your results and return only the top 20 links and 20 countries in descending order. •Hint: you will want to update your reducer to accomplish this. •Run the job on your VM. Hadoop EXERCISE 2.3 Try on your own for 5 minutes and then we’ll review a solution Top Links & Countries
44. •Build on Exercise 2.3 •Merge in the agency data •Calculate the top 20 agencies, in addition to the top countries and links. •Run the job on your VM. Hadoop EXERCISE 2.4 Try on your own for 5 minutes and then we’ll review a solution Join With Agencies
45. •Build on Exercise 2.4 •Filter to clicks on links with a agency of “Department of State”. •Calculate the top 20 links and countries within the filter. •Run the job on your VM. Hadoop EXERCISE 2.5 Try on your own for 5 minutes and then we’ll review a solution Join With Agencies
46. Spark Made for large-scale data!
47. •Large-scale distributed data processing tool •SQL and streaming tools •Faster than Hadoop •Python, Scala, Java, R APIs Spark The What
48. •Fast. Really fast. •SQL layer – kind of like Hive •Distributed scientific tools •Python! Sometimes. •Cutting edge technology Spark The Why
49. •Partitions your data to operate over in parallel •Capability to add map/ reduce features •Lazy – only operates when a method is called (ex. collect()/or writing to file) Spark The How
50. •Documentation is not mature •Major/frequent releases •Python API not as robust as Scala API Spark Caveats
51. •Lots of data •Need for large-scale analysis tools •Used to process data, build machine learning systems, explore relationships in data Spark Spark at Bitly
52. •RDDs •DataFrames •Launch the interactive PySpark shell by typing "pyspark" in the vm •Let's walk through some basics! Spark Basics
53. •To the shell! •RDD vs. DataFrame WALKTHROUGH Spark Count Total Clicks
54. •Group by WALKTHROUGH Spark Count Clicks By Hash
55. •Join WALKTHROUGH Spark Join to Agency Data
56. •What are the top countries per link? •What are the top links per country? •Looking at 20 is fine (show() will output 20) EXERCISE 3.1 Spark Top Links & Countries Try on your own for 5 minutes and then we’ll review a solution
57. •What are the top links with the Library of Congress as the agency? •What are the top countries with the Department of Education as the agency? EXERCISE 3.2 Spark Filter by Agency Try on your own for 5 minutes and then we’ll review a solution
58. Your mission Should you choose to accept it...
59. •Find the breakdown of agencies for China (CN), Denmark (DE), and France (FR) •Find the most popular links for 3 agencies of your choice •How do they differ between countries? Are there any similarities? EXERCISE 4 More agencies, Try on your own for 15 minutes and then we’ll review a solution links, and countries
60. Solutions!
61. •https:/ /github.com/SeanOC/ pycon-2016-tutorial-vm •https:/ /github.com/sarguido/ large-scale-data-analysis •https:/ /www.python.org/ •https:/ /pythonhosted.org/ mrjob/ •https:/ /spark.apache.org/ • https:/ /databricks.com/blog • http:/ /blog.cloudera.com/ blog/category/spark/ Resources Check these out to learn more!
62. THANK YOU Sarah Guido Lead Data Scientist @sarah_guido Sean O'Connor Director of Application Engineering @theSeanOC