Apache™ Pig - An Introduction

Quick introduction to Apache™ Pig

Transcript

1. Apache™ Pig An Introduction Moty Michaely June, 2015
2. Outline What is Apache™ Pig Motivation Hands On Use Cases Questions
3. Outline What is Apache™ Pig Motivation Hands On Use Cases Questions
4. Outline What is Apache™ Pig Motivation Hands On Use Cases Questions
5. Outline What is Apache™ Pig Motivation Hands On Use Cases Questions
6. Outline What is Apache™ Pig Motivation Hands On Use Cases Questions
7. What Is Apache™ Pig
8. “High-level platform for creating MapReduce programs used with Hadoop.” (Wikipedia)
9. Apache™ Pig is High Level Scripting Language Pig Latin Hadoop MapReduce/Tez Compiler Commonly Used Open Source
10. Apache™ Pig is High Level Scripting Language Hadoop MapReduce/Tez Compiler Supports MapReduce and Tez Commonly Used Open Source
11. Apache™ Pig is High Level Scripting Language Hadoop MapReduce/Tez Compiler Commonly Used Netflix, Xplenty, eBay, Yahoo, Wix... Open Source
12. Apache™ Pig is High Level Scripting Language Hadoop MapReduce/Tez Compiler Commonly Used Open Source Backed by the community
13. Why Pig (the name)? Pigs Eat Anything Relational, Nested, Unstructured Files, Key/Value stores, Databases Pigs Live Anywhere Pigs Are Domestic Animals Pigs Fly
14. Why Pig (the name)? Pigs Eat Anything Pigs Live Anywhere Not tied to particular framework Pigs Are Domestic Animals Pigs Fly
15. Why Pig (the name)? Pigs Eat Anything Pigs Live Anywhere Pigs Are Domestic Animals Easily controlled Integration of user code Pigs Fly
16. Why Pig (the name)? Pigs Eat Anything Pigs Live Anywhere Pigs Are Domestic Animals Pigs Fly Faster development Improved performance
17. Apache™ Pig architecture
18. APACHE PIG - Pig Latin scripting language - MR/Tez Compiler = RECAP
19. Motivation for Apache™ Pig
20. Motivation for Pig Increase productivity 10 lines of Pig Latin ≈ 200 lines of Java 4 hours of Java ≈ 15 minutes of Pig Latin Open to non-java developers Optimization opportunities Extensibility
21. Motivation for Pig Increase productivity Open to non-java developers It’s like SQL Optimization opportunities Extensibility
22. Motivation for Pig Increase productivity Open to non-java developers Optimization opportunities No need to tune Hadoop for your needs Execution plan, optimizer Extensibility
23. Motivation for Pig Increase productivity Open to non-java developers Optimization opportunities Extensibility User defined functions Integration with Python, Ruby and JS
24. MOTIVATION - Productivity - Non-java developers - Optimization - Extensibility = RECAP
25. Hands On Apache™ Pig
26. Pig Latin (For Apache™ Pig) “A high-level language that allows you to write data processing and analysis programs.”
27. Pig Latin For Apache™ Pig Relations - A relation (table) is a bag - A bag is a collection of tuples - A tuple (row) is an ordered set of fields - A field is a piece of data
28. Running Pig - Two execution modes Interactive Mode $ cd /path/to/pig/bin/ $ pig grunt> a = LOAD ‘/path/to/file’; grunt> DUMP a; Batch Mode
29. Running Pig Pig supports two execution modes Interactive Mode Batch Mode $ cd /path/to/pig/bin/ $ pig -f /path/to/pig/file.pig
30. Word Count Problem “Given text files, return how often words occur”
31. Word Count in MR
32. Word Count in MR (Mapper)
33. Word Count in MR (Reducer)
34. Word Count in MR
35. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
36. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
37. -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount'; Word Count in Pig
38. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
39. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
40. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
41. Word Count in Pig -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; STORE counts INTO 'wordcount';
42. Word Count in Pig - Sorted -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; sorted_counts = ORDER counts BY count DESC, word ASC; STORE counts INTO 'wordcount';
43. Word Count in Pig - Sorted -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(words) AS count; sorted_counts = ORDER counts BY count DESC, word ASC; STORE sorted_counts INTO 'wordcount_sorted';
44. Word Count - MR vs. Pig 63 Lines of code 5 lines of code -- Word Count Script (wordcount.pig) text = LOAD 'word_count_text.txt'; words = FOREACH text GENERATE FLATTEN(TOKENIZE((chararray)$0)) AS word; grouped_words = GROUP words BY word; counts = FOREACH grouped_words GENERATE group AS word, COUNT(grouped_words) AS count; STORE counts INTO 'wordcount';
45. HANDS ON - Word Count - Execution Plan - Optimization = RECAP
46. Use Cases Of Apache™ Pig
47. “80% of the work in any data project is in cleaning the data.” (D.J Patel, Data Jujitsu)
48. Pig is great for Web log processing Data processing for web search platforms Ad hoc queries across large data sets Rapid prototyping of algorithms for processing large data sets
49. Pig is great for Web log processing Data processing for web search platforms Ad hoc queries across large data sets Rapid prototyping of algorithms for processing large data sets
50. Pig is great for Web log processing Data processing for web search platforms Ad hoc queries across large data sets Rapid prototyping of algorithms for processing large data sets
51. Pig is great for Web log processing Data processing for web search platforms Ad hoc queries across large data sets Rapid prototyping of algorithms for processing large data sets
52. Questions
53. Resources Apache Pig Philosophy Apache Pig 0.14 Documentation Word Count Example Introduction to Apache Tez Pig (Language) Pig for dummies Pig Latin (Language Game) Xplenty Data Jujitsu: The art of turning data into product Pig Cheat Sheet
54. THANK YOU moty.mi@gmail.com