Hadooping with Ruby

Using Hadoop Streaming with Ruby

Transcript

1. system insights Deepak Prasanna R System Insights www.systeminsights.com www.manufacturingbigdata.com Hadooping with Ruby Tuesday, April 23, 13
2. system insights • R & D engineer @systeminsights • Hacking ruby, scala and javascript. • Filtercoffee enthusiast • VIM fanboy • Chennai machan About me Tuesday, April 23, 13
3. system insights • Brief introduction to Hadoop • Hadoop streaming with ruby Agenda Tuesday, April 23, 13
4. system insights What we do? The predictive analytics platform for manufacturing intelligence Tuesday, April 23, 13
5. system insights • Cycle classifications. Ex: producing, loading • Downtime classifications • Machine tool analysis • Machine tool recommendations • And more... What we do with data? Tuesday, April 23, 13
6. system insights • 2008: Google processes 20 PB a day • 2009: Facebook has 2.5 PB user data + 15 TB/day • 2009: eBay has 6.5 PB user data + 50 TB/day • 2011: Yahoo! has 180-200 PB of data • 2012: Facebook ingests 500 TB/day How big is big data for the big guys? Tuesday, April 23, 13
7. system insights • 2009 - ~ 1TB • 2010 - ~ 1.5 TB • 2011 - ~ 2.5 TB • 2012 - ~ 4 TB • 2013 - ~ 3 TB and counting ... How big is big data for us? Tuesday, April 23, 13
8. system insights • Machine tool data • Power data What kind of data? Tuesday, April 23, 13
9. How we analyze data now? system insights Tuesday, April 23, 13
10. system insights • Has been the next big thing for a long time Why Hadoop? Tuesday, April 23, 13
11. Divide and Conquer system insights Tuesday, April 23, 13
12. system insights • How do we assign tasks to workers? • What happens when tasks fail? • How do you handle distributed synchronization? • And more.. But parallel processing is super difficult Tuesday, April 23, 13
13. system insights • Hadoop is a framework which helps you to setup an infrastructure which will do • Distributed storage(HDFS) • Distributed computing(Map/Reduce) What is Hadoop? Tuesday, April 23, 13
14. system insights • Distributed file system designed to run on commodity hardware • Highly fault-tolerant and designed to handle hardware failure HDFS Tuesday, April 23, 13
15. system insights • Files are split in 64MB chunks • Processes • NameNode • DataNode HDFS Tuesday, April 23, 13
16. HDFS architechture system insights Tuesday, April 23, 13
17. system insights hadoop dfs -lsr hadoop dfs -copyFromLocal files/* /user/hduser/gutenberg hadoop dfs -lsr Putting some files inside HDFS Tuesday, April 23, 13
18. system insights #wordcount #what goes to a mapper Hello World Bye World Hello Hadoop Goodbye Hadoop #what comes out of a mapper < Hello, 1> < World, 1> < Bye, 1> < World, 1> < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> Mapper Tuesday, April 23, 13
19. system insights #Before shuffling and sorting < Hello, 1> < World, 1> < Bye, 1> < World, 1> < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> #After shuffling and sorting < Bye, 1> < Goodbye, 1> < Hadoop, 1> < Hadoop, 1> < Hello, 1> < Hello, 1> < World, 1> < World, 1> Shuffling and Sorting Tuesday, April 23, 13
20. system insights #what goes to a reducer < Bye, 1> < Goodbye, 1> < Hadoop, 1> < Hadoop, 1> < Hello, 1> < Hello, 1> < World, 1> < World, 1> #what comes out of a reducer < Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> Reducer Tuesday, April 23, 13
21. system insights • Hadoop Streaming is a utility which allows users to create and run jobs with any executables as the mapper and/or the reducer. • STDIN, STDOUT magic <key><tab><value><newline> Ex: “Hello \t 1 \n” What is Hadoop Streaming? Tuesday, April 23, 13
22. system insights # Ruby code for map.rb ARGF.each do |line| # remove any newline line = line.chomp line.split(" ").each do |w| # output to STDOUT # <key><tab><value><newline> puts w.downcase + "\t" + 1.to_s end end mapper.rb Tuesday, April 23, 13
23. system insights # Ruby code for reduce.rb prev_key = nil key_total = 0 ARGF.each do |line| # remove any newline line = line.chomp # split key and value on tab character (key, value) = line.split(/\t/) # check for new key if prev_key && key != prev_key && key_total > 0 # output total for previous key # <key><tab><value><newline> puts prev_key + "\t" + key_total.to_s # reset key total for new key prev_key = key key_total = 0 elsif ! prev_key prev_key = key end # add to count for this current key key_total += value.to_i end reducer.rb Tuesday, April 23, 13
24. system insights #!/bin/bash hadoop \ jar /usr/local/hadoop/contrib/streaming/hadoop- streaming-1.1.1.jar \ -mapper 'ruby /home/hduser/ruby/map.rb' \ -reducer 'ruby /home/hduser/ruby/reduce.rb' \ -file /home/hduser/ruby/map.rb \ -file /home/hduser/ruby/reduce.rb \ -input '/user/hduser/gutenberg/*' \ -output '/user/hduser/gutenberg-output/' Hadoop Streaming Tuesday, April 23, 13
25. system insights • Bill Graham, @billgraham • Rohit Menon, http://www.rohitmenon.com/ • Phil Whelan, @philwhln • Michael G. Noll, @miguno Credits Tuesday, April 23, 13