Hadoop on Node.js

Transcript

1. Hadoop & Node.js Casear Chu
2. Analysis
3. SQL? OLAP ? Cube? Tabular?
4. BIG DATA
5. Big Data       
       
      • • • •
6. Data Process
7. 12TB? 1HDD=75MB/sec
8. 45Mins!!!!
9. 1000HDDs =75MB/s*1000 =75GB/s
10. 3Mins!!
11. Hadoop
12. Hadoop Node  2011120 (2010  66) 40%10-100 52%100-1000 Hadoop Data  2011㾝 202 PB (2010  3.4
    ) • • • • • •
13. Hadoop=HDFS+MapReduce
14. Example Google Facebook Yahoo IBM eBay
15. Hadoop Version 0.20  
     0.20.X 0.20 
     0.20.20X.YLegacy Stable  Security Branch 0.20.205.0 1.0.XCurrent Stable • • • • •
16. Role NameNode Data Node JobTracker TaskTracker • • • •
17. MapReduce
18. MapReduce      
    • •
19. Code
    Compile
20. HIVE
21. HIVE
22. PIG
23. 
    
25. Mahout
26. Long Tail
27. Hadoop Setting
28. Local (Standalone) Mode Pseudo-Distributed Mode Fully-Distributed Mode
29. hadoop-env.sh export JAVA_HOME=”jdk path” •
30. core-site.xml <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration> • • • • • •
31. hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> • • • • • •
32. mapred.xml <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:9001</value> </property> </configuration> • • • • • •
34. Timothy Open Source Use Hadoop Stream API • •
35. Why node.js No binaries NPM Support • •
36. npm install timothy
37. Function configure dependencies setup map reduce run • • • • • •
38. Word Count
39. require('timothy') .map(function(line){ var words = line.split(" "); for(var i=0; i<words.length; i++) this.emit(words[i], 1); }) .reduce(function(word,counts){ this.emit(word, counts.length); }) .runLocal("~/Desktop/test_input.txt"); •
40. hadoop fs -put <src> <dest> hadoop fs -get <src> <dest>