Timothy

Write Hadoop Jobs in NodeJS by Antonio Garrote and Abhinay Mehta

Transcript

1. Timothy https://github.com/forward/timothy Antonio Garrote Abhinay Mehta
2. Hadoop MapReduce in Node.js
3. Hadoop • Distributed processing of large data • Derived from Google MapReduce and GFS • Fast becoming the de facto standard • Large ecosystem • Java • Master/Slave setup
4. Hadoop Architecture HDFS MapReduce Output Input
5. • Open Source • Uses Hadoop Streaming API • No binaries • NPM support Timothy
6. $ npm install timothy
7. require('timothy') .configure({ config: './hadoop.xml', input: '/tmp/loremipsum.txt', output: '/tmp/wordcount/', name: 'Timothy Word Count Example' }) .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){ emit(word, counts.length); }) .run(function(err){ .. }); Word Count
8. require('timothy') .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){ emit(word, counts.length); }) .runLocal("/local/input/path"); Local Runner
9. require('timothy') .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){ emit(word, counts.length); }) .runLocal("/local/input/path"); Local Runner
10. Dependencies require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ var S = require('string'); line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ...
11. Dependencies require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ var S = require('string'); line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ...
12. Setup require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ... .setup(function() { S = require('string'); })
13. Setup require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ... .setup(function() { S = require('string'); })
14. require('timothy') .configure({ ... }) .map(function(line){ emit(line, 1); }) .reduce(function(line,counts){ emit(line, counts.length); }) .map(function(line, count){ emit(line[0], count); }) .reduce(function(letter, counts){ var sum = counts.reduce(function(a,i) { return a+i; }); emit(letter, sum); }) .run(); Method Chaining
15. • Update Job Status • Create and update counters • Pass env vars to jobs • More examples on github page Other features
16. Motivation • Big data is now a thing • Lower the barrier to entry • Benefits of NodeJS on Hadoop • Development Speed
17. Limitations • Setup method cannot block • Lack support for lexical scoping • NodeJS needs to be pre-installed on slaves • Probably more we haven’t thought of yet!
18. Improvements • Bundling local JS scripts • JSON for intermediary data format • JVM support
19. Thank you! https://github.com/forward/timothy