Timothy

Transcript

1. Timothy https://github.com/forward/timothy Antonio Garrote Abhinay Mehta

2. Hadoop MapReduce in Node.js

3. Hadoop • Distributed processing of large data • Derived from

   Google MapReduce and GFS • Fast becoming the de facto standard • Large ecosystem • Java • Master/Slave setup

4. Hadoop Architecture HDFS MapReduce Output Input

5. • Open Source • Uses Hadoop Streaming API • No

   binaries • NPM support Timothy

6. $ npm install timothy

7. require('timothy') .configure({ config: './hadoop.xml', input: '/tmp/loremipsum.txt', output: '/tmp/wordcount/', name: 'Timothy

   Word Count Example' }) .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){ emit(word, counts.length); }) .run(function(err){ .. }); Word Count

8. require('timothy') .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){

   emit(word, counts.length); }) .runLocal("/local/input/path"); Local Runner

9. require('timothy') .map(function(line){ line.split(" ").forEach(function(word) { emit(word, 1); }); }) .reduce(function(word,counts){

   emit(word, counts.length); }) .runLocal("/local/input/path"); Local Runner

10. Dependencies require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ var

    S = require('string'); line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ...

11. Dependencies require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ var

    S = require('string'); line.split(" ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ...

12. Setup require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ line.split("

    ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ... .setup(function() { S = require('string'); })

13. Setup require('timothy') .configure({ ... }) .dependencies({'string' : '0.2.1-2'}) .map(function(line){ line.split("

    ").forEach(function(word) { if (S(word).isAlphaNumeric()) { emit(word, 1); } }); }) ... .setup(function() { S = require('string'); })

14. require('timothy') .configure({ ... }) .map(function(line){ emit(line, 1); }) .reduce(function(line,counts){ emit(line,

    counts.length); }) .map(function(line, count){ emit(line[0], count); }) .reduce(function(letter, counts){ var sum = counts.reduce(function(a,i) { return a+i; }); emit(letter, sum); }) .run(); Method Chaining

15. • Update Job Status • Create and update counters •

    Pass env vars to jobs • More examples on github page Other features

16. Motivation • Big data is now a thing • Lower

    the barrier to entry • Benefits of NodeJS on Hadoop • Development Speed

17. Limitations • Setup method cannot block • Lack support for

    lexical scoping • NodeJS needs to be pre-installed on slaves • Probably more we haven’t thought of yet!

18. Improvements • Bundling local JS scripts • JSON for intermediary

    data format • JVM support

19. Thank you! https://github.com/forward/timothy