Hadoop 101

Slides from BarCamp

Transcript

1. Hadoop 101
   Mohit Soni
   eBay Inc.
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

2. About Me
   • I work as a Software Engineer at eBay
   • Worked on large-scale data processing with
   eBay Research Labs
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

3. First Things First
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

4. • Inspired from functional operations
   – Map
   – Reduce
   • Functional operations do not modify data,
   they generate new data
   • Original data remains unmodified
   MapReduce
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

5. MapReduce
   def MapReduce(data, mapper, reducer):
   return reduce(reducer, map(mapper, data))
   MapReduce(list, sqr, add) -> 30
   Functional Operations
   BarCamp Chennai - 5 Mohit Soni
   Map
   def sqr(n):
   return n * n
   list = [1,2,3,4]
   map(sqr, list) -> [1,4,9,16]
   Reduce
   def add(i, j):
   return i + j
   list = [1,2,3,4]
   reduce(add, list) -> 10
   Python code
   

   View Slide

6. BarCamp Chennai - 5 Mohit Soni
   

   View Slide

7. • Framework for large-scale data processing
   • Based on Google’s MapReduce and GFS
   • An Apache Software Foundation project
   • Open Source!
   • Written in Java
   • Oh, btw
   What is Hadoop ?
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

8. • Need to process lots of data (PetaByte scale)
   • Need to parallelize processing across
   multitude of CPUs
   • Achieves above while KeepIng Software
   Simple
   • Gives scalability with low-cost commodity
   hardware
   Why Hadoop ?
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

9. Source: Hadoop Wiki
   Hadoop fans
   BarCamp Chennai - 5 Mohit Soni
   

   View Slide

10. Hadoop is a good choice for:
    • Indexing data
    • Log Analysis
    • Image manipulation
    • Sorting large-scale data
    • Data Mining
    When to use and not-use Hadoop ?
    BarCamp Chennai - 5 Mohit Soni
    Hadoop is not a good choice:
    • For real-time processing
    • For processing intensive tasks with little data
    • If you have Jaguar or RoadRunner in your stock
    

    View Slide

11. • Hadoop Distributed File System
    • Based on Google’s GFS (Google File System)
    • Write once read many access model
    • Fault tolerant
    • Efficient for batch-processing
    HDFS – Overview
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

12. • HDFS splits input data into blocks
    • Block size in HDFS: 64/128MB (configurable)
    • Block size *nix: 4KB
    HDFS – Blocks
    BarCamp Chennai - 5 Mohit Soni
    Block 1
    Block 2
    Block 3
    Input Data
    

    View Slide

13. • Blocks are replicated across nodes to handle hardware failure
    • Node failure is handled gracefully, without loss of data
    HDFS – Replication
    BarCamp Chennai - 5 Mohit Soni
    Block 1
    Block 2
    Block 1
    Block 3
    Block 2
    Block 3
    

    View Slide

14. HDFS – Architecture
    BarCamp Chennai - 5 Mohit Soni
    NameNode
    Client
    Cluster
    DataNodes
    

    View Slide

15. • NameNode (Master)
    – Manages filesystem metadata
    – Manages replication of blocks
    – Manages read/write access to files
    • Metadata
    – List of files
    – List of blocks that constitutes a file
    – List of DataNodes on which blocks reside, etc
    • Single Point of Failure (candidate for spending $$)
    HDFS – NameNode
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

16. • DataNode (Slave)
    – Contains actual data
    – Manages data blocks
    – Informs NameNode about block IDs stored
    – Client read/write data blocks from DataNode
    – Performs block replication as instructed by NameNode
    • Block Replication
    – Supports various pluggable replication strategies
    – Clients read blocks from nearest DataNode
    • Data Pipelining
    – Client write block to first DataNode
    – First DataNode forwards data to next DataNode in pipeline
    – When block is replicated across all replicas, next block is chosen
    HDFS – DataNode
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

17. Hadoop - Architecture
    BarCamp Chennai - 5 Mohit Soni
    JobTracker
    TaskTracker TaskTracker
    NameNode
    DataNode
    DataNode
    DataNode
    DataNode
    DataNode
    DataNode
    User
    

    View Slide

18. • JobTracker (Master)
    – 1 Job Tracker per cluster
    – Accepts job requests from users
    – Schedule Map and Reduce tasks for TaskTrackers
    – Monitors tasks and TaskTrackers status
    – Re-execute task on failure
    • TaskTracker (Slave)
    – Multiple TaskTrackers in a cluster
    – Run Map and Reduce tasks
    Hadoop - Terminology
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

19. Input
    Data
    Input Map Shuffle + Sort Reduce Output
    Map
    Map
    Map
    Output
    Data
    Reduce
    Reduce
    BarCamp Chennai - 5 Mohit Soni
    MapReduce – Flow
    

    View Slide

20. Word Count
    Hadoop’s HelloWorld
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

21. • Input
    – Text files
    • Output
    – Single file containing (Word Count)
    • Map Phase
    – Generates (Word, Count) pairs
    – [{a,1}, {b,1}, {a,1}] [{a,2}, {b,3}, {c,5}] [{a,3}, {b,1}, {c,1}]
    • Reduce Phase
    – For each word, calculates aggregate
    – [{a,7}, {b,5}, {c,6}]
    BarCamp Chennai - 5 Mohit Soni
    Word Count Example
    

    View Slide

22. public class WordCountMapper extends MapReduceBase implements
    Mapper {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    public void map(LongWritable key, Text value, OutputCollectorIntWritable> out, Reporter reporter) throws Exception {
    String l = value.toString();
    StringTokenizer t = new StringTokenizer(l);
    while(t.hasMoreTokens()) {
    word.set(t.nextToken());
    out.collect(word, one);
    }
    }
    }
    BarCamp Chennai - 5 Mohit Soni
    Word Count – Mapper
    

    View Slide

23. public class WordCountReducer extends MapReduceBase implements
    Reducer {
    public void reduce(Text key, Iterator values,
    OutputCollector out, Reporter reporter) throws
    Exception {
    int sum = 0;
    while(values.hasNext()) {
    sum += values.next().get();
    }
    out.collect(key, new IntWritable(sum));
    }
    }
    BarCamp Chennai - 5 Mohit Soni
    Word Count – Reducer
    

    View Slide

24. public class WordCountConfig {
    public static void main(String[] args) throws Exception {
    if (args.length() != 2) {
    System.exit(1);
    }
    JobConf conf = new JobConf(WordCountConfig.class);
    conf.setJobName(“Word Counter”);
    FileInputFormat.addInputPath(conf, new Path(args[0]);
    FileInputFormat.addOutputPath(conf, new Path(args[1]));
    conf.setMapperClass(WordCountMapper.class);
    conf.setCombinerClass(WordCountReducer.class);
    conf.setReducerClass(WordCountReducer.class);
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);
    JobClient.runJob(conf);
    }
    }
    BarCamp Chennai - 5 Mohit Soni
    Word Count – Config
    

    View Slide

25. • http://hadoop.apache.org/
    • Jeffrey Dean and Sanjay Ghemwat, MapReduce:
    Simplified Data Processing on Large Clusters
    • Tom White, Hadoop: The Definitive Guide, O’Reilly
    • Setting up a Single-Node Cluster: http://bit.ly/glNzs4
    • Setting up a Multi-Node Cluster: http://bit.ly/f5KqCP
    Diving Deeper
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide

26. • Follow me on twitter @mohitsoni
    • http://mohitsoni.com/
    Catching-Up
    BarCamp Chennai - 5 Mohit Soni
    

    View Slide