Hello World! in Hadoop

Every journey to learn new programming language or paradigm starts with the simple Hello World! program. The Hadoop world is no different, and it's Hello World equivalent is the Word Count Program.

The presentation covers the basic concepts required to understand the Word Count program and how one can go about setting it up and executing it on your own.

Transcript

1. HELLO WORLD! IN HADOOP Chandra Yarlagadda randommuses chandrayarlagadda twitter.com/ linkedin.com/in/ Email : cdyarlagadda@gmail.com
2. Titanium Sponsors Platinum Sponsors Gold Sponsors
5. Location / People / Events SMB Analytics Infrastructure Analytics Applications Data Sources Storage Security Crowdsourcing Hadoop Related NoSQL Databases Data Visualization Cluster Services Personal Data MPP Databases Industry Applications Social Media Sentiment Analysis Analytics Solutions Crowdsourced Analytics IT Analytics Data Sources Data Marketplaces Publisher Tools Marketing Management / Monitoring Real-Time Ad Optimization Statistical Computing Cross Infrastructure / Analytics Application Service Providers Big Data Search Analytics Services Collection / Transport Open Source Projects Framework Query / Data Flow Data Access Coordination / Workflow Machine Learning Real - Time Statistical Tools Cloud Deployment NewSQL Databases Source : www.bigdatalandscape.com
6. Location / People / Events SMB Analytics Infrastructure Analytics Applications Data Sources Storage Security Crowdsourcing Hadoop Related NoSQL Databases Data Visualization Cluster Services Personal Data MPP Databases Industry Applications Social Media Sentiment Analysis Analytics Solutions Crowdsourced Analytics IT Analytics Data Sources Data Marketplaces Publisher Tools Marketing Management / Monitoring Real-Time Ad Optimization Statistical Computing Cross Infrastructure / Analytics Application Service Providers Big Data Search Analytics Services Collection / Transport Open Source Projects Framework Query / Data Flow Data Access Coordination / Workflow Machine Learning Real - Time Statistical Tools Cloud Deployment NewSQL Databases Source : www.bigdatalandscape.com
9. HDFS
10. HDFS MAP REDUCE
12. What is Big Data?
13. What is Big Data?
14. What is Big Data? Volume Variety Velocity
16. Before We Jump In!
17. Before We Jump In! • Hadoop 1.2.1
18. Before We Jump In! • Hadoop 1.2.1 • Daemon
19. Before We Jump In! • Hadoop 1.2.1 • Daemon • Pseudo - Distributed Mode
20. Before We Jump In! • Hadoop 1.2.1 • Daemon • Pseudo - Distributed Mode • Cloudera VM
21. Before We Jump In! • Hadoop 1.2.1 • Daemon • Pseudo - Distributed Mode • Cloudera VM • Do Not Start the Cloudera Manager
22. Before We Jump In! • Hadoop 1.2.1 • Daemon • Pseudo - Distributed Mode • Cloudera VM • Do Not Start the Cloudera Manager • Word Count
24. HDFS
25. HDFS • Hadoop Distributed File System
26. HDFS • Hadoop Distributed File System • Designed for Storing Very large files with
27. HDFS • Hadoop Distributed File System • Designed for Storing Very large files with • Streaming data access
28. HDFS • Hadoop Distributed File System • Designed for Storing Very large files with • Streaming data access • running on clusters of Commodity hardware
30. `
31. ` Data Nodes
32. ` Data Nodes
33. ` Name Node Data Nodes
34. ` Name Node Data Nodes
35. ` Name Node Block1 Block2 Block3 Data Nodes
36. ` Name Node Block1 Block2 Block3 Block1 Block2 Block3 Data Nodes
37. ` Name Node Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes
38. ` Name Node Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes
39. ` Name Node Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
40. ` Name Node fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
41. ` Name Node fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
42. ` Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
43. ` Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
44. ` Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
45. ` Name Node Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
46. ` Name Node Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
47. ` Name Node Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
48. ` Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
49. ` Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
50. ` Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
51. ` Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
52. ` Secondary Name Node Edit Log fsImage Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
53. ` Secondary Name Node Edit Log fsImage New Name Node Block1 Block2 Block3 Block1 Block1 Block2 Block2 Block3 Block3 Data Nodes Block1
55. HDFS - ATTRIBUTES
56. HDFS - ATTRIBUTES • Distributed
57. HDFS - ATTRIBUTES • Distributed • Write Once - Read Many Times Pattern
58. HDFS - ATTRIBUTES • Distributed • Write Once - Read Many Times Pattern • High Throughput - High Latency
59. HDFS - ATTRIBUTES • Distributed • Write Once - Read Many Times Pattern • High Throughput - High Latency • CAP ( Consitency - Availability - Parition Tolerance)
60. HDFS - ATTRIBUTES • Distributed • Write Once - Read Many Times Pattern • High Throughput - High Latency • CAP ( Consitency - Availability - Parition Tolerance) • Not Suitable for a lot of Small Files
61. HDFS - ATTRIBUTES • Distributed • Write Once - Read Many Times Pattern • High Throughput - High Latency • CAP ( Consitency - Availability - Parition Tolerance) • Not Suitable for a lot of Small Files • High Network Utilization and Disk I/O
63. MAP REDUCE
64. MAP REDUCE It is a programming model and an associated implementation for
65. MAP REDUCE It is a programming model and an associated implementation for • Processing and Generating large data sets with a
66. MAP REDUCE It is a programming model and an associated implementation for • Processing and Generating large data sets with a • Parallel,
67. MAP REDUCE It is a programming model and an associated implementation for • Processing and Generating large data sets with a • Parallel, • Distributed Algorithm on a
68. MAP REDUCE It is a programming model and an associated implementation for • Processing and Generating large data sets with a • Parallel, • Distributed Algorithm on a • Cluster ( HDFS)
70. MAP REDUCE - KEY POINTS
71. MAP REDUCE - KEY POINTS • Map and Reduce Phase
72. MAP REDUCE - KEY POINTS • Map and Reduce Phase • User defines the Map and Reduce Functions
73. MAP REDUCE - KEY POINTS • Map and Reduce Phase • User defines the Map and Reduce Functions • Both Phases have a set of Key - Value Pairs
75. MAP REDUCE - SINGLE REDUCER
76. MAP REDUCE - SINGLE REDUCER
77. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Reducer
78. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer
79. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer
80. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer
81. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer
82. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer Name Node
83. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Name Node
84. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node
85. MAP REDUCE - SINGLE REDUCER Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node
86. MAP REDUCE - MULTIPLE REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node Reducer
87. MAP REDUCE - MULTIPLE REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node Reducer
88. MAP REDUCE - MULTIPLE REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node Reducer Sort and Shuffle
89. MAP REDUCE - MULTIPLE REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node Reducer Sort and Shuffle
90. MAP REDUCE - MULTIPLE REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 Reducer HDFS Job Tracker Task Tracker Name Node Reducer Sort and Shuffle
91. MAP REDUCE - ZERO REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 HDFS Job Tracker Name Node
92. MAP REDUCE - ZERO REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 HDFS Job Tracker Name Node
93. MAP REDUCE - ZERO REDUCERS Mapper Mapper Mapper Input Split 1 Input Split 2 Input Split 3 HDFS Job Tracker Task Tracker Name Node
95. Key - Value
96. Key - Value map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue)
97. Key - Value 0 Twinkle Twinkle Little Star Little 1 Star 1 Twinkle 1,1 => map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue)
98. Key - Value 0 Twinkle Twinkle Little Star Little 1 Star 1 Twinkle 1,1 => map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue)
99. Key - Value 0 Twinkle Twinkle Little Star Little 1 Star 1 Twinkle 1,1 => Little 1 Star 1 Twinkle 1,1 => Little 1 Star 1 Twinkle 2 map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue)
101. MAP CODE
102. MAP CODE map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue)
103. MAP CODE public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue)
104. MAP CODE public static class extends map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) private Text word = new Text(); private final static IntWritable one = new IntWritable(1);
105. MAP CODE public static class extends map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); private private final static
106. MAP CODE public static class extends map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); private private final static 0 Twinkle Twinkle Little Star
107. MAP CODE public static class extends map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); private private final static 0 Twinkle Twinkle Little Star Little 1 Star 1 Twinkle 1,1
108. MAP CODE public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ map ( InputKey, InputValue) => Set of (IntermediateKey,IntermediateValue) public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); private Text word = new Text(); private final static IntWritable one = new IntWritable(1); } } }
110. REDUCE CODE
111. REDUCE CODE reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue)
112. REDUCE CODE public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{ reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue)
113. REDUCE CODE public static class extends reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue) private IntWritable result = new IntWritable();
114. REDUCE CODE public static class extends reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue) public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val:values) { sum += val.get(); } result.set(sum); context.write(key, result); private
115. REDUCE CODE public static class extends reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue) public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val:values) { sum += val.get(); } result.set(sum); context.write(key, result); private Little 1 Star 1 Twinkle 1,1 Little 1 Star 1 Twinkle 2
116. REDUCE CODE public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{ reduce(IntermediateKeys,IntermediateValues) => Set of (OutputKey,OutputValue) public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val:values) { sum += val.get(); } result.set(sum); context.write(key, result); private IntWritable result = new IntWritable(); } }
117. MAIN() public static void main(String[] args) throws Exception { Job job = Job.getInstance(conf, "word count"); Configuration conf = new Configuration(); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } job.setInputFormatClass(TextInputFormat.class);
119. DEMO TIME
121. RESOUCES
122. RESOUCES • https://hadoop.apache.org/
123. RESOUCES • https://hadoop.apache.org/ • Hadoop the Definitive Guide - Tom White
124. RESOUCES • https://hadoop.apache.org/ • Hadoop the Definitive Guide - Tom White • http://hortonworks.com/tutorials/
125. RESOUCES • https://hadoop.apache.org/ • Hadoop the Definitive Guide - Tom White • http://hortonworks.com/tutorials/ • http://www.cloudera.com/content/cloudera/en/documentation/core/ v5-3-x/topics/cloudera_quickstart_vm.html
127. Questions?