How It Works - Hadoop

A series of talks on Data Engineering

Transcript

1. HOW IT WORKS HOW IT WORKS HADOOP HADOOP 1
2. PLAN PLAN What is Hadoop? HDFS MapReduce Yarn EMR Ecosystem Example & Walkthrough 2 . 1
3. WHAT IS HADOOP? WHAT IS HADOOP? Original: Framework (Hadoop Core) HDFS MapReduce (Yarn) Ecosystem/Solution "Hadoop solution" vs classic RDBMS warehousing Ecosystem of tools and frameworks originated from Hadoop or its principles 3 . 1
4. HADOOP CLUSTER HADOOP CLUSTER 4 . 1
5. 4 . 2
6. HADOOP CHARACTERISTICS HADOOP CHARACTERISTICS open-source commodity hardware [semi/un]structured data, schema-on-read fault-tolerant, highly-available scalable (thousands of nodes) highly-parallel computation data-locality 5 . 1
7. HDFS HDFS 6 . 1
8. 6 . 2
9. 6 . 3
10. HDFS CHARACTERISTICS HDFS CHARACTERISTICS almost POSIX `hdfs dfs -ls -a user/spark..` URI scheme: hdfs://92.23.23.23/user/spark/.. les split into 64mb blocks 6 . 4
11. MAPREDUCE MAPREDUCE 7 . 1
12. MAPREDUCE HIGHLIGHTS MAPREDUCE HIGHLIGHTS any job is represented as series of map/reduce steps embedded data locality high scalability relatively slow complex 7 . 2
13. YARN YARN 8 . 1
14. YARN CHARACTERISTICS YARN CHARACTERISTICS containers, exible resource-utilization multi-tenancy transparant model 8 . 2
15. HADOOP CLUSTER (AGAIN) HADOOP CLUSTER (AGAIN) 9 . 1
17. 9 . 2
18. HADOOP LIMITATIONS HADOOP LIMITATIONS a lot of small les (each le still occupies 64mb block) MapReduce is slow no ACID, append-only (or rewrite) only batch-processing highly complex (con guration & programming api) 10 . 1
19. MAPREDUCE EXAMPLE MAPREDUCE EXAMPLE 11 . 1
20. HADOOP ECOSYSTEM HADOOP ECOSYSTEM 12 . 1
21. HBASE HBASE 12 . 2
22. 12 . 3
23. ECOSYSTEM ECOSYSTEM 12 . 4
24. 12 . 5
25. AWS ELASTIC MAPREDUCE (EMR) AWS ELASTIC MAPREDUCE (EMR) 13 . 1
26. HADOOP DISTRIBUTIONS HADOOP DISTRIBUTIONS On-premise/cloud Cloudera Hortonworks MapR Cloud AWS EMR Azure HDInsight GCP DataProc IBM BigInsights 13 . 2
27. EMR ARCHITECTURE EMR ARCHITECTURE 13 . 3
28. EMR FEATURES EMR FEATURES ec2 control con guration & components version mngmnt exible resource-utilization, cost model fast provisioning auto-scaling high availability integration cloud-formation cloud-watch s3 (emrfs) aws glue, dynamodb 13 . 4
29. EMR APPLICATIONS EMR APPLICATIONS 13 . 5
30. TIM WALKTHROUGH TIM WALKTHROUGH emr console yarn interface master overview cf overview 13 . 6
31. WHAT'S NEXT WHAT'S NEXT Hadoop: The De nitive Guide WordCount with MapReduce 14 . 1
32. 15 . 1
33. DEMO DEMO hadoop version: 3.1.3 core-site: <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> hdfs-site: <property> <name>dfs.replication</name <value>1</value> </property> mapred-site: <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOPMAPREDHOME=/home/twist/Down 3.1.3</value> </property> <property>
34. <name>mapreduce.map.env</name> <value>HADOOPMAPREDHOME=/home/twist/Down 3.1.3</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOPMAPREDHOME=/home/twist/Down 3.1.3</value> </property> yarn-site: <property> <name>yarn.nodemanager.au services</name> <value>mapreduceshuf e</value> <property> <name>yarn.nodemanager.aux- services.mapreduceshuf e.class</name> <value>org.apache.hadoop.mapred.Shuf eHandler< </property> format hdfs
35. start dfs, start yarn word-count javac WordCount.java export JAVAHOME=/usr/lib/jv export PATH= {PATH} export HADOOPCLASSPATH=${JAVAHOME}/lib/tools.jar jar import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount { public static class TokenizerMapper J AV OM E/bin : AH
36. WordCount*.class bin/hadoop jar wc.jar WordCount /user/twist/input /user/twist/wordcount/output 16 . 1