yarn分享

简单介绍一下yarn的原理。

Transcript

1. YARN 一淘搜索技术 2013-04-22 恨少
2. 1.   YARN介绍 2.   使用YARN 3.   参考资料
3. 1.   YARN介绍 2.   使用YARN 3.   参考资料
5. YARN: Yet Another Resource Negotiator
6. 23 May, 2012: Release 2.0.0-alpha available This is the first (alpha) version in the hadoop-2.x series. This delivers significant major features over the currently stable hadoop-1.x series including: HDFS HA for NameNode (manual failover) YARN aka NextGen MapReduce HDFS Federation Performance Wire-compatibility for both HDFS and YARN/MapReduce (using protobufs) Please see the Hadoop 2.0.0-alpha Release Notes for details. Hadoop 2.0 release note
7. 一些直观的感觉
10. RM： ResourceManager AsM: ApplicationsManager RM = AsM + Scheduler NM: NodeManager AM: ApplicationMaster
11. [master sbin] jps 1611 NameNode 2809 JobHistoryServer 1727 DataNode 2126 ResourceManager 1906 SecondaryNameNode 2229 NodeManager [slave~]$ jps 17687 NodeManager 17551 DataNode
12. Yarn与cgroups
13. REST API's $ curl -s 'http://master:8088/proxy/ application_1366622139965_0006/ws/v1/mapreduce/info' { "info": {  "appId":"application_1366622139965_0006",  "name":"wordcount",  "user":"henshao",  "startedOn":1366726680300,  "elapsedTime":28513 } }
14. 1.   YARN介绍 2.   使用YARN 3.   参考资料
15. hadoop jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar --jar hadoop-yarn-applications-distributedshell-2.0.3-alpha.jar   org.apache.hadoop.yarn.applications.distributedshell.Client --shell_command ~/start_sp.sh 调度一个服务？
16. The interfaces you'd most like be concerned with are: 1、ClientRMProtocol - Client<-->ResourceManager 2、AMRMProtocol - ApplicationMaster<-->ResourceManager 3、ContainerManager - ApplicationMaster<-->NodeManager 协议
19. $ ls hadoop-yarn-project/hadoop-yarn/hadoop-yarn- applications/hadoop-yarn-applications-distributedshell/ src/main/java/org/apache/hadoop/yarn/applications/ distributedshell/ ApplicationMaster.java Client.java  DSConstants.java 官方的例子
20. 几个问题 1、数据迁移问题。 YARN不适合调度IO密集型的任务。 2、AM单点问题。 需要YARN 继续完善其作为 Service 调度器的 功能。
21. 1.   YARN介绍 2.   使用YARN 3.   参考资料
22. 1、yarn 官方文档 2、 董的博客