An Introduction to Apache Hadoop Yarn

An Introduction to Apache Hadoop Yarn, what is it and why is
it important ? What does it improve in Apache Hadoop ?

Transcript

1. Apache Hadoop Yarn • What is Yarn • Problems with Hadoop • What does Yarn Do ? • Old Architecture • New Architecture • Yarn Example • Additions
2. Hadoop Yarn – What is it ? • Next Generation MapReduce MRv2 • Split Job Tracker into – Resource Manager – Scheduling / Monitoring • Improves scaling • Improves resource management • Already used by Yahoo
3. Problems with Hadoop 1.0 • Problems with large scaling – > 4000 nodes – > 40k concurrent tasks • Problems with resource utilization • Slots only for Map or Reduce • Single NameNode, single point of failure • Clients and Cluster must be at same version
4. What does Yarn do ? • Provides a cluster level resource manager • Adds application level resource management • Provides slots for jobs other than Map / Reduce • Improves resource utilization
5. Old Architecture • Cluster level Job Tracker, Task Tracker on data node
6. New Architecture
7. New Architecture • Resource Manager – Cluster level resource manager – Long life • Node Manager – One per data server – Monitors resources on node • Application Master – One per application – Short life – Manages task / scheduling
8. Yarn Example
9. Yarn Example • 1) Client -> Resource Manager – Submit App Master • 2) Resource Manager -> Node Manager – Start App Master • 3) Application Master -> Resource Manager – Request and release containers • 4) Resource Manager -> Node Manager – Start tasks in containers
10. Additions • Consider Weave – Simplifies the use of Yarn – Reduced development effort – Simplified API
11. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems