An Introduction to Apache Hama

An Introduction to Apache Hama, what is it ?
Why is it useful for scientific computations
on big data ?

Transcript

1. Apache Hama • What is it ? • What is it used for ? • How does it work ? • Architecture www.semtech-solutions.co.nz info@semtech-solutions.co.nz
2. Hama – What is it ? • A Distributed computing framework • For massive scientific calculations • For distributed computing • Uses Bulk Synchronous Parallel Computing ( BSP ) • Written in Java • Licensed via Apache • Available for Hadoop greater than 1.0.x • Requires Java 1.6.x or higher www.semtech-solutions.co.nz info@semtech-solutions.co.nz
3. Hama – What is it used for ? • large scale computations that – exceed the abilities of Map reduce • Machine learning • Matrix • Graph • Network Algorithms www.semtech-solutions.co.nz info@semtech-solutions.co.nz
4. Hama – How does it work ? • A series of super steps • Each of which is – Concurrent computation on local data – Point to point communication – Synchronise all processors • Uses Hadoop RPC for communication www.semtech-solutions.co.nz info@semtech-solutions.co.nz
5. Hama – Architecture Hama consists of BSP Master, GroomServers, ZooKeeper www.semtech-solutions.co.nz info@semtech-solutions.co.nz
6. Hama – Architecture • Role of BSPMaster – Maintain groom server statuses – Controlling cluster super steps – Maintaining job progress information – Scheduling Jobs – Assign tasks to groom servers – Fault control – Provide cluster control to users www.semtech-solutions.co.nz info@semtech-solutions.co.nz
7. Hama – Architecture • Role of GroomServer – Performs BSP tasks – Reports status to master – Runs on a single HDFS data node – Starts with a BSP Peer instance www.semtech-solutions.co.nz info@semtech-solutions.co.nz
8. Hama – Architecture • Role of ZooKeeper – Manages barrier synchronisation of BSP Peers www.semtech-solutions.co.nz info@semtech-solutions.co.nz
9. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems