HadoopStack: Big Data Processing on Cloud

This poster was presented at Workshop on Understanding Big Data Analytics, ACM India Special Interest Group on Knowledge Discovery and Data Mining (iKDD)

Transcript

1. HadoopStack: Big Data Processing on Cloud
   Dharmesh Kakadia Shashank Sahni Vasudeva Varma
   SIEL@IIIT-H
   HadoopStack allows enterprises to process Big Data
   across multiple clouds seamlessly. It reduces job
   completion time and improves resource utilization
   using machine learning based job scheduling.
   Use Cases
   •  Hassle free Big Data processing on cloud
   •  Single platform for all Big Data processing needs
   •  Complicated workflow pipelines
   •  Leveraging both Public and Private Infrastructure
   •  Managed deployment of multiple clusters on
   cloud
   Components
   •  Provisioning – Spawning Hadoop clusters on
   demand.
   •  Job Scheduling
   •  Deadline aware
   •  Cost aware
   •  Monitoring – Leveraging usage metrics for auto
   scaling and scheduling
   •  Job Management – Single pane of Glass management
   for all job spread across multiple clouds
   RoadMap
   •  Integrate other Hadoop Ecosystem projects –
   mahout, hive, pig etc.
   •  Integration of other data processing frameworks –
   spark, graphlab and R
   Motivation
   •  High Cost of setting up Big Data clusters
   •  Inability to grow and shrink
   •  Multiple clouds
   Job  
   Submission  
   •  Mul-ple  client  interfaces  
   •  Web  UI,  Command-­‐line  tools,  API  
   Job  Manager  
   •  Support  for  mul--­‐tenancy  
   •  Support  for  job  workflows  
   Scheduling  
   •  Cost  and  deadline  aware  Scheduling  
   •  Support  for  custom  scheduler  plugins  
   Provisioning  
   •  Spawning  on-­‐demand  clusters  suppor-ng    
   mul-ple  IaaS  
   •  Provider  specific  features  
   Monitoring  
   •  Support  for  mul-ple  monitoring  plaGorms.  
   •  Framework  instrumenta-on  for  fine-­‐grained  
   monitoring  
   Scaling  
   •  Automa-c  and  user-­‐triggered  scaling  
   •  Cost  efficient  via  spot  instances  and  scaling  
   down  
   Repor-ng  
   •  Repor-ng  Job  status,  resource  usage,  cost  
   etc.  
   User Workflow
   Challenges
   •  Deadline aware scheduling
   •  Exploiting IaaS specific features – instance
   grouping in AWS, determining optimal instance
   configuration
   •  Predicting Job completion time
   •  Characterizing Jobs
   •  Tracking spot instance for price/performance
   tradeoff
   siel-iiith/hadoopstack
   dharmesh.kakadia@research.iiit.ac.in
   

   View Slide