Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MultiStack @ OpenStack Mini-conference - OSIDays

MultiStack @ OpenStack Mini-conference - OSIDays

MultiStack presentation at OpenStack mini-conference during OpenSource India days 2013.

Other talks at http://dharmeshkakadia.github.io/talks

dharmeshkakadia

November 11, 2013
Tweet

More Decks by dharmeshkakadia

Other Decks in Technology

Transcript

  1. $whoarewe • Research Students @ IIIT-H with Prof. Vasudeva Varma

    • Cloud research group - Scheduling, MultiCloud, SDN • Manage OpenStack based small private cloud and data infrastructure of SIEL • Available for hire :)
  2. About SIEL • Deal with lot of Data and wide

    range of processing frameworks • Early adopters (and contributors) of cloud and big data • FOSS Lovers • Academic environment (limited resources, unlimited scope)
  3. But we still need Hadoop • Shell Scripts • worked

    fine • for short time !! • Chef-cookbooks • Need higher abstraction
  4. HadoopStack • Abstraction of a Hadoop job • It worked

    great !! Everyone liked it !! • They wanted to run more frameworks • We ran out of capacity on our OpenStack cloud • Need more Hardware ? • Need more stacks ?
  5. Features • Support for multiple clouds and frameworks • Auto-scaling

    • Smart scheduling - deadline, cost • Job workflows • Auto-selection of frameworks
  6. Challenge Accepted • Hiding difference between clouds and leveraging cloud-specific

    features • Run mysecurejob on-premise, batchjob on public cloud • Spot Instance on EC2 • Quota on OpenStack • Everything Dynamic and Automated • Opportunities to improve • Better caching - Tachyon • Optimized configuration
  7. Currently working on • Better UI (no more curls) •

    Better Documentation (to get started quickly) • Monitoring + Logging (App + Cloud) • Test-cases • Production-ready • Logo :)
  8. Next is what? • More Clouds (RackSpace, CloudStack, Azure) •

    More Frameworks (GraphLab) • Vertical frameworks (Hive, Shark, Mahout, MLBase, …)
  9. Lessons • Need “One BigData Cloud”. • Give users what

    they want(which they don’t know yet…)