MultiStack @ OpenStack Mini-conference - OSIDays

MultiStack presentation at OpenStack mini-conference during OpenSource India days 2013.

Other talks at http://dharmeshkakadia.github.io/talks

Transcript

1. MultiStack Dharmesh Kakadia Shashank Sahni Search and Information Extraction Lab IIIT-Hyderabad
2. $whoarewe • Research Students @ IIIT-H with Prof. Vasudeva Varma • Cloud research group - Scheduling, MultiCloud, SDN • Manage OpenStack based small private cloud and data infrastructure of SIEL • Available for hire :)
3. About SIEL • Deal with lot of Data and wide range of processing frameworks • Early adopters (and contributors) of cloud and big data • FOSS Lovers • Academic environment (limited resources, unlimited scope)
4. Our Story Hadoop Cluster • Unmanaged • Underutilized
5. Our Story • Managed Infrastructure • Quick Cloud Adoption
6. But we still need Hadoop • Shell Scripts • worked fine • for short time !! • Chef-cookbooks • Need higher abstraction
7. HadoopStack • Abstraction of a Hadoop job • It worked great !! Everyone liked it !! • They wanted to run more frameworks • We ran out of capacity on our OpenStack cloud • Need more Hardware ? • Need more stacks ?
8. Going MultiCloud… Public clouds are the new Hardware
9. MultiStack
10. Multi Stack Data Infrastructure
11. Multi Stack Data! Infrastructure
12. Multi Stack Data! Infrastructure
13. Multi Stack Data Infrastructure
14. Multi Stack Data Infrastructure
15. Demo {Start Praying}
16. Features • Support for multiple clouds and frameworks • Auto-scaling • Smart scheduling - deadline, cost • Job workflows • Auto-selection of frameworks
18. API Server - Flask
19. Scheduling - Quota based API Server - Flask
20. Provisioning - Chef Scheduling - Quota based API Server - Flask
21. Provisioning - Chef Scheduling - Quota based API Server - Flask Database - MongoDB
22. Challenge Accepted • Hiding difference between clouds and leveraging cloud-specific features • Run mysecurejob on-premise, batchjob on public cloud • Spot Instance on EC2 • Quota on OpenStack • Everything Dynamic and Automated • Opportunities to improve • Better caching - Tachyon • Optimized configuration
23. Currently working on • Better UI (no more curls) • Better Documentation (to get started quickly) • Monitoring + Logging (App + Cloud) • Test-cases • Production-ready • Logo :)
24. Next is what? • More Clouds (RackSpace, CloudStack, Azure) • More Frameworks (GraphLab) • Vertical frameworks (Hive, Shark, Mahout, MLBase, …)
25. Lessons • Need “One BigData Cloud”. • Give users what they want(which they don’t know yet…)
26. Get in touch ! MultiStack.org @dharmeshkakadia @shredder12