DeathStar: Easy, Dynamic, Multi-tenant HBase via YARN

Multi-tenancy is a hard problem, made harder in distributed systems like HBase. In this talk we describe DeathStar, a solution built at Rocket Fuel on top of Apache Slider and YARN to make HBase multi-tenancy easy. We also discuss the understanding we have developed about various HBase access patterns and multi-tenancy scenario. We cover how we use a single YARN cluster to host multiple smaller and highly customized HBase clusters and how we leverage dynamic provisioning and elastic scaling made possible in this model. Finally, we describe some of the the improvements we have made to Slider and HBase to improve availability, monitorability, automation and usability.

Transcript

1. DeathStar Easy, Dynamic, Multi-tenant HBase via YARN Ishan Chhabra, Nitin Aggarwal Rocketfuel Inc.
2. In a not so distant past…
3. 1000 node cluster
4. Rogue Applications
5. Cannot customize per application
6. Hard to capacity plan or support new applications
7. The Common Solution: Separate Clusters
8. Non uniform network usage Different DFSs’, leading to lot of copying of data Low cluster utilization High lead time for new applications
9. Run HBase on YARN Built on top of Slider Solution: DeathStar
10. Hangar App Cluster 3 Provisioning Model App Cluster 2 App Cluster 1
11. (grid/deathstar): $ git commit Capacity planning and configuration discussion Create simple JSON config As applications mature from hangar to their cluster
12. Dynamic Cluster: Make API call to start, stop and scale cluster Static Cluster: Good to go
14. Strict Isolation
15. Common HDFS Layer Bulkload MapReduce over snapshots
16. Dynamic config and cluster size changes
17. Fits into organization’s capacity planning model
18. Clusters out of thin air
19. Hot swap a new cluster (human error / corruption) Easier HBase version upgrades and testing Temporary scale up for backfill “Dynamic” enables interesting use cases
20. Key Challenges and Solutions
21. Another failure mode Taken care of by auto restarts RM HA in the works
22. Early Days: Bugs Slider did not acknowledge container allocations correctly Fixed scheduled for 0.8: SLIDER-828
23. Not easily reproducible, still debugging Zombie Regionservers Early Days: Bugs
24. Long running apps a secondary use case Logging, an unsolved problem Store logs on local disks, considering ELK
25. YARN/Slider lack certain scheduling constraints At most x instances per node for spread and availability Custom patch in-house
26. Rolling restarts for config changes Unsolved. SLIDER-226
27. Data Locality Locality Anti-Locality
28. Metrics Reporting Custom hadoop metrics OpenTSDB reporter App name passed via config
29. Conclusion: Is it for me?
30. Conclusion: Is it worth it?
31. Thank you! Questions? Reach us at: ishan@rocketfuel.com naggarwal@rocketfuel.com