Using OpenEBS and MayaData technologies for WFH and cost savings

A new normal is being set for cloud native developers and SREs where working from home is the default mode and saving $$$ is the untold mandate. OpenEBS is used to setup EBS equivalent persistent storage at home office and it's thin provisioning capabilities save lots of $$$ on cloud.

Transcript

1. Using OpenEBS successfully for remote work and cost reduction @uma_mukkara Co-Founder & COO MayaData 1st April, 2020 Virtual Event
2. Introduction • 6th largest “Independent” is not a company • #1 private company • DevStats itself runs on OpenEBS @uma_mukkara Co-Founder & COO MayaData Co-Creator of the OSS projects Cloud Native Data Management
3. Introduction to MayaData Using Kubernetes itself to deliver data management and storage to Kubernetes SREs. The data agility company.
4. Agenda • The new normal for the cloud native world • Remote work scenarios and associated costs • Use cases of remote work and cost reduction • Getting started
5. The new normal for cloud native world Work at office (90%) WFH (10%) New normal WFH (90%) WAO (10%) End of the year WFH (50%) WAO (50%) Home office
6. Dev environments - impact on costs Data Center + Cloud Home Office (10%) Default choice Cloud Home Office (10%) Cloud Spend Remember - Cloud environments are usually sticky
7. Dev environments - impact on costs Data Center + Cloud Home Office (10%) Default choice Cloud Home Office (10%) Cloud Spend If you continue on cloud (by end of the year) Cloud Spend Cloud Home Office (10%)
8. New challenges • I wanted to be in hybrid-cloud environments to avoid cloud stickiness, but now I seem to have no choice • We managed to bring down the costs through hybrid cloud strategy, but we are now reversing it. Cloud is the only choice • Apart from the spike in costs, I am afraid that cloud stickiness will come back in. Will I be able to move to OnPrem easily once I am out of this tragedy?
9. Possible solutions • Pick a solution where cloud lockin issue is resolved • You can install Kubernetes easily at your home office. Now pick a persistent data solution equivalent to EBS or GPD at your home office. • Deploy your stateful apps, CI/CD pipelines and DevOps environments on low cost architectures on clouds. • Find out free solutions to meet your cloud native data management needs.
10. Introducing OpenEBS + Director • With OpenEBS and Director - You can ◦ Easily deploy EBS or GPD equivalent at your home office ◦ Reduce your cloud costs by at least 50% in majority of the scenarios ◦ Mimic the hybrid cloud strategy at your home office ◦ Get free tools for cloud native data management and monitoring ◦ Stay away from the worry of cloud-lockin
11. Use cases of OpenEBS and Director 1. Setup EBS/GPD equivalent persistent storage at your home office a. Work from home + Up to 90% reduction in costs; works only for dev environment and non-critical apps 2. Build RDS equivalent cloud native databases on Cloud at 50% cost 3. Build scalable CI/CD persistent storage for CI pipelines at upto 90% cost reduction 4. RWX - NFS storage on cloud or home office at 40% or most savings
12. The problem with Stateful microservice microservice microservice microservice microservice microservice
13. microservice microservice microservice microservice microservice microservice Kubernetes + OpenEBS
14. Introducing OpenEBS
15. OpenEBS • GitHub: https://github.com/openebs/openebs • Website: https://openebs.io/ • Slack: https://slack.openebs.io • Twitter: https://twitter.com/openebs • Overall 350+ Code contributors • 2000+ Slack Members, 600+ Forks, 6000+ stars • 1.0 released in June 2019. In production in 100s of enterprises. • Deployed in 1000s of clusters every week.
16. OpenEBS is CAS • CAS ◦ Container Attached Storage ◦ https://bit.ly/cncf-cas ◦ OpenEBS the most popular open source “CAS” ◦ Uses Kubernetes itself as a substrate for data management ◦ Extremely appealing to engineers and teams that embrace DevOps and Kubernetes
17. WFH at MayaData: • As a remote first company, we adopted WFH early on, and it is helping now. • We have adopted the use cases of WFH, CI and RDS in our Dev and Operating environments. Moved to LocalPV for some databases. – We saved lots of $$$ – Helped community users and other enterprises to do the same
18. WFH $$$ Use case #1 Persistent storage at your home office
19. Persistent storage at your home office Physical Server(s) + Virtualization stack (VMware ESXi) VM (virtual disks) VM (virtual disks) VM (virtual disks) VMs OpenEBS cStor Pools EBS/GPD equivalent layer Stateful Application Pods • Get EBS/GPD equivalent featured persistent storage at your home office • Snapshots, clones, thin provisioning, resize, CSI compliant • Develop at home move to cloud or OnPrem with higher performing disks using DMaaS • Application can be be migrated with data Features • 100% productive while working from home • Cost is just the physical servers.. rest is free • Reduce the costs by up to 90% compared to cloud costs Benefits
20. Move applications using DMaaS • Move your application along with the configuration to any other Kubernetes cluster • Work on your application on your laptop or home office environment and then move it to another cluster either on Cloud or OnPrem • Enables work from home • Saves $$$ Benefits
21. DB @ 50% Use case #2 Building RDS equivalent Databases on Cloud platforms at 50% TCO
22. AWS RDS vs OpenEBS Open Database Typical setup with RDS and applications on EKS Application Pods EBS RDS IP
23. RDS typical cost for calculations RDS Cost $5744 per month on AWS RDS Profile
24. RDS equivalent with OpenEBS OpenEBS Storage Pools EBS/GPD equivalent layer Database Pods Database Cost of 3 nodes with 24 core / 64GB ram per month = $2100 EBS cost = $600 OpenEBS license = $300 Total = $3000/month vs $5744/month Additional cost savings: - For large capacities, using thin provisioning will avoid upfront provisioning and saves $$$ - The compute for DB is shared among other application PODs. TCO comes down further.
25. CI storage @ 50% Use case #3 Building a persistent storage solution for CI pipelines at 50% TCO
26. Persistent volumes on a CI pipelines in Cloud • Hundreds of small volumes in each pipeline of certain capacity • Cloud vendor charges for provisioned storage • When pipelines fail, if the volumes are not cleaned up, the cost goes up and for a long while Challenges No of pods = 200 No of PVs = 500 No of EBS volumes = 500 Provisioned capacity = 500*10GB = 5TB for 2 hours Used capacity = 20% of provisioned = 1TB
27. Persistent volumes on a CI pipelines in Cloud • Have the volumes from OpenEBS storage pools with thin provision • Moderate performance is acceptable for CI pipelines, use EBS underneath with only 20% of the capacity • OpenEBS supports snaps and clones. Pipelines are more efficient now Solution No of pods = 200 No of PVs = 500 No of OpenEBS volumes = 500 Thin provisioned capacity = 500*10GB = 5TB for 2 hours EBS provisioned capacity = 20% of OpenEBS provisioned = 1TB 80% cost savings + increased efficiency
28. NFS storage @ 40% Use case #4 NFS/RWX at 40-50% cost savings
29. RWX/NFS cost savings • Hundreds of NFS shares for long standing data with millions of files many a times. Assume that the data keeps growing at 5% every week. • EBS volumes are typically provisioned to operate at around 60% max capacity • Cloud providers charge on provisioned capacity. Used capacity is averages between 50 to 60% on a well managed systems. It could be less than that. Challenges
30. RWX/NFS cost savings • Thin provision the NFS volume for 1000% on day-1. Automatically Add EBS volumes when used capacity reaches 90%. • Average used capacity of EBS/GPD volumes will e 80% in this case which is up by about 40%, resulting in savings of 40%. Solution 40% to 50% cost savings on EBS/GPD volumes
31. Thin provisioning Case Study Thin provisioning savings - Example analysis
32. Cost analysis Google Cloud Storage GCE (Worker nodes) GCP PD (Provisioned) GKE (Developer clusters) 30% 25% 30% 10% 5% Others $78,245 / month $23,473 $19,561 $23,473 $7,824 $3,912 https://cloud.google.com/products/calculator • Developer clusters • SREs / Build teams • Solution team workload tests • E2E clusters • SaaS product
33. Reduce storage over-provisioning Google Cloud Storage GCE (Worker nodes) GCP PD (Provisioned) GKE (Developer clusters) Others $78,245 / month $23,473 $19,561 $23,473 $7,824 $3,912 https://cloud.google.com/products/calculator Google Cloud Storage GCP PD (Thin-Provisioned) Others $4,694 $782 $3,912 %90 reduction %80 reduction
34. Reduce compute using LocalPV Google Cloud Storage GCE (Worker nodes) GCP PD (Provisioned) GKE (Developer clusters) Others $78,245 / month $23,473 $19,561 $23,473 $7,824 $3,912 https://cloud.google.com/products/calculator Google Cloud Storage GCE (Worker nodes) GCP PD (Thin-Provisioned) GKE (Developer clusters) Others $25,036 / month $5,868 $9,780 $4,694 $782 $3,912 %90 reduction %80 reduction %50 reduction %75 reduction %68 Total Reduction $53K/month saved
35. Getting started • Sign up at mayadata.io to get free tools to monitor and manage your Kubernetes instances – Instant Prometheus for metrics – Instant ElasticSearch for logs – Instant Scope for topology views • Install OpenEBS • Setup OpenEBS • Manage your data • Move your applications across clusters • https://help.mayadata.io
36. First steps Sign up at https://mayadata.io Access Director self service portal Or Install Director on your own K8S Connect your K8S clusters Install OpenEBS Provision EBS style storage
37. Sign-Up to MayaData
38. Thank you