Learning From Two Years Of Kubernetes In Production

Transcript

1. Learning From 2 Years Of Kubernetes In Production
2. Learning From 2 3 Years Of Kubernetes In Production
3. Tech Consultant, Fractional CTO/VPE Past Life: • 6.5 Years at BlinkIt (formerly Grofers) • 2 Years at Wingify (VWO.com) I’m Vaidik
5. You build it, you run it
6. Then came Kubernetes
7. State of Kubernetes at Grofers - 2022 PRODUCTION • 90% of targeted workload migrated to Kubernetes • Stateful services and 10% of targeted workload will not be migrated DEVELOPMENT • Development in the cloud • Kubernetes as extension of laptop for development • On-demand dev environments under 10 minutes
8. For Developers TOOLING • Ansible based tooling for stateful workloads in production • Kubernetes for everything else in production and non-production YOU BUILD IT, YOU RUN IT • Developers get better abstractions that are provided by platform teams • Platform teams work towards reducing ops overhead at scale. Things like cost, security, reliability become “lesser” of a concern for developers. • More ownership
9. 2 years TIME TAKEN What it took us to get here 9 engineers INFRASTRUCTURE TEAM $500K AWS SPEND ~45 dev months PRODUCT ENGINEERING TEAMS
10. Early 2018 Illusion of Agility
11. Proliferation of Microservices
12. Distributed Monolith
13. Lesson: A DevOps strategy requires good to ensure accountability so that teams can truly benefit from autonomy.
14. Developers were... • Unhappy about the drag caused by poor development experience • Releasing software that was not tested enough • Fire fighting because of high number of bugs and incidents in production • Stressed because of sleepless nights and busy weekends
15. March 2018 Project ShipIt
16. Journey of Project ShipIt FEB 2018 Started with Docker and docker-compose JUN 2018 Still optimizing docker-compose and our orchestration tooling AUG 2018 Frustrated with dev/prod disparity. SEP 2018 Kubernetes decided as the future. JAN 2019 Kubernetes in production with a few services to prove scale MAR 2018 Ran first set of end-to-end tests on a fully orchestrated container based environment MAY 2019 We were running tests
17. What are your reasons? • More agility - consistent developer and DevOps experience, internal DevOps platforms, CI/CD, abstractions for declarative infrastructure management • Better reliability - better scalability, features for resilience, primitives for streamlined operations. • Cost - reduced hosting cost and support cost, but most likely a high migration cost. • Portability - deploy across varied environments (multi cloud, hybrid, etc.).
18. Is Kubernetes the only way to achieve all of this?
19. Building more incrementally vs rebuilding everything
20. Was migrating to Kubernetes worth the time and money we spent?
21. Was migrating to Kubernetes worth the time and money we spent? Was the process of adopting Kubernetes right for us?
22. An Alternate Way SHORT TERM TRACK • Keep using EC2 VMs, Ansible and Consul • Strengthen service discovery with Consul • Speed up Ansible further • Spin up new EC2 VMs LONG TERM TRACK • Build a solid Kubernetes platform for organic adoption • Migrate complex infrastructure first to streamline operation • It’s fine to take more time
23. Do you really need Kubernetes and if yes, do you need it now?
24. Kubernetes is a new paradigm. Lift-and-shift is not a good idea.
25. There is a steep learning curve for your development and operations teams.
26. Out-of-the-box Kubernetes is almost never enough for anyone
27. It is not a PaaS solution but a building block to build your own PaaS solution
28. Your Kubernetes platform will shape according to your context REFLECTION OF YOUR: • Product & Business Context • Engineering Team • Existing applications and their infrastructure • Ability to spend money to migrate to a new way of working
29. PaaS with Kubernetes • Infrastructure • Developer Experience • Configuration & Secret Management • Local Development • Packaging, Deployment • Continuous Integration • Continuous Delivery (+ GitOps) • Metrics • Logs • Distributed Tracing • Governance • Cost • Learning & Development
30. PaaS with Kubernetes • Kops, migrated to EKS • Built in-house on OSS, tutorials • Consul, Vault, consul-template • Skaffold, Telepresence.io • Skaffold, Kustomize, Helm • Tekton • Tekton • Prometheus, Grafana, NewRelic • Loki, Grafana • Not even solved yet • OPA and Kyverno • Self-managed spot instances • Katacoda, Internally built • Infrastructure • Developer Experience • Configuration & Secret Management • Local Development • Packaging, Deployment • Continuous Integration • Continuous Delivery (+ GitOps) • Metrics • Logs • Distributed Tracing • Governance • Cost • Training
31. Lesson: You are limited by your legacy applications.
32. What about adoption and migration?
33. Not enough incentive to migrate and take the risk
34. Lesson: Better platform is the best incentive to migrate
35. Lack of proactive migration support made the journey more painful
36. Lesson: Running parallel stacks is hard
37. Lesson: Operating a Kubernetes cluster is hard.
38. CRDs, Operators, Controllers, Mutating Webhooks
39. Reminder - do you need to do this now?
40. But if you must…
41. But if you must… DON’T do it yourself
42. ̄Awesome solutions available today (not exhaustive) • EKS, AKS, GKE, Civo • Devtron, Porter, Shipa • Hashicorp Vault, Doppler • Devtron, Signodot, Ambassador • Devtron, Gitlab, Harness, Circle CI • Devtron, Harness, Circle CI, LaunchDarkly • Honeycomb, Grafana Cloud,, DataDog • Devtron, Opslevel • Cast.ai, Kubecost, Spot.io • CloudYuga PaaS with Kubernetes Today Big decisions to make • Infrastructure • Developer Experience • Configuration & Secret Management • Environments, Local Development • Continuous Integration • Continuous Delivery (+ GitOps) • Observability (logs, metrics, tracing) • Governance • Cost • Training & Labs
43. Thank You! Vaidik Kapoor Technology Consultant vaidik.in @vaidikkapoor Questions?