The DevOps Awakens: Moving to a Multi-Cloud Architecture

https://www.papercall.io/talks/32136

Transcript

1. The DevOps Awakens: Moving to a Multi-Cloud Architecture jonstacks @stacks_overflow
3. Organizers
4. What is
5. The Problem
6. The Solution
7. Collaborative Cash Flow Optimization
8. The Roles We Play Provide innovative, reliable, easy-to-use applications that help to liberate working capital for the benefit of both the buyers and suppliers using C2FO.
10. Our Journey to +
11. Q4 2016 Build Registry Deploy 1 Build 2 Push Images 4 Stop/Start Blue 5 Start/Stop Green 3 Pull Images
12. Q4 2016 Full Stack Staging User Acceptance Testing Production
13. Q1 2017
15. New Requirements • Multi-Cloud • Scale Globally in the Future • Go Live in 5-6 months • “Keep the Lights On”
17. More New Requirements • ... • Deploy 3 new applications
18. Q1 2017 - Fleet to be deprecated... Fleet
20. Q1 2017 - Decisions... Fleet Kubernetes
22. Q1 2017 • Started getting infrastructure stood up on GCP • Began deploying some VMs and running tests • Stood up our first K8s cluster ◦ Deployed some simple apps ◦ See what we could do to break it
23. Q2 2017 • Automated deployments to kubernetes using helm. • Got another DevOps Engineer! ◦ Instrumentation! ◦ Monitoring! • Integration with our current systems
24. Q3 2017 • Staging & UAT environment up and running • Deadline push ◦ Solidify process ◦ Lots of load testing ◦ Few bug fixes • We got prod up!
25. Q3 2017 Average time to spin up a new environment: ~ 10 mins Environment Deployments Pods FSS 47 111 UAT 47 110 Production 47 110 Cluster Pods Nodes Pods/Node Testing 19 4 4.75 Non-Prod 273 14 19.5 Production 152 10 15.2
26. Why?
27. Open Source Software +
28. Open Source Software • Kuberntes ◦ Reduced our Infrastructure footprint ◦ Lots of features ▪ Rolling Deploys ▪ Healthchecks ▪ Namespaced DNS ▪ Service Discovery ◦ No pets or snowflakes. ◦ Declarative syntax • Helm ◦ Release management ◦ Version our Kubernetes deployments ◦ Specify our application dependencies
29. Processes • We value automated, repeatable deployments. • Anyone can deploy a development environment. • Working on more blameless post mortems. • Processes are very flexible to change. • Quick all-hands meetings
30. Culture • We hire really good people • No “sacred cows” • We trust our employees ◦ We have the freedom to innovate and make changes ◦ We are allowed to take calculated risks
31. Lessons Learned
32. We could have collaborated more often & tried to deploy DEV OPS FEB. AUG. GCP, K8S POC, Deployment Integration App Architecture & Code Changes X
33. Not all software is created Equal • Some software took more effort to containerize properly for kubernetes. • Best results from apps that follow the 12-factor pattern. • Frustrations with Consul ◦ Doesn’t handle changes in IP very well.
35. !=
36. Implementing Devops processes is hard • Hindsight is 20/20 • Mastery takes time & effort • Well worth it
37. Thank you!
38. is hiring!
39. References 1. By GitHub - https://github.com/github/octicons, MIT, https://commons.wikimedia.org/w/index.php?curid=33440579 2. https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior 3. https://brorlandi.github.io/StarWarsIntroCreator/ 4. https://12factor.net/ 5. https://github.com/kubernetes/charts/issues/1892 6. https://github.com/hashicorp/consul/issues/1580