The DevOps Awakens: Moving to a Multi-Cloud Architecture

Slide 1

Slide 1 text

The DevOps Awakens: Moving to a Multi-Cloud Architecture jonstacks @stacks_overflow

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Organizers

Slide 4

Slide 4 text

What is

Slide 5

Slide 5 text

The Problem

Slide 6

Slide 6 text

The Solution

Slide 7

Slide 7 text

Collaborative Cash Flow Optimization

Slide 8

Slide 8 text

The Roles We Play Provide innovative, reliable, easy-to-use applications that help to liberate working capital for the benefit of both the buyers and suppliers using C2FO.

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Our Journey to +

Slide 11

Slide 11 text

Q4 2016 Build Registry Deploy 1 Build 2 Push Images 4 Stop/Start Blue 5 Start/Stop Green 3 Pull Images

Slide 12

Slide 12 text

Q4 2016 Full Stack Staging User Acceptance Testing Production

Slide 13

Slide 13 text

Q1 2017

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

New Requirements ● Multi-Cloud ● Scale Globally in the Future ● Go Live in 5-6 months ● “Keep the Lights On”

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

More New Requirements ● ... ● Deploy 3 new applications

Slide 18

Slide 18 text

Q1 2017 - Fleet to be deprecated... Fleet

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Q1 2017 - Decisions... Fleet Kubernetes

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Q1 2017 ● Started getting infrastructure stood up on GCP ● Began deploying some VMs and running tests ● Stood up our first K8s cluster ○ Deployed some simple apps ○ See what we could do to break it

Slide 23

Slide 23 text

Q2 2017 ● Automated deployments to kubernetes using helm. ● Got another DevOps Engineer! ○ Instrumentation! ○ Monitoring! ● Integration with our current systems

Slide 24

Slide 24 text

Q3 2017 ● Staging & UAT environment up and running ● Deadline push ○ Solidify process ○ Lots of load testing ○ Few bug fixes ● We got prod up!

Slide 25

Slide 25 text

Q3 2017 Average time to spin up a new environment: ~ 10 mins Environment Deployments Pods FSS 47 111 UAT 47 110 Production 47 110 Cluster Pods Nodes Pods/Node Testing 19 4 4.75 Non-Prod 273 14 19.5 Production 152 10 15.2

Slide 26

Slide 26 text

Why?

Slide 27

Slide 27 text

Open Source Software +

Slide 28

Slide 28 text

Open Source Software ● Kuberntes ○ Reduced our Infrastructure footprint ○ Lots of features ■ Rolling Deploys ■ Healthchecks ■ Namespaced DNS ■ Service Discovery ○ No pets or snowflakes. ○ Declarative syntax ● Helm ○ Release management ○ Version our Kubernetes deployments ○ Specify our application dependencies

Slide 29

Slide 29 text

Processes ● We value automated, repeatable deployments. ● Anyone can deploy a development environment. ● Working on more blameless post mortems. ● Processes are very flexible to change. ● Quick all-hands meetings

Slide 30

Slide 30 text

Culture ● We hire really good people ● No “sacred cows” ● We trust our employees ○ We have the freedom to innovate and make changes ○ We are allowed to take calculated risks

Slide 31

Slide 31 text

Lessons Learned

Slide 32

Slide 32 text

We could have collaborated more often & tried to deploy DEV OPS FEB. AUG. GCP, K8S POC, Deployment Integration App Architecture & Code Changes X

Slide 33

Slide 33 text

Not all software is created Equal ● Some software took more effort to containerize properly for kubernetes. ● Best results from apps that follow the 12-factor pattern. ● Frustrations with Consul ○ Doesn’t handle changes in IP very well.

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Implementing Devops processes is hard ● Hindsight is 20/20 ● Mastery takes time & effort ● Well worth it

Slide 37

Slide 37 text

Thank you!

Slide 38

Slide 38 text

is hiring!

Slide 39

Slide 39 text

References 1. By GitHub - https://github.com/github/octicons, MIT, https://commons.wikimedia.org/w/index.php?curid=33440579 2. https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#node-oom-behavior 3. https://brorlandi.github.io/StarWarsIntroCreator/ 4. https://12factor.net/ 5. https://github.com/kubernetes/charts/issues/1892 6. https://github.com/hashicorp/consul/issues/1580