The Future of Ops - Speaker Deck

Slide 1

Slide 1 text

@tyler_treat Tyler Treat • DevOpsDays Des Moines • 4/13/18 The Future of Ops

Slide 2

Slide 2 text

@tyler_treat Welcome to the  world of tomorrow!

Slide 3

Slide 3 text

@tyler_treat @tyler_treat

Slide 4

Slide 4 text

@tyler_treat Tyler Treat Managing Partner @ Real Kinetic Former infrastructure engineering manager @ Workiva bravenewgeek.com

Slide 5

Slide 5 text

@tyler_treat Data Center

Slide 6

Slide 6 text

@tyler_treat Data Center Compute Network Storage

Slide 7

Slide 7 text

@tyler_treat Data Center Compute Network Storage App Servers Security Backups/DR Monitoring

Slide 8

Slide 8 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring

Slide 9

Slide 9 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring App App App App App App App

Slide 10

Slide 10 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring

Slide 11

Slide 11 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring App App App App App App App Ops

Slide 12

Slide 12 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring App App App App App App App DevOps

Slide 13

Slide 13 text

@tyler_treat App App App App App App App NoOps

Slide 14

Slide 14 text

@tyler_treat App App App App App App App Infrastructure Automation Deployment Automation Conﬁguration Management Log Management Monitoring NewOps

Slide 15

Slide 15 text

@tyler_treat

Slide 16

Slide 16 text

@tyler_treat DevOps is a journey, not a destination.

Slide 17

Slide 17 text

@tyler_treat Manual Provisioning Self-Service The DevOps Scale of Automation

Slide 18

Slide 18 text

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup

Slide 19

Slide 19 text

@tyler_treat Scaling DevOps

Slide 20

Slide 20 text

@tyler_treat @tyler_treat Why do silos form?

Slide 21

Slide 21 text

@tyler_treat Many companies start with a “DevOps” approach.

Slide 22

Slide 22 text

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup DevOps by Necessity  Devs push to production, unstable, high-risk, minimal cost control

Slide 23

Slide 23 text

@tyler_treat As the product scales, we specialize. @tyler_treat

Slide 24

Slide 24 text

@tyler_treat As the business scales,  we add safety checks.

Slide 25

Slide 25 text

@tyler_treat Developers write code.

Slide 26

Slide 26 text

@tyler_treat Ops people run it.

Slide 27

Slide 27 text

@tyler_treat QA gets blamed for defects.

Slide 28

Slide 28 text

@tyler_treat Security blocks everything.

Slide 29

Slide 29 text

@tyler_treat And management wonders why nothing gets shipped.

Slide 30

Slide 30 text

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup Ops as Gatekeepers  Stable, cost-controlled, risk-averse,  delivery and innovation bottleneck 

Slide 31

Slide 31 text

@tyler_treat Specialization is good!

Slide 32

Slide 32 text

@tyler_treat Misalignment is not good.

Slide 33

Slide 33 text

@tyler_treat How do we scale specialization?

Slide 34

Slide 34 text

@tyler_treat Cross-functional teams? @tyler_treat

Slide 35

Slide 35 text

@tyler_treat DevOps encourages cooperation!

Slide 36

Slide 36 text

@tyler_treat Just add an ops engineer to each team.

Slide 37

Slide 37 text

@tyler_treat And maybe a reliability engineer.

Slide 38

Slide 38 text

@tyler_treat Maybe a few extra for on-call backup.

Slide 39

Slide 39 text

@tyler_treat And of course we need a QA engineer too.

Slide 40

Slide 40 text

@tyler_treat Done!

Slide 41

Slide 41 text

@tyler_treat Also, $$$

Slide 42

Slide 42 text

@tyler_treat @tyler_treat

Slide 43

Slide 43 text

@tyler_treat How do we scale specialization?

Slide 44

Slide 44 text

@tyler_treat Vision and Product

Slide 45

Slide 45 text

@tyler_treat Vision: a mental image of what the future could be like. @tyler_treat

Slide 46

Slide 46 text

@tyler_treat Vision enables independent decision making and alignment.

Slide 47

Slide 47 text

@tyler_treat But vision without execution is just hallucination…

Slide 48

Slide 48 text

@tyler_treat Products are how we scale execution.

Slide 49

Slide 49 text

@tyler_treat

Slide 50

Slide 50 text

@tyler_treat The evolution of QA Test-focused Tools-focused

Slide 51

Slide 51 text

@tyler_treat The evolution of QA QA SDET “Combined” Engineering

Slide 52

Slide 52 text

@tyler_treat Production CD Pipeline CI

Slide 53

Slide 53 text

@tyler_treat QA teams are shrinking, but what’s growing are the teams building the tools.

Slide 54

Slide 54 text

@tyler_treat The same is becoming true of Ops.

Slide 55

Slide 55 text

@tyler_treat build/release/deploy conﬁguration management infrastructure automation logging & instrumentation monitoring

Slide 56

Slide 56 text

@tyler_treat By productizing our infrastructure, we scaled.

Slide 57

Slide 57 text

@tyler_treat We controlled costs.

Slide 58

Slide 58 text

@tyler_treat We reduced risk.

Slide 59

Slide 59 text

@tyler_treat We accelerated development.

Slide 60

Slide 60 text

@tyler_treat We delivered value to customers faster…

Slide 61

Slide 61 text

@tyler_treat from 3 - 4 releases per year to multiple releases per day.

Slide 62

Slide 62 text

@tyler_treat Rethinking Ops

Slide 63

Slide 63 text

@tyler_treat

Slide 64

Slide 64 text

@tyler_treat

Slide 65

Slide 65 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring App App App App App App App Wake me up if anything goes wrong here. Ops as Masters of Production

Slide 66

Slide 66 text

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance App Servers Security Backups/DR Monitoring App App App App App App App Jim Bob’s Frobulator service is out of memory. Ops as Masters of Production

Slide 67

Slide 67 text

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup PaaS  Stable, cost-controlled, risk-averse,  delivery enabler, innovation bottleneck

Slide 68

Slide 68 text

@tyler_treat

Slide 69

Slide 69 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services… @tyler_treat The Vision

Slide 70

Slide 70 text

@tyler_treat “Here’s a CloudFormation template and access to production…”

Slide 71

Slide 71 text

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup IaaS  Devs provision infrastructure as code, free-for-all, cost explosion, high-risk,  delivery and innovation enabler

Slide 72

Slide 72 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services… @tyler_treat The Vision

Slide 73

Slide 73 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services…with minimal Ops intervention. @tyler_treat The Vision

Slide 74

Slide 74 text

@tyler_treat

Slide 75

Slide 75 text

@tyler_treat

Slide 76

Slide 76 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services…with minimal Ops intervention. @tyler_treat The Vision

Slide 77

Slide 77 text

@tyler_treat App App App App App App App Infrastructure Automation Deployment Automation Conﬁguration Management Log Management Monitoring Ops as Product Team

Slide 78

Slide 78 text

@tyler_treat App App App App App App App Infrastructure Automation Deployment Automation Conﬁguration Management Log Management Monitoring Products Ops as Product Team

Slide 79

Slide 79 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services…with minimal Ops intervention. @tyler_treat The Vision

Slide 80

Slide 80 text

@tyler_treat Pain-Driven Development: making locally optimal decisions to minimize pain.

Slide 81

Slide 81 text

@tyler_treat Silos promote pain displacement. Product Development QA Ops

Slide 82

Slide 82 text

@tyler_treat Silos promote pain displacement. Product Development QA Ops pain of running software pain of testing software pain of building software

Slide 83

Slide 83 text

@tyler_treat Misaligned incentives!

Slide 84

Slide 84 text

@tyler_treat How do you expect devs to care about quality if they’re not on the hook?

Slide 85

Slide 85 text

@tyler_treat How do you expect devs to care about operability if they’re not on the hook?

Slide 86

Slide 86 text

@tyler_treat Devs won’t build truly reliable systems until they are on-call for them.

Slide 87

Slide 87 text

@tyler_treat BUT!

Slide 88

Slide 88 text

@tyler_treat Responsibility requires empowerment.

Slide 89

Slide 89 text

@tyler_treat You can’t ask someone to care about something and ﬁx it without also giving them the power to do so.

Slide 90

Slide 90 text

@tyler_treat Most Ops teams simply haven’t done enough to empower and offload responsibility onto dev teams.

Slide 91

Slide 91 text

@tyler_treat Products enable ownership.

Slide 92

Slide 92 text

@tyler_treat App App App App App App App Infrastructure Automation Deployment Automation Conﬁguration Management Log Management Monitoring Products Ops as Product Team

Slide 93

Slide 93 text

@tyler_treat App App App App App App App Infrastructure Automation Deployment Automation Conﬁguration Management Log Management Monitoring Products The Frobulator service is out of memory… Since you are the Frobulator expert, here are these tools to help you diagnose and resolve the problem autonomously. Ops as Product Team

Slide 94

Slide 94 text

@tyler_treat Enable developers to self-service through tooling and automation and empower them to deploy and operate their services…with minimal Ops intervention. @tyler_treat The Vision

Slide 95

Slide 95 text

@tyler_treat Products maintain control through enablement.

Slide 96

Slide 96 text

@tyler_treat Enable teams to follow best practices.

Slide 97

Slide 97 text

@tyler_treat Best practices for builds.

Slide 98

Slide 98 text

@tyler_treat Best practices for testing.

Slide 99

Slide 99 text

@tyler_treat Best practices for deploys.

Slide 100

Slide 100 text

@tyler_treat Best practices for support.

Slide 101

Slide 101 text

@tyler_treat Best practices for compliance.

Slide 102

Slide 102 text

@tyler_treat Encode compliance and SDLC requirements into tooling and process.

Slide 103

Slide 103 text

@tyler_treat Snowflakes kill…  Use pain-driven development to your advantage by creating paths of least resistance.

Slide 104

Slide 104 text

@tyler_treat Teams must make a case for going off-menu.

Slide 105

Slide 105 text

@tyler_treat Products in Practice

Slide 106

Slide 106 text

@tyler_treat Build Release Deploy Operate

Slide 107

Slide 107 text

@tyler_treat Build Release Deploy Operate

Slide 108

Slide 108 text

@tyler_treat Code Repository Dev Push change to branch Review by Peers Build QA & Compliance Continuous Integration

Slide 109

Slide 109 text

@tyler_treat @tyler_treat

Slide 110

Slide 110 text

@tyler_treat @tyler_treat

Slide 111

Slide 111 text

@tyler_treat @tyler_treat

Slide 112

Slide 112 text

@tyler_treat

Slide 113

Slide 113 text

@tyler_treat

Slide 114

Slide 114 text

@tyler_treat

Slide 115

Slide 115 text

@tyler_treat @tyler_treat

Slide 116

Slide 116 text

@tyler_treat • Build plan part of the code, not baked into build tool  • Dev teams fully control their builds  • Deep integration with GitHub   • Build controls into the process Continuous Integration

Slide 117

Slide 117 text

@tyler_treat Build Release Deploy Operate

Slide 118

Slide 118 text

@tyler_treat Code Repository Dev Tag branch for release Build/QA Continuous Delivery Dev Artifact Repository Sign-Off Prod Artifact Repository Deploy

Slide 119

Slide 119 text

@tyler_treat • Artifact build/tagging/promotion automation  • Container/machine image auditing  • Machine image and security patch automation  • Streamlining sign-off from different parties Continuous Delivery

Slide 120

Slide 120 text

@tyler_treat Build Release Deploy Operate

Slide 121

Slide 121 text

@tyler_treat

Slide 122

Slide 122 text

@tyler_treat

Slide 123

Slide 123 text

@tyler_treat • Self-service deploys  • Self-service conﬁguration (with guard rails)  • Infrastructure provisioning is automated  • No ticket-driven development Continuous Deployment

Slide 124

Slide 124 text

@tyler_treat Build Release Deploy Operate

Slide 125

Slide 125 text

@tyler_treat • Logging  - Structured logging spec  - Language libs implementing spec  - Log pipeline (i.e. agent, collector, storage, search)  • Telemetry, tracing, health checks, alerting  • Canary deploys, A/B testing, trafﬁc shadowing, etc. Continuous Operations

Slide 126

Slide 126 text

@tyler_treat Many off-the-shelf solutions just need “glued” together.

Slide 127

Slide 127 text

@tyler_treat Most problems are cultural, not technical.

Slide 128

Slide 128 text

@tyler_treat Technology will not ﬁx your broken culture!

Slide 129

Slide 129 text

@tyler_treat Solutions need to ﬁt the company, its culture, and its architecture.

Slide 130

Slide 130 text

@tyler_treat Get the workflow correct, start manual, then automate.

Slide 131

Slide 131 text

@tyler_treat Wrapping Up

Slide 132

Slide 132 text

@tyler_treat Specialization leads to misalignment and broken feedback loops.

Slide 133

Slide 133 text

@tyler_treat But specialization is an important part of scaling a business.

Slide 134

Slide 134 text

@tyler_treat The question is:  how do we specialize?

Slide 135

Slide 135 text

@tyler_treat The traditional Ops model does not scale.

Slide 136

Slide 136 text

@tyler_treat DevOps is about tightening feedback loops and building empathy.

Slide 137

Slide 137 text

@tyler_treat NewOps is about empowering teams and providing autonomy.

Slide 138

Slide 138 text

@tyler_treat It’s not a replacement for DevOps, it’s an evolution of it.

Slide 139

Slide 139 text

@tyler_treat It’s applying a product mindset to the traditional Ops model.

Slide 140

Slide 140 text

@tyler_treat Ops teams should be redeﬁning their vision:

Slide 141

Slide 141 text

@tyler_treat from masters of production to enablers of production.

Slide 142

Slide 142 text

@tyler_treat Ops capabilities should be embedded within dev teams…

Slide 143

Slide 143 text

@tyler_treat but they need to be enabled!

Slide 144

Slide 144 text

@tyler_treat

Slide 145

Slide 145 text

@tyler_treat NewOps treats Ops like a product team whose product is infrastructure.

Slide 146

Slide 146 text

@tyler_treat Creating guard rails, not walls.

Slide 147

Slide 147 text

@tyler_treat Offloading responsibility helps correct and scale feedback loops.

Slide 148

Slide 148 text

@tyler_treat Traditional Ops isn’t going away, it’s just getting a product manager.

Slide 149

Slide 149 text

@tyler_treat Thanks!  bravenewgeek.com realkinetic.com