Increasing Engineering Tempo at Splice

Slide 1

Slide 1 text

Juan Pablo Buritica - @buritica  speakerdeck.com/buritica  Increasing Engineering Tempo at

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

sounds studio plugins

Slide 4

Slide 4 text

sounds studio plugins

Slide 5

Slide 5 text

sounds studio plugins

Slide 6

Slide 6 text

sounds studio plugins

Slide 7

Slide 7 text

Empower musicians to realize their creative potential and unleash it on the world.

Slide 8

Slide 8 text

@buritica

Slide 9

Slide 9 text

@buritica

Slide 10

Slide 10 text

@buritica I've had to solve two difficult problems in the pursuit of this mission

Slide 11

Slide 11 text

@buritica 1. grow the team to adapt to a new strategy

Slide 12

Slide 12 text

Remember how we said we weren't planning on growing that much? yup... Change of plans, what do you need to build W, X, Y, and Z? more than a few engineers Cool, here's some $$$

Slide 13

Slide 13 text

@buritica ¡ay dios mio!

Slide 14

Slide 14 text

@buritica and so we grew...

Slide 15

Slide 15 text

@buritica Splice Engineering Team Size 2014 2015 2016 2017 2018 2019 20 40 60

Slide 16

Slide 16 text

@buritica Splice Engineering Team Size 2014 2015 2016 2017 2018 2019 20 40 60

Slide 17

Slide 17 text

@buritica

Slide 18

Slide 18 text

@buritica

Slide 19

Slide 19 text

@buritica 2. fix the mess I caused by growing the team that fast

Slide 20

Slide 20 text

@buritica Splice Engineering's Mission: We enable Splice to learn faster than the market by delivering production-ready software at an accelerating tempo.

Slide 21

Slide 21 text

@buritica we wanted to learn faster than the market but everything felt slow

Slide 22

Slide 22 text

@buritica it was only a feeling

Slide 23

Slide 23 text

Things feel slow, don't they? yeah... What are we going to do about it? I'm not sure yet, but we'll come up with a plan to fix it ok

Slide 24

Slide 24 text

¡ay dios mio!

Slide 25

Slide 25 text

@buritica what felt slow?

Slide 26

Slide 26 text

@buritica moving completed engineering work out to customers felt like an uphill battle

Slide 27

Slide 27 text

@buritica "staging is blocked"

Slide 28

Slide 28 text

@buritica "can you merge my PR?"

Slide 29

Slide 29 text

@buritica "how long has this been broken for?"

Slide 30

Slide 30 text

@buritica team growth had taken the majority of our attention, and we let our delivery process stagnate

Slide 31

Slide 31 text

@buritica given our org size, how we managed change was as important as the change desired

Slide 32

Slide 32 text

@buritica we needed a strategy

Slide 33

Slide 33 text

@buritica plan for the plan  define metrics set targets measure make changes measure again

Slide 34

Slide 34 text

@buritica we researched

Slide 35

Slide 35 text

@buritica

Slide 36

Slide 36 text

@buritica possible metrics Time to Implementation Time to Merge Time to Deploy Deploy Confidence Deploy Frequency Support Time / Frequency System Stability Product <> Eng Partnership? Code review and testing? Deployment pipelines state? Test coverage? Deployment pipelines state? Quality? Quality?

Slide 37

Slide 37 text

@buritica delivery bugs

Slide 38

Slide 38 text

@buritica delivery bugs opportunities

Slide 39

Slide 39 text

@buritica we researched

Slide 40

Slide 40 text

@buritica we researched we discussed we disagreed we committed

Slide 41

Slide 41 text

@buritica and we introduced our first proposal to the team

Slide 42

Slide 42 text

@buritica our vision:   A future where shipping working code is one of the fastest, safest and, most effective ways to learn and to test new ideas.

Slide 43

Slide 43 text

@buritica a future where:   Every engineer is empowered by tools and processes, willing and able to take more significant risks.

Slide 44

Slide 44 text

@buritica a future where:   Engineers are confident that the code they deliver won't break Splice and when it does, they can fix it quickly.

Slide 45

Slide 45 text

@buritica a future where:   Everyone feels comfortable and welcome contributing code to any part of our codebase, including End to End tests.

Slide 46

Slide 46 text

@buritica a future where:   ⚡We continuously expose all Splice employees to new functionality and learnings.

Slide 47

Slide 47 text

@buritica timeframe: crawl the first month &walk by end of Q3 ' run by end of H2

Slide 48

Slide 48 text

@buritica metrics & targets:

Slide 49

Slide 49 text

@buritica time to merge   By the end of Q3, we are merging 100% of Pull Requests within 36 hours.

Slide 50

Slide 50 text

@buritica deploy frequency   By the end of Q3, all product teams are deploying at least once a day.

Slide 51

Slide 51 text

@buritica new e2e test coverage   By the end of Q3, 100% of Engineers have written an End to End Test in an improved testing environment.

Slide 52

Slide 52 text

@buritica delivery working group   three person infra team a backend & a frontend engineer a quality engineer management

Slide 53

Slide 53 text

@buritica [Read the full RFC here]

Slide 54

Slide 54 text

@buritica middle of the quarter came along

Slide 55

Slide 55 text

@buritica mid Q3 report "We have not moved Time to Merge in any significant way since Q3 began"

Slide 56

Slide 56 text

@buritica

Slide 57

Slide 57 text

@buritica

Slide 58

Slide 58 text

@buritica mid Q3 report "Daily deploys have gone up significantly since the introduction of Frontend Preview Branches"

Slide 59

Slide 59 text

@buritica

Slide 60

Slide 60 text

@buritica

Slide 61

Slide 61 text

@buritica

Slide 62

Slide 62 text

@buritica mid Q3 report "Some Frontend Engineers have begun to contribute to E2E tests, we still have significant ground to cover before end of quarter"

Slide 63

Slide 63 text

@buritica outlook didn't look that good

Slide 64

Slide 64 text

i'm v busy, sorry

Slide 65

Slide 65 text

@buritica our instincts were right, but our story wasn't as convincing

Slide 66

Slide 66 text

@buritica I needed better metrics and measurements

Slide 67

Slide 67 text

@buritica "I wish someone surveyed thousands of organizations about their delivery practices" - @buritica

Slide 68

Slide 68 text

@buritica

Slide 69

Slide 69 text

@buritica "I wish someone else would tell me the important bits about all this data" - @buritica

Slide 70

Slide 70 text

@buritica

Slide 71

Slide 71 text

@buritica

Slide 72

Slide 72 text

@buritica

Slide 73

Slide 73 text

@buritica

Slide 74

Slide 74 text

@buritica Product Design & Development Product Delivery Create new products and services that solve customer problems using hypothesis-driven delivery, modern UX, design thinking. Enable fast flow from development to production and reliable releases by standardizing work, and reducing variability and batch sizes. Feature design and implementation may require work that has never been performed before. Integration, test, and deployment must be performed continuously as quickly as possible. Estimates are highly uncertain. Cycle times should be well-known and predictable. Outcomes are highly variable Outcomes should have low variability Novelty implemented Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 75

Slide 75 text

@buritica Splice Tech:    Product Management Product Design Engineering

Slide 76

Slide 76 text

@buritica Splice Tech is responsible for Design & Development

Slide 77

Slide 77 text

@buritica Splice Engineering is solely responsible for Delivery

Slide 78

Slide 78 text

@buritica Product Design & Development

Slide 79

Slide 79 text

@buritica first  commit Product Design & Development

Slide 80

Slide 80 text

@buritica first  commit product delivery product design Product Design & Development

Slide 81

Slide 81 text

@buritica first  commit product delivery product design PR merge Product Design & Development

Slide 82

Slide 82 text

@buritica first  commit product delivery product design PR merge deployment Product Design & Development

Slide 83

Slide 83 text

@buritica first  commit product delivery product design PR merge deployment release Product Design & Development

Slide 84

Slide 84 text

@buritica work starts Lead Time vs Cycle Time cycle time work requested work finished lead time

Slide 85

Slide 85 text

@buritica Delivery Performance Metrics: ⌚ Lead Time (commit to prod) Deployment Frequency Mean Time to Restore  Change Failure Rate Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 86

Slide 86 text

@buritica High Performing Teams: ⌚ Less than one hour Multiple deploys per day Less than one hour  0 - 15% Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 87

Slide 87 text

@buritica Splice Engineering at the time: ⌚ Between one week and one month Once per week to several times a month Between one hour to one day  ??? * Anecdotal experience

Slide 88

Slide 88 text

@buritica What about points & velocity?

Slide 89

Slide 89 text

@buritica Velocity is designed to be used as a capacity planning tool.

Slide 90

Slide 90 text

@buritica Velocity is designed to be used as a capacity planning tool. Using it as a productivity metric has several flaws.

Slide 91

Slide 91 text

@buritica Velocity is relative and team dependent.

Slide 92

Slide 92 text

@buritica Velocity is relative and team dependent. When used as productivity metric, teams game it, which impacts collaboration.

Slide 93

Slide 93 text

@buritica Velocity is relative and team dependent. When used as productivity metric, teams game it, which impacts collaboration. Using capacity to measure productivity leads to high utilization, which reduces ability to take unplanned work.

Slide 94

Slide 94 text

@buritica “Queue theory in math tells us that as utilization approaches 100%, lead times approach infinity—in other words, once you get to very high levels of utilization, it takes teams exponentially longer to get anything done. ” Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 95

Slide 95 text

@buritica velocity is good for planning work and bad for measuring teams

Slide 96

Slide 96 text

@buritica hey boss, check this out

Slide 97

Slide 97 text

@buritica Delivery Matters Engineering performance affects an organization's ability to achieve goals beyond profit and revenue.

Slide 98

Slide 98 text

@buritica Delivery Matters Whatever the mission, engineering performance can predict overall organizational performance Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 99

Slide 99 text

@buritica How does Splice Engineering become high- performing?

Slide 100

Slide 100 text

@buritica By driving the team towards a performance oriented culture

Slide 101

Slide 101 text

@buritica CONTINUOUS DELIVERY PERFORMANCE ORIENTED CULTURE LEAN PRODUCT MANAGEMENT DELIVERY PERFORMANCE ORGANIZATIONAL PERFORMANCE Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.” culture drivers culture outcomes

Slide 102

Slide 102 text

@buritica To achieve this goal: We relentlessly pursue continuous improvement to drive the Tech Team's culture towards high-performance by using a capabilities model proven to predict high performance in organizations.

Slide 103

Slide 103 text

@buritica Capability Categories Continuous Delivery Architecture Product & Process Lean management & monitoring Cultural Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

Slide 104

Slide 104 text

@buritica Continuous Delivery Architecture Product & Process Lean Mgt & Monitoring Culture Version control for all artifacts Loosely coupled architecture Customer feedback implementation Lightweight change approval Support a generative culture Automated deployment Architect for empowered teams Visibility into flow of work through value stream Infrastructure monitoring Encourage and support learning Continuous integration Small batched work Proactive system health checks Support and facilitate cross-team collaboration Trunk-based development Enable experimentation Work In Progress limits Provide resources and tools that make work meaningful Test automation Visualize work to monitor quality Transformational leadership Test data management Integrated security Continuous delivery Capabilities

Slide 105

Slide 105 text

Slide 106

Slide 106 text

@buritica "I wish someone made a tool that helped me measure my team easily" - @buritica

Slide 107

Slide 107 text

@buritica disclaimer: this is not a sponsored talk

Slide 108

Slide 108 text

@buritica

Slide 109

Slide 109 text

@buritica

Slide 110

Slide 110 text

@buritica our working group has been successful, we'd like to invest more sounds good Gorsuch, you got this reeeeleaaase the

Slide 111

Slide 111 text

@buritica Production Engineering Tooling, services, and expertise that enables teams to deliver and operate high quality, production-level software and services.    Security Engineering  Site Reliability Engineering  Quality Reliability Engineering  Developer Experience (DevX)

Slide 112

Slide 112 text

@buritica Production Engineering Primary Metric: Cycle Time (commit to production) Secondary Metrics:   Mean Time to Restore, Deploy Frequency, Time to First Commit

Slide 113

Slide 113 text

@buritica where are we now?

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

wtf cycle time

Slide 116

Slide 116 text

cycle time!

Slide 117

Slide 117 text

an accelerating organization

Slide 118

Slide 118 text

118 a steady rhythm

Slide 119

Slide 119 text

119 a performance oriented culture We have created a culture of continuous systems and process improvement. We also have a language and a framework to measure this change.

Slide 120

Slide 120 text

@buritica what changes caused most impact?

Slide 121

Slide 121 text

@buritica Notable Changes • Quality Expansion & Acceleration • Focused Quality Engs • Transferred Quality Ownership to Eng • New & Improved Testing Infra & Tools • Decentralized Staging Environment • Sub 2 min API Deploys • Stabilized Search Infrastructure • Decoupled Desktop UI • Feature Flags