Increasing Engineering Tempo at Splice

Increasing Engineering Tempo at Splice

A video of this talk can be found at https://codeclimate.wistia.com/medias/jcrn6ahlp1

The Splice Engineering team grew almost ten times in eighteen months. The delivery practices that worked for us when we were five broke way before we got to fifty. We wanted our organization to learn faster than the market; to do this, we had to find a way to unlock our delivery. We used insights found on the State of DevOps Report and Accelerate the book. With this knowledge and the help of metrics to visualize our delivery, we were able to get back on a high-performance track.

This talk tells the story of how we used data to engineer the performance of our engineering organization and gives you practical takeaways on how you can do it too.

D7f0df31a2e02ffbb0e6a77b8099035c?s=128

Juan Pablo Buriticá

May 22, 2019
Tweet

Transcript

  1. Juan Pablo Buritica - @buritica
 speakerdeck.com/buritica
 Increasing Engineering Tempo at

  2. None
  3. sounds studio plugins

  4. sounds studio plugins

  5. sounds studio plugins

  6. sounds studio plugins

  7. Empower musicians to realize their creative potential and unleash it

    on the world.
  8. @buritica

  9. @buritica

  10. @buritica I've had to solve two difficult problems in the

    pursuit of this mission
  11. @buritica 1. grow the team to adapt to a new

    strategy
  12. Remember how we said we weren't planning on growing that

    much? yup... Change of plans, what do you need to build W, X, Y, and Z? more than a few engineers Cool, here's some $$$
  13. @buritica ¡ay dios mio!

  14. @buritica and so we grew...

  15. @buritica Splice Engineering Team Size 2014 2015 2016 2017 2018

    2019 20 40 60
  16. @buritica Splice Engineering Team Size 2014 2015 2016 2017 2018

    2019 20 40 60
  17. @buritica

  18. @buritica

  19. @buritica 2. fix the mess I caused by growing the

    team that fast
  20. @buritica Splice Engineering's Mission: We enable Splice to learn faster

    than the market by delivering production-ready software at an accelerating tempo.
  21. @buritica we wanted to learn faster than the market but

    everything felt slow
  22. @buritica it was only a feeling

  23. Things feel slow, don't they? yeah... What are we going

    to do about it? I'm not sure yet, but we'll come up with a plan to fix it ok
  24. ¡ay dios mio!

  25. @buritica what felt slow?

  26. @buritica moving completed engineering work out to customers felt like

    an uphill battle
  27. @buritica "staging is blocked"

  28. @buritica "can you merge my PR?"

  29. @buritica "how long has this been broken for?"

  30. @buritica team growth had taken the majority of our attention,

    and we let our delivery process stagnate
  31. @buritica given our org size, how we managed change was

    as important as the change desired
  32. @buritica we needed a strategy

  33. @buritica plan for the plan
 define metrics set targets measure

    make changes measure again
  34. @buritica we researched

  35. @buritica

  36. @buritica possible metrics Time to Implementation Time to Merge Time

    to Deploy Deploy Confidence Deploy Frequency Support Time / Frequency System Stability Product <> Eng Partnership? Code review and testing? Deployment pipelines state? Test coverage? Deployment pipelines state? Quality? Quality?
  37. @buritica delivery bugs

  38. @buritica delivery bugs opportunities

  39. @buritica we researched

  40. @buritica we researched we discussed we disagreed we committed

  41. @buritica and we introduced our first proposal to the team

  42. @buritica our vision: 
 A future where shipping working code

    is one of the fastest, safest and, most effective ways to learn and to test new ideas.
  43. @buritica a future where: 
 Every engineer is empowered by

    tools and processes, willing and able to take more significant risks.
  44. @buritica a future where: 
 Engineers are confident that the

    code they deliver won't break Splice and when it does, they can fix it quickly.
  45. @buritica a future where: 
 Everyone feels comfortable and welcome

    contributing code to any part of our codebase, including End to End tests.
  46. @buritica a future where: 
 ⚡We continuously expose all Splice

    employees to new functionality and learnings.
  47. @buritica timeframe: crawl the first month &walk by end of

    Q3 ' run by end of H2
  48. @buritica metrics & targets:

  49. @buritica time to merge 
 By the end of Q3,

    we are merging 100% of Pull Requests within 36 hours.
  50. @buritica deploy frequency 
 By the end of Q3, all

    product teams are deploying at least once a day.
  51. @buritica new e2e test coverage 
 By the end of

    Q3, 100% of Engineers have written an End to End Test in an improved testing environment.
  52. @buritica delivery working group 
 three person infra team a

    backend & a frontend engineer a quality engineer management
  53. @buritica [Read the full RFC here]

  54. @buritica middle of the quarter came along

  55. @buritica mid Q3 report "We have not moved Time to

    Merge in any significant way since Q3 began"
  56. @buritica

  57. @buritica

  58. @buritica mid Q3 report "Daily deploys have gone up significantly

    since the introduction of Frontend Preview Branches"
  59. @buritica

  60. @buritica

  61. @buritica

  62. @buritica mid Q3 report "Some Frontend Engineers have begun to

    contribute to E2E tests, we still have significant ground to cover before end of quarter"
  63. @buritica outlook didn't look that good

  64. i'm v busy, sorry

  65. @buritica our instincts were right, but our story wasn't as

    convincing
  66. @buritica I needed better metrics and measurements

  67. @buritica "I wish someone surveyed thousands of organizations about their

    delivery practices" - @buritica
  68. @buritica

  69. @buritica "I wish someone else would tell me the important

    bits about all this data" - @buritica
  70. @buritica

  71. @buritica

  72. @buritica

  73. @buritica

  74. @buritica Product Design & Development Product Delivery Create new products

    and services that solve customer problems using hypothesis-driven delivery, modern UX, design thinking. Enable fast flow from development to production and reliable releases by standardizing work, and reducing variability and batch sizes. Feature design and implementation may require work that has never been performed before. Integration, test, and deployment must be performed continuously as quickly as possible. Estimates are highly uncertain. Cycle times should be well-known and predictable. Outcomes are highly variable Outcomes should have low variability Novelty implemented Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  75. @buritica Splice Tech:
 
 Product Management Product Design Engineering

  76. @buritica Splice Tech is responsible for Design & Development

  77. @buritica Splice Engineering is solely responsible for Delivery

  78. @buritica Product Design & Development

  79. @buritica first
 commit Product Design & Development

  80. @buritica first
 commit product delivery product design Product Design &

    Development
  81. @buritica first
 commit product delivery product design PR merge Product

    Design & Development
  82. @buritica first
 commit product delivery product design PR merge deployment

    Product Design & Development
  83. @buritica first
 commit product delivery product design PR merge deployment

    release Product Design & Development
  84. @buritica work starts Lead Time vs Cycle Time cycle time

    work requested work finished lead time
  85. @buritica Delivery Performance Metrics: ⌚ Lead Time (commit to prod)

    Deployment Frequency Mean Time to Restore
 Change Failure Rate Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  86. @buritica High Performing Teams: ⌚ Less than one hour Multiple

    deploys per day Less than one hour
 0 - 15% Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  87. @buritica Splice Engineering at the time: ⌚ Between one week

    and one month Once per week to several times a month Between one hour to one day
 ??? * Anecdotal experience
  88. @buritica What about points & velocity?

  89. @buritica Velocity is designed to be used as a capacity

    planning tool.
  90. @buritica Velocity is designed to be used as a capacity

    planning tool. Using it as a productivity metric has several flaws.
  91. @buritica Velocity is relative and team dependent.

  92. @buritica Velocity is relative and team dependent. When used as

    productivity metric, teams game it, which impacts collaboration.
  93. @buritica Velocity is relative and team dependent. When used as

    productivity metric, teams game it, which impacts collaboration. Using capacity to measure productivity leads to high utilization, which reduces ability to take unplanned work.
  94. @buritica “Queue theory in math tells us that as utilization

    approaches 100%, lead times approach infinity—in other words, once you get to very high levels of utilization, it takes teams exponentially longer to get anything done. ” Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  95. @buritica velocity is good for planning work and bad for

    measuring teams
  96. @buritica hey boss, check this out

  97. @buritica Delivery Matters Engineering performance affects an organization's ability to

    achieve goals beyond profit and revenue.
  98. @buritica Delivery Matters Whatever the mission, engineering performance can predict

    overall organizational performance Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  99. @buritica How does Splice Engineering become high- performing?

  100. @buritica By driving the team towards a performance oriented culture

  101. @buritica CONTINUOUS DELIVERY PERFORMANCE ORIENTED CULTURE LEAN PRODUCT MANAGEMENT DELIVERY

    PERFORMANCE ORGANIZATIONAL PERFORMANCE Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.” culture drivers culture outcomes
  102. @buritica To achieve this goal: We relentlessly pursue continuous improvement

    to drive the Tech Team's culture towards high-performance by using a capabilities model proven to predict high performance in organizations.
  103. @buritica Capability Categories Continuous Delivery Architecture Product & Process Lean

    management & monitoring Cultural Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
  104. @buritica Continuous Delivery Architecture Product & Process Lean Mgt &

    Monitoring Culture Version control for all artifacts Loosely coupled architecture Customer feedback implementation Lightweight change approval Support a generative culture Automated deployment Architect for empowered teams Visibility into flow of work through value stream Infrastructure monitoring Encourage and support learning Continuous integration Small batched work Proactive system health checks Support and facilitate cross-team collaboration Trunk-based development Enable experimentation Work In Progress limits Provide resources and tools that make work meaningful Test automation Visualize work to monitor quality Transformational leadership Test data management Integrated security Continuous delivery Capabilities
  105. @buritica Continuous Delivery Architecture Product & Process Lean Mgt &

    Monitoring Culture Version control for all artifacts Loosely coupled architecture Customer feedback implementation Lightweight change approval Support a generative culture Automated deployment Architect for empowered teams Visibility into flow of work through value stream Infrastructure monitoring Encourage and support learning Continuous integration Small batched work Proactive system health checks Support and facilitate cross-team collaboration Trunk-based development Enable experimentation Work In Progress limits Provide resources and tools that make work meaningful Test automation Visualize work to monitor quality Transformational leadership Test data management Integrated security Continuous delivery Capabilities
  106. @buritica "I wish someone made a tool that helped me

    measure my team easily" - @buritica
  107. @buritica disclaimer: this is not a sponsored talk

  108. @buritica

  109. @buritica

  110. @buritica our working group has been successful, we'd like to

    invest more sounds good Gorsuch, you got this reeeeleaaase the
  111. @buritica Production Engineering Tooling, services, and expertise that enables teams

    to deliver and operate high quality, production-level software and services.
 
 Security Engineering
 Site Reliability Engineering
 Quality Reliability Engineering
 Developer Experience (DevX)
  112. @buritica Production Engineering Primary Metric: Cycle Time (commit to production)

    Secondary Metrics: 
 Mean Time to Restore, Deploy Frequency, Time to First Commit
  113. @buritica where are we now?

  114. None
  115. wtf cycle time

  116. cycle time!

  117. an accelerating organization

  118. 118 a steady rhythm

  119. 119 a performance oriented culture We have created a culture

    of continuous systems and process improvement. We also have a language and a framework to measure this change.
  120. @buritica what changes caused most impact?

  121. @buritica Notable Changes • Quality Expansion & Acceleration • Focused

    Quality Engs • Transferred Quality Ownership to Eng • New & Improved Testing Infra & Tools • Decentralized Staging Environment • Sub 2 min API Deploys • Stabilized Search Infrastructure • Decoupled Desktop UI • Feature Flags
  122. @buritica what I learned

  123. @buritica success: trust & change management

  124. @buritica we can take a scientific approach as we build

    our teams
  125. @buritica I can use cycle time to strategize about investments

  126. my team is

  127. what's next?

  128. Thank you @buritica