Increasing Engineering Tempo at Splice

A video of this talk can be found at https://codeclimate.wistia.com/medias/jcrn6ahlp1

The Splice Engineering team grew almost ten times in eighteen months. The delivery practices that worked for us when we were five broke way before we got to fifty. We wanted our organization to learn faster than the market; to do this, we had to find a way to unlock our delivery. We used insights found on the State of DevOps Report and Accelerate the book. With this knowledge and the help of metrics to visualize our delivery, we were able to get back on a high-performance track.

This talk tells the story of how we used data to engineer the performance of our engineering organization and gives you practical takeaways on how you can do it too.

Transcript

1. Juan Pablo Buritica - @buritica

   speakerdeck.com/buritica

   Increasing
   Engineering
   Tempo at
   

   View Slide

2. View Slide

3. sounds
   studio plugins
   

   View Slide

4. sounds
   studio plugins
   

   View Slide

5. sounds
   studio plugins
   

   View Slide

6. sounds
   studio plugins
   

   View Slide

7. Empower musicians
   to realize their creative potential
   and unleash it on the world.
   

   View Slide

8. @buritica
   

   View Slide

9. @buritica
   

   View Slide

10. @buritica
    I've had to solve two
    difficult problems in
    the pursuit of this
    mission
    

    View Slide

11. @buritica
    1. grow the team to
    adapt to a new
    strategy
    

    View Slide

12. Remember how we said we weren't
    planning on growing that much?
    yup...
    Change of plans, what do you need
    to build W, X, Y, and Z?
    more than a few engineers
    Cool, here's some $$$
    

    View Slide

13. @buritica
    ¡ay dios mio!
    

    View Slide

14. @buritica
    and so we grew...
    

    View Slide

15. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60
    

    View Slide

16. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60
    

    View Slide

17. @buritica
    

    View Slide

18. @buritica
    

    View Slide

19. @buritica
    2. fix the mess I
    caused by growing
    the team that fast
    

    View Slide

20. @buritica
    Splice Engineering's Mission:
    We enable Splice to learn faster
    than the market by delivering
    production-ready software at
    an accelerating tempo.
    

    View Slide

21. @buritica
    we wanted to learn
    faster than the market
    but everything felt slow
    

    View Slide

22. @buritica
    it was only a feeling
    

    View Slide

23. Things feel slow, don't they?
    yeah...
    What are we going to do about it?
    I'm not sure yet, but we'll come up
    with a plan to fix it
    ok
    

    View Slide

24. ¡ay dios mio!
    

    View Slide

25. @buritica
    what felt slow?
    

    View Slide

26. @buritica
    moving completed
    engineering work
    out to customers felt
    like an uphill battle
    

    View Slide

27. @buritica
    "staging is blocked"
    

    View Slide

28. @buritica
    "can you merge my PR?"
    

    View Slide

29. @buritica
    "how long has this
    been broken for?"
    

    View Slide

30. @buritica
    team growth had
    taken the majority of
    our attention, and
    we let our delivery
    process stagnate
    

    View Slide

31. @buritica
    given our org size,
    how we managed
    change was as
    important as the
    change desired
    

    View Slide

32. @buritica
    we needed a strategy
    

    View Slide

33. @buritica
    plan for the plan

    define metrics
    set targets
    measure
    make changes
    measure again
    

    View Slide

34. @buritica
    we researched
    

    View Slide

35. @buritica
    

    View Slide

36. @buritica
    possible metrics
    Time to Implementation
    Time to Merge
    Time to Deploy
    Deploy Confidence
    Deploy Frequency
    Support Time / Frequency
    System Stability
    Product <> Eng Partnership?
    Code review and testing?
    Deployment pipelines state?
    Test coverage?
    Deployment pipelines state?
    Quality?
    Quality?
    

    View Slide

37. @buritica
    delivery bugs
    

    View Slide

38. @buritica
    delivery bugs
    opportunities
    

    View Slide

39. @buritica
    we researched
    

    View Slide

40. @buritica
    we researched
    we discussed
    we disagreed
    we committed
    

    View Slide

41. @buritica
    and we introduced
    our first proposal to
    the team
    

    View Slide

42. @buritica
    our vision:
    

    A future where shipping working
    code is one of the fastest, safest
    and, most effective ways to learn
    and to test new ideas.
    

    View Slide

43. @buritica
    a future where:
    

    Every engineer is empowered
    by tools and processes, willing and
    able to take more significant risks.
    

    View Slide

44. @buritica
    a future where:
    

    Engineers are confident that
    the code they deliver won't break
    Splice and when it does, they can
    fix it quickly.
    

    View Slide

45. @buritica
    a future where:
    

    Everyone feels comfortable and
    welcome contributing code to any
    part of our codebase, including
    End to End tests.
    

    View Slide

46. @buritica
    a future where:
    

    ⚡We continuously expose all
    Splice employees to new
    functionality and learnings.
    

    View Slide

47. @buritica
    timeframe:
    crawl the first month
    &walk by end of Q3
    ' run by end of H2
    

    View Slide

48. @buritica
    metrics & targets:
    

    View Slide

49. @buritica
    time to merge
    

    By the end of Q3, we are
    merging 100% of Pull Requests
    within 36 hours.
    

    View Slide

50. @buritica
    deploy frequency
    

    By the end of Q3, all product
    teams are deploying at least once
    a day.
    

    View Slide

51. @buritica
    new e2e test coverage
    

    By the end of Q3, 100% of
    Engineers have written an End to
    End Test in an improved testing
    environment.
    

    View Slide

52. @buritica
    delivery working group
    

    three person infra team
    a backend & a frontend engineer
    a quality engineer
    management
    

    View Slide

53. @buritica
    [Read the full RFC here]
    

    View Slide

54. @buritica
    middle of the
    quarter came along
    

    View Slide

55. @buritica
    mid Q3 report
    "We have not moved Time to
    Merge in any significant way since
    Q3 began"
    

    View Slide

56. @buritica
    

    View Slide

57. @buritica
    

    View Slide

58. @buritica
    mid Q3 report
    "Daily deploys have gone up
    significantly since the introduction
    of Frontend Preview Branches"
    

    View Slide

59. @buritica
    

    View Slide

60. @buritica
    

    View Slide

61. @buritica
    

    View Slide

62. @buritica
    mid Q3 report
    "Some Frontend Engineers have
    begun to contribute to E2E tests,
    we still have significant ground to
    cover before end of quarter"
    

    View Slide

63. @buritica
    outlook didn't look
    that good
    

    View Slide

64. 
    i'm v busy, sorry
    
    
    
    

    View Slide

65. @buritica
    our instincts were
    right, but our story
    wasn't as convincing
    

    View Slide

66. @buritica
    I needed better
    metrics and
    measurements
    

    View Slide

67. @buritica
    "I wish someone
    surveyed thousands
    of organizations
    about their delivery
    practices" - @buritica
    

    View Slide

68. @buritica
    

    View Slide

69. @buritica
    "I wish someone else
    would tell me the
    important bits about
    all this data" -
    @buritica
    

    View Slide

70. @buritica
    

    View Slide

71. @buritica
    

    View Slide

72. @buritica
    

    View Slide

73. @buritica
    

    View Slide

74. @buritica
    Product Design & Development Product Delivery
    Create new products and services that solve
    customer problems using hypothesis-driven delivery,
    modern UX, design thinking.
    Enable fast flow from development to production
    and reliable releases by standardizing work, and
    reducing variability and batch sizes.
    Feature design and implementation may require
    work that has never been performed before.
    Integration, test, and deployment must be
    performed continuously as quickly as possible.
    Estimates are highly uncertain. Cycle times should be well-known and predictable.
    Outcomes are highly variable Outcomes should have low variability
    Novelty implemented
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    

    View Slide

75. @buritica
    Splice Tech:

    

    Product Management
    Product Design
    Engineering
    

    View Slide

76. @buritica
    Splice Tech is
    responsible for
    Design &
    Development
    

    View Slide

77. @buritica
    Splice Engineering
    is solely responsible
    for Delivery
    

    View Slide

78. @buritica
    Product Design & Development
    

    View Slide

79. @buritica
    first

    commit
    Product Design & Development
    

    View Slide

80. @buritica
    first

    commit
    product delivery
    product design
    Product Design & Development
    

    View Slide

81. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    Product Design & Development
    

    View Slide

82. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    Product Design & Development
    

    View Slide

83. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    release
    Product Design & Development
    

    View Slide

84. @buritica
    work starts
    Lead Time vs Cycle Time
    cycle time
    work requested work finished
    lead time
    

    View Slide

85. @buritica
    Delivery Performance Metrics:
    ⌚ Lead Time (commit to prod)
    Deployment Frequency
    Mean Time to Restore

    Change Failure Rate
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    

    View Slide

86. @buritica
    High Performing Teams:
    ⌚ Less than one hour
    Multiple deploys per day
    Less than one hour

    0 - 15%
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    

    View Slide

87. @buritica
    Splice Engineering at the time:
    ⌚ Between one week and one month
    Once per week to several times a month
    Between one hour to one day

    ???
    * Anecdotal experience
    

    View Slide

88. @buritica
    What about points
    & velocity?
    

    View Slide

89. @buritica
    Velocity is designed to be used
    as a capacity planning tool.
    

    View Slide

90. @buritica
    Velocity is designed to be used
    as a capacity planning tool.
    Using it as a productivity metric
    has several flaws.
    

    View Slide

91. @buritica
    Velocity is relative and team dependent.
    

    View Slide

92. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.
    

    View Slide

93. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.
    Using capacity to measure productivity
    leads to high utilization, which reduces
    ability to take unplanned work.
    

    View Slide

94. @buritica
    “Queue theory in math tells us that as
    utilization approaches 100%, lead times
    approach infinity—in other words, once
    you get to very high levels of utilization, it
    takes teams exponentially longer to get
    anything done. ”
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    

    View Slide

95. @buritica
    velocity is good for
    planning work and bad
    for measuring teams
    

    View Slide

96. @buritica
    hey boss, check this out
    
    
    

    View Slide

97. @buritica
    Delivery Matters
    Engineering performance
    affects an organization's ability
    to achieve goals beyond profit
    and revenue.
    

    View Slide

98. @buritica
    Delivery Matters
    Whatever the mission, engineering
    performance can predict overall
    organizational performance
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    

    View Slide

99. @buritica
    How does Splice
    Engineering
    become high-
    performing?
    

    View Slide

100. @buritica
     By driving the
     team towards a
     performance
     oriented culture
     

     View Slide

101. @buritica
     CONTINUOUS
     DELIVERY
     PERFORMANCE
     ORIENTED CULTURE
     LEAN PRODUCT
     MANAGEMENT
     DELIVERY
     PERFORMANCE
     ORGANIZATIONAL
     PERFORMANCE
     Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
     culture drivers culture outcomes
     

     View Slide

102. @buritica
     To achieve this goal:
     We relentlessly pursue continuous
     improvement to drive the Tech Team's
     culture towards high-performance by
     using a capabilities model proven to
     predict high performance in
     organizations.
     

     View Slide

103. @buritica
     Capability Categories
     Continuous Delivery
     Architecture
     Product & Process
     Lean management & monitoring
     Cultural
     Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
     

     View Slide

104. @buritica
     Continuous Delivery Architecture Product & Process
     Lean Mgt &
     Monitoring
     Culture
     Version control for all
     artifacts
     Loosely coupled
     architecture
     Customer feedback
     implementation
     Lightweight change
     approval
     Support a generative
     culture
     Automated deployment
     Architect for empowered
     teams
     Visibility into flow of work
     through value stream
     Infrastructure monitoring
     Encourage and support
     learning
     Continuous integration Small batched work
     Proactive system health
     checks
     Support and facilitate
     cross-team collaboration
     Trunk-based development Enable experimentation Work In Progress limits
     Provide resources and tools
     that make work meaningful
     Test automation
     Visualize work to monitor
     quality
     Transformational
     leadership
     Test data management
     Integrated security
     Continuous delivery
     Capabilities
     

     View Slide

105. @buritica
     Continuous Delivery Architecture Product & Process
     Lean Mgt &
     Monitoring
     Culture
     Version control for all
     artifacts
     Loosely coupled
     architecture
     Customer feedback
     implementation
     Lightweight change
     approval
     Support a generative
     culture
     Automated deployment
     Architect for empowered
     teams
     Visibility into flow of work
     through value stream
     Infrastructure monitoring
     Encourage and support
     learning
     Continuous integration Small batched work
     Proactive system health
     checks
     Support and facilitate
     cross-team collaboration
     Trunk-based development Enable experimentation Work In Progress limits
     Provide resources and tools
     that make work meaningful
     Test automation
     Visualize work to monitor
     quality
     Transformational
     leadership
     Test data management
     Integrated security
     Continuous delivery
     Capabilities
     

     View Slide

106. @buritica
     "I wish someone
     made a tool that
     helped me measure
     my team easily" -
     @buritica
     

     View Slide

107. @buritica
     disclaimer: this is
     not a sponsored
     talk
     

     View Slide

108. @buritica
     

     View Slide

109. @buritica
     

     View Slide

110. @buritica
     our working group has been
     successful, we'd like to invest more
     sounds good
     Gorsuch, you got this
     reeeeleaaase the
     

     View Slide

111. @buritica
     Production Engineering
     Tooling, services, and expertise that enables teams to deliver and
     operate high quality, production-level software and services.

     

     Security Engineering

     Site Reliability Engineering

     Quality Reliability Engineering

     Developer Experience (DevX)
     

     View Slide

112. @buritica
     Production Engineering
     Primary Metric: Cycle Time (commit to production)
     Secondary Metrics: 

     Mean Time to Restore, Deploy Frequency, Time to First Commit
     

     View Slide

113. @buritica
     where are we now?
     

     View Slide

114. View Slide

115. wtf cycle time
     

     View Slide

116. cycle time!
     

     View Slide

117. an accelerating organization
     

     View Slide

118. 118
     a steady rhythm
     

     View Slide

119. 119
     a performance oriented culture
     We have created a
     culture of continuous
     systems and process
     improvement. We also
     have a language and
     a framework to
     measure this change.
     

     View Slide

120. @buritica
     what changes
     caused most
     impact?
     

     View Slide

121. @buritica
     Notable Changes
     • Quality Expansion & Acceleration
     • Focused Quality Engs
     • Transferred Quality Ownership to Eng
     • New & Improved Testing Infra & Tools
     • Decentralized Staging Environment
     • Sub 2 min API Deploys
     • Stabilized Search Infrastructure
     • Decoupled Desktop UI
     • Feature Flags
     

     View Slide

122. @buritica
     what I learned
     

     View Slide

123. @buritica
     success: trust &
     change management
     

     View Slide

124. @buritica
     we can take a
     scientific approach
     as we build our teams
     

     View Slide

125. @buritica
     I can use cycle time
     to strategize about
     investments
     

     View Slide

126. my team is
     

     View Slide

127. what's next?
     

     View Slide

128. Thank you
     @buritica
     

     View Slide