Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Increasing Engineering Tempo at Splice

Increasing Engineering Tempo at Splice

A video of this talk can be found at https://codeclimate.wistia.com/medias/jcrn6ahlp1

The Splice Engineering team grew almost ten times in eighteen months. The delivery practices that worked for us when we were five broke way before we got to fifty. We wanted our organization to learn faster than the market; to do this, we had to find a way to unlock our delivery. We used insights found on the State of DevOps Report and Accelerate the book. With this knowledge and the help of metrics to visualize our delivery, we were able to get back on a high-performance track.

This talk tells the story of how we used data to engineer the performance of our engineering organization and gives you practical takeaways on how you can do it too.

Juan Pablo Buriticá

May 22, 2019
Tweet

More Decks by Juan Pablo Buriticá

Other Decks in Technology

Transcript

  1. Juan Pablo Buritica - @buritica

    speakerdeck.com/buritica

    Increasing
    Engineering
    Tempo at

    View Slide

  2. View Slide

  3. sounds
    studio plugins

    View Slide

  4. sounds
    studio plugins

    View Slide

  5. sounds
    studio plugins

    View Slide

  6. sounds
    studio plugins

    View Slide

  7. Empower musicians
    to realize their creative potential
    and unleash it on the world.

    View Slide

  8. @buritica

    View Slide

  9. @buritica

    View Slide

  10. @buritica
    I've had to solve two
    difficult problems in
    the pursuit of this
    mission

    View Slide

  11. @buritica
    1. grow the team to
    adapt to a new
    strategy

    View Slide

  12. Remember how we said we weren't
    planning on growing that much?
    yup...
    Change of plans, what do you need
    to build W, X, Y, and Z?
    more than a few engineers
    Cool, here's some $$$

    View Slide

  13. @buritica
    ¡ay dios mio!

    View Slide

  14. @buritica
    and so we grew...

    View Slide

  15. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60

    View Slide

  16. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60

    View Slide

  17. @buritica

    View Slide

  18. @buritica

    View Slide

  19. @buritica
    2. fix the mess I
    caused by growing
    the team that fast

    View Slide

  20. @buritica
    Splice Engineering's Mission:
    We enable Splice to learn faster
    than the market by delivering
    production-ready software at
    an accelerating tempo.

    View Slide

  21. @buritica
    we wanted to learn
    faster than the market
    but everything felt slow

    View Slide

  22. @buritica
    it was only a feeling

    View Slide

  23. Things feel slow, don't they?
    yeah...
    What are we going to do about it?
    I'm not sure yet, but we'll come up
    with a plan to fix it
    ok

    View Slide

  24. ¡ay dios mio!

    View Slide

  25. @buritica
    what felt slow?

    View Slide

  26. @buritica
    moving completed
    engineering work
    out to customers felt
    like an uphill battle

    View Slide

  27. @buritica
    "staging is blocked"

    View Slide

  28. @buritica
    "can you merge my PR?"

    View Slide

  29. @buritica
    "how long has this
    been broken for?"

    View Slide

  30. @buritica
    team growth had
    taken the majority of
    our attention, and
    we let our delivery
    process stagnate

    View Slide

  31. @buritica
    given our org size,
    how we managed
    change was as
    important as the
    change desired

    View Slide

  32. @buritica
    we needed a strategy

    View Slide

  33. @buritica
    plan for the plan

    define metrics
    set targets
    measure
    make changes
    measure again

    View Slide

  34. @buritica
    we researched

    View Slide

  35. @buritica

    View Slide

  36. @buritica
    possible metrics
    Time to Implementation
    Time to Merge
    Time to Deploy
    Deploy Confidence
    Deploy Frequency
    Support Time / Frequency
    System Stability
    Product <> Eng Partnership?
    Code review and testing?
    Deployment pipelines state?
    Test coverage?
    Deployment pipelines state?
    Quality?
    Quality?

    View Slide

  37. @buritica
    delivery bugs

    View Slide

  38. @buritica
    delivery bugs
    opportunities

    View Slide

  39. @buritica
    we researched

    View Slide

  40. @buritica
    we researched
    we discussed
    we disagreed
    we committed

    View Slide

  41. @buritica
    and we introduced
    our first proposal to
    the team

    View Slide

  42. @buritica
    our vision:

    A future where shipping working
    code is one of the fastest, safest
    and, most effective ways to learn
    and to test new ideas.

    View Slide

  43. @buritica
    a future where:

    Every engineer is empowered
    by tools and processes, willing and
    able to take more significant risks.

    View Slide

  44. @buritica
    a future where:

    Engineers are confident that
    the code they deliver won't break
    Splice and when it does, they can
    fix it quickly.

    View Slide

  45. @buritica
    a future where:

    Everyone feels comfortable and
    welcome contributing code to any
    part of our codebase, including
    End to End tests.

    View Slide

  46. @buritica
    a future where:

    ⚡We continuously expose all
    Splice employees to new
    functionality and learnings.

    View Slide

  47. @buritica
    timeframe:
    crawl the first month
    &walk by end of Q3
    ' run by end of H2

    View Slide

  48. @buritica
    metrics & targets:

    View Slide

  49. @buritica
    time to merge

    By the end of Q3, we are
    merging 100% of Pull Requests
    within 36 hours.

    View Slide

  50. @buritica
    deploy frequency

    By the end of Q3, all product
    teams are deploying at least once
    a day.

    View Slide

  51. @buritica
    new e2e test coverage

    By the end of Q3, 100% of
    Engineers have written an End to
    End Test in an improved testing
    environment.

    View Slide

  52. @buritica
    delivery working group

    three person infra team
    a backend & a frontend engineer
    a quality engineer
    management

    View Slide

  53. @buritica
    [Read the full RFC here]

    View Slide

  54. @buritica
    middle of the
    quarter came along

    View Slide

  55. @buritica
    mid Q3 report
    "We have not moved Time to
    Merge in any significant way since
    Q3 began"

    View Slide

  56. @buritica

    View Slide

  57. @buritica

    View Slide

  58. @buritica
    mid Q3 report
    "Daily deploys have gone up
    significantly since the introduction
    of Frontend Preview Branches"

    View Slide

  59. @buritica

    View Slide

  60. @buritica

    View Slide

  61. @buritica

    View Slide

  62. @buritica
    mid Q3 report
    "Some Frontend Engineers have
    begun to contribute to E2E tests,
    we still have significant ground to
    cover before end of quarter"

    View Slide

  63. @buritica
    outlook didn't look
    that good

    View Slide


  64. i'm v busy, sorry



    View Slide

  65. @buritica
    our instincts were
    right, but our story
    wasn't as convincing

    View Slide

  66. @buritica
    I needed better
    metrics and
    measurements

    View Slide

  67. @buritica
    "I wish someone
    surveyed thousands
    of organizations
    about their delivery
    practices" - @buritica

    View Slide

  68. @buritica

    View Slide

  69. @buritica
    "I wish someone else
    would tell me the
    important bits about
    all this data" -
    @buritica

    View Slide

  70. @buritica

    View Slide

  71. @buritica

    View Slide

  72. @buritica

    View Slide

  73. @buritica

    View Slide

  74. @buritica
    Product Design & Development Product Delivery
    Create new products and services that solve
    customer problems using hypothesis-driven delivery,
    modern UX, design thinking.
    Enable fast flow from development to production
    and reliable releases by standardizing work, and
    reducing variability and batch sizes.
    Feature design and implementation may require
    work that has never been performed before.
    Integration, test, and deployment must be
    performed continuously as quickly as possible.
    Estimates are highly uncertain. Cycle times should be well-known and predictable.
    Outcomes are highly variable Outcomes should have low variability
    Novelty implemented
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  75. @buritica
    Splice Tech:


    Product Management
    Product Design
    Engineering

    View Slide

  76. @buritica
    Splice Tech is
    responsible for
    Design &
    Development

    View Slide

  77. @buritica
    Splice Engineering
    is solely responsible
    for Delivery

    View Slide

  78. @buritica
    Product Design & Development

    View Slide

  79. @buritica
    first

    commit
    Product Design & Development

    View Slide

  80. @buritica
    first

    commit
    product delivery
    product design
    Product Design & Development

    View Slide

  81. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    Product Design & Development

    View Slide

  82. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    Product Design & Development

    View Slide

  83. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    release
    Product Design & Development

    View Slide

  84. @buritica
    work starts
    Lead Time vs Cycle Time
    cycle time
    work requested work finished
    lead time

    View Slide

  85. @buritica
    Delivery Performance Metrics:
    ⌚ Lead Time (commit to prod)
    Deployment Frequency
    Mean Time to Restore

    Change Failure Rate
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  86. @buritica
    High Performing Teams:
    ⌚ Less than one hour
    Multiple deploys per day
    Less than one hour

    0 - 15%
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  87. @buritica
    Splice Engineering at the time:
    ⌚ Between one week and one month
    Once per week to several times a month
    Between one hour to one day

    ???
    * Anecdotal experience

    View Slide

  88. @buritica
    What about points
    & velocity?

    View Slide

  89. @buritica
    Velocity is designed to be used
    as a capacity planning tool.

    View Slide

  90. @buritica
    Velocity is designed to be used
    as a capacity planning tool.
    Using it as a productivity metric
    has several flaws.

    View Slide

  91. @buritica
    Velocity is relative and team dependent.

    View Slide

  92. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.

    View Slide

  93. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.
    Using capacity to measure productivity
    leads to high utilization, which reduces
    ability to take unplanned work.

    View Slide

  94. @buritica
    “Queue theory in math tells us that as
    utilization approaches 100%, lead times
    approach infinity—in other words, once
    you get to very high levels of utilization, it
    takes teams exponentially longer to get
    anything done. ”
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  95. @buritica
    velocity is good for
    planning work and bad
    for measuring teams

    View Slide

  96. @buritica
    hey boss, check this out


    View Slide

  97. @buritica
    Delivery Matters
    Engineering performance
    affects an organization's ability
    to achieve goals beyond profit
    and revenue.

    View Slide

  98. @buritica
    Delivery Matters
    Whatever the mission, engineering
    performance can predict overall
    organizational performance
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  99. @buritica
    How does Splice
    Engineering
    become high-
    performing?

    View Slide

  100. @buritica
    By driving the
    team towards a
    performance
    oriented culture

    View Slide

  101. @buritica
    CONTINUOUS
    DELIVERY
    PERFORMANCE
    ORIENTED CULTURE
    LEAN PRODUCT
    MANAGEMENT
    DELIVERY
    PERFORMANCE
    ORGANIZATIONAL
    PERFORMANCE
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    culture drivers culture outcomes

    View Slide

  102. @buritica
    To achieve this goal:
    We relentlessly pursue continuous
    improvement to drive the Tech Team's
    culture towards high-performance by
    using a capabilities model proven to
    predict high performance in
    organizations.

    View Slide

  103. @buritica
    Capability Categories
    Continuous Delivery
    Architecture
    Product & Process
    Lean management & monitoring
    Cultural
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View Slide

  104. @buritica
    Continuous Delivery Architecture Product & Process
    Lean Mgt &
    Monitoring
    Culture
    Version control for all
    artifacts
    Loosely coupled
    architecture
    Customer feedback
    implementation
    Lightweight change
    approval
    Support a generative
    culture
    Automated deployment
    Architect for empowered
    teams
    Visibility into flow of work
    through value stream
    Infrastructure monitoring
    Encourage and support
    learning
    Continuous integration Small batched work
    Proactive system health
    checks
    Support and facilitate
    cross-team collaboration
    Trunk-based development Enable experimentation Work In Progress limits
    Provide resources and tools
    that make work meaningful
    Test automation
    Visualize work to monitor
    quality
    Transformational
    leadership
    Test data management
    Integrated security
    Continuous delivery
    Capabilities

    View Slide

  105. @buritica
    Continuous Delivery Architecture Product & Process
    Lean Mgt &
    Monitoring
    Culture
    Version control for all
    artifacts
    Loosely coupled
    architecture
    Customer feedback
    implementation
    Lightweight change
    approval
    Support a generative
    culture
    Automated deployment
    Architect for empowered
    teams
    Visibility into flow of work
    through value stream
    Infrastructure monitoring
    Encourage and support
    learning
    Continuous integration Small batched work
    Proactive system health
    checks
    Support and facilitate
    cross-team collaboration
    Trunk-based development Enable experimentation Work In Progress limits
    Provide resources and tools
    that make work meaningful
    Test automation
    Visualize work to monitor
    quality
    Transformational
    leadership
    Test data management
    Integrated security
    Continuous delivery
    Capabilities

    View Slide

  106. @buritica
    "I wish someone
    made a tool that
    helped me measure
    my team easily" -
    @buritica

    View Slide

  107. @buritica
    disclaimer: this is
    not a sponsored
    talk

    View Slide

  108. @buritica

    View Slide

  109. @buritica

    View Slide

  110. @buritica
    our working group has been
    successful, we'd like to invest more
    sounds good
    Gorsuch, you got this
    reeeeleaaase the

    View Slide

  111. @buritica
    Production Engineering
    Tooling, services, and expertise that enables teams to deliver and
    operate high quality, production-level software and services.


    Security Engineering

    Site Reliability Engineering

    Quality Reliability Engineering

    Developer Experience (DevX)

    View Slide

  112. @buritica
    Production Engineering
    Primary Metric: Cycle Time (commit to production)
    Secondary Metrics: 

    Mean Time to Restore, Deploy Frequency, Time to First Commit

    View Slide

  113. @buritica
    where are we now?

    View Slide

  114. View Slide

  115. wtf cycle time

    View Slide

  116. cycle time!

    View Slide

  117. an accelerating organization

    View Slide

  118. 118
    a steady rhythm

    View Slide

  119. 119
    a performance oriented culture
    We have created a
    culture of continuous
    systems and process
    improvement. We also
    have a language and
    a framework to
    measure this change.

    View Slide

  120. @buritica
    what changes
    caused most
    impact?

    View Slide

  121. @buritica
    Notable Changes
    • Quality Expansion & Acceleration
    • Focused Quality Engs
    • Transferred Quality Ownership to Eng
    • New & Improved Testing Infra & Tools
    • Decentralized Staging Environment
    • Sub 2 min API Deploys
    • Stabilized Search Infrastructure
    • Decoupled Desktop UI
    • Feature Flags

    View Slide

  122. @buritica
    what I learned

    View Slide

  123. @buritica
    success: trust &
    change management

    View Slide

  124. @buritica
    we can take a
    scientific approach
    as we build our teams

    View Slide

  125. @buritica
    I can use cycle time
    to strategize about
    investments

    View Slide

  126. my team is

    View Slide

  127. what's next?

    View Slide

  128. Thank you
    @buritica

    View Slide