Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Increasing Engineering Tempo at Splice

Increasing Engineering Tempo at Splice

A video of this talk can be found at https://codeclimate.wistia.com/medias/jcrn6ahlp1

The Splice Engineering team grew almost ten times in eighteen months. The delivery practices that worked for us when we were five broke way before we got to fifty. We wanted our organization to learn faster than the market; to do this, we had to find a way to unlock our delivery. We used insights found on the State of DevOps Report and Accelerate the book. With this knowledge and the help of metrics to visualize our delivery, we were able to get back on a high-performance track.

This talk tells the story of how we used data to engineer the performance of our engineering organization and gives you practical takeaways on how you can do it too.

Juan Pablo Buriticá

May 22, 2019
Tweet

More Decks by Juan Pablo Buriticá

Other Decks in Technology

Transcript

  1. Juan Pablo Buritica - @buritica

    speakerdeck.com/buritica

    Increasing
    Engineering
    Tempo at

    View full-size slide

  2. sounds
    studio plugins

    View full-size slide

  3. sounds
    studio plugins

    View full-size slide

  4. sounds
    studio plugins

    View full-size slide

  5. sounds
    studio plugins

    View full-size slide

  6. Empower musicians
    to realize their creative potential
    and unleash it on the world.

    View full-size slide

  7. @buritica
    I've had to solve two
    difficult problems in
    the pursuit of this
    mission

    View full-size slide

  8. @buritica
    1. grow the team to
    adapt to a new
    strategy

    View full-size slide

  9. Remember how we said we weren't
    planning on growing that much?
    yup...
    Change of plans, what do you need
    to build W, X, Y, and Z?
    more than a few engineers
    Cool, here's some $$$

    View full-size slide

  10. @buritica
    ¡ay dios mio!

    View full-size slide

  11. @buritica
    and so we grew...

    View full-size slide

  12. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60

    View full-size slide

  13. @buritica
    Splice Engineering Team Size
    2014 2015 2016 2017 2018 2019
    20
    40
    60

    View full-size slide

  14. @buritica
    2. fix the mess I
    caused by growing
    the team that fast

    View full-size slide

  15. @buritica
    Splice Engineering's Mission:
    We enable Splice to learn faster
    than the market by delivering
    production-ready software at
    an accelerating tempo.

    View full-size slide

  16. @buritica
    we wanted to learn
    faster than the market
    but everything felt slow

    View full-size slide

  17. @buritica
    it was only a feeling

    View full-size slide

  18. Things feel slow, don't they?
    yeah...
    What are we going to do about it?
    I'm not sure yet, but we'll come up
    with a plan to fix it
    ok

    View full-size slide

  19. ¡ay dios mio!

    View full-size slide

  20. @buritica
    what felt slow?

    View full-size slide

  21. @buritica
    moving completed
    engineering work
    out to customers felt
    like an uphill battle

    View full-size slide

  22. @buritica
    "staging is blocked"

    View full-size slide

  23. @buritica
    "can you merge my PR?"

    View full-size slide

  24. @buritica
    "how long has this
    been broken for?"

    View full-size slide

  25. @buritica
    team growth had
    taken the majority of
    our attention, and
    we let our delivery
    process stagnate

    View full-size slide

  26. @buritica
    given our org size,
    how we managed
    change was as
    important as the
    change desired

    View full-size slide

  27. @buritica
    we needed a strategy

    View full-size slide

  28. @buritica
    plan for the plan

    define metrics
    set targets
    measure
    make changes
    measure again

    View full-size slide

  29. @buritica
    we researched

    View full-size slide

  30. @buritica
    possible metrics
    Time to Implementation
    Time to Merge
    Time to Deploy
    Deploy Confidence
    Deploy Frequency
    Support Time / Frequency
    System Stability
    Product <> Eng Partnership?
    Code review and testing?
    Deployment pipelines state?
    Test coverage?
    Deployment pipelines state?
    Quality?
    Quality?

    View full-size slide

  31. @buritica
    delivery bugs

    View full-size slide

  32. @buritica
    delivery bugs
    opportunities

    View full-size slide

  33. @buritica
    we researched

    View full-size slide

  34. @buritica
    we researched
    we discussed
    we disagreed
    we committed

    View full-size slide

  35. @buritica
    and we introduced
    our first proposal to
    the team

    View full-size slide

  36. @buritica
    our vision:

    A future where shipping working
    code is one of the fastest, safest
    and, most effective ways to learn
    and to test new ideas.

    View full-size slide

  37. @buritica
    a future where:

    Every engineer is empowered
    by tools and processes, willing and
    able to take more significant risks.

    View full-size slide

  38. @buritica
    a future where:

    Engineers are confident that
    the code they deliver won't break
    Splice and when it does, they can
    fix it quickly.

    View full-size slide

  39. @buritica
    a future where:

    Everyone feels comfortable and
    welcome contributing code to any
    part of our codebase, including
    End to End tests.

    View full-size slide

  40. @buritica
    a future where:

    ⚡We continuously expose all
    Splice employees to new
    functionality and learnings.

    View full-size slide

  41. @buritica
    timeframe:
    crawl the first month
    &walk by end of Q3
    ' run by end of H2

    View full-size slide

  42. @buritica
    metrics & targets:

    View full-size slide

  43. @buritica
    time to merge

    By the end of Q3, we are
    merging 100% of Pull Requests
    within 36 hours.

    View full-size slide

  44. @buritica
    deploy frequency

    By the end of Q3, all product
    teams are deploying at least once
    a day.

    View full-size slide

  45. @buritica
    new e2e test coverage

    By the end of Q3, 100% of
    Engineers have written an End to
    End Test in an improved testing
    environment.

    View full-size slide

  46. @buritica
    delivery working group

    three person infra team
    a backend & a frontend engineer
    a quality engineer
    management

    View full-size slide

  47. @buritica
    [Read the full RFC here]

    View full-size slide

  48. @buritica
    middle of the
    quarter came along

    View full-size slide

  49. @buritica
    mid Q3 report
    "We have not moved Time to
    Merge in any significant way since
    Q3 began"

    View full-size slide

  50. @buritica
    mid Q3 report
    "Daily deploys have gone up
    significantly since the introduction
    of Frontend Preview Branches"

    View full-size slide

  51. @buritica
    mid Q3 report
    "Some Frontend Engineers have
    begun to contribute to E2E tests,
    we still have significant ground to
    cover before end of quarter"

    View full-size slide

  52. @buritica
    outlook didn't look
    that good

    View full-size slide


  53. i'm v busy, sorry



    View full-size slide

  54. @buritica
    our instincts were
    right, but our story
    wasn't as convincing

    View full-size slide

  55. @buritica
    I needed better
    metrics and
    measurements

    View full-size slide

  56. @buritica
    "I wish someone
    surveyed thousands
    of organizations
    about their delivery
    practices" - @buritica

    View full-size slide

  57. @buritica
    "I wish someone else
    would tell me the
    important bits about
    all this data" -
    @buritica

    View full-size slide

  58. @buritica
    Product Design & Development Product Delivery
    Create new products and services that solve
    customer problems using hypothesis-driven delivery,
    modern UX, design thinking.
    Enable fast flow from development to production
    and reliable releases by standardizing work, and
    reducing variability and batch sizes.
    Feature design and implementation may require
    work that has never been performed before.
    Integration, test, and deployment must be
    performed continuously as quickly as possible.
    Estimates are highly uncertain. Cycle times should be well-known and predictable.
    Outcomes are highly variable Outcomes should have low variability
    Novelty implemented
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  59. @buritica
    Splice Tech:


    Product Management
    Product Design
    Engineering

    View full-size slide

  60. @buritica
    Splice Tech is
    responsible for
    Design &
    Development

    View full-size slide

  61. @buritica
    Splice Engineering
    is solely responsible
    for Delivery

    View full-size slide

  62. @buritica
    Product Design & Development

    View full-size slide

  63. @buritica
    first

    commit
    Product Design & Development

    View full-size slide

  64. @buritica
    first

    commit
    product delivery
    product design
    Product Design & Development

    View full-size slide

  65. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    Product Design & Development

    View full-size slide

  66. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    Product Design & Development

    View full-size slide

  67. @buritica
    first

    commit
    product delivery
    product design
    PR merge
    deployment
    release
    Product Design & Development

    View full-size slide

  68. @buritica
    work starts
    Lead Time vs Cycle Time
    cycle time
    work requested work finished
    lead time

    View full-size slide

  69. @buritica
    Delivery Performance Metrics:
    ⌚ Lead Time (commit to prod)
    Deployment Frequency
    Mean Time to Restore

    Change Failure Rate
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  70. @buritica
    High Performing Teams:
    ⌚ Less than one hour
    Multiple deploys per day
    Less than one hour

    0 - 15%
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  71. @buritica
    Splice Engineering at the time:
    ⌚ Between one week and one month
    Once per week to several times a month
    Between one hour to one day

    ???
    * Anecdotal experience

    View full-size slide

  72. @buritica
    What about points
    & velocity?

    View full-size slide

  73. @buritica
    Velocity is designed to be used
    as a capacity planning tool.

    View full-size slide

  74. @buritica
    Velocity is designed to be used
    as a capacity planning tool.
    Using it as a productivity metric
    has several flaws.

    View full-size slide

  75. @buritica
    Velocity is relative and team dependent.

    View full-size slide

  76. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.

    View full-size slide

  77. @buritica
    Velocity is relative and team dependent.
    When used as productivity metric, teams
    game it, which impacts collaboration.
    Using capacity to measure productivity
    leads to high utilization, which reduces
    ability to take unplanned work.

    View full-size slide

  78. @buritica
    “Queue theory in math tells us that as
    utilization approaches 100%, lead times
    approach infinity—in other words, once
    you get to very high levels of utilization, it
    takes teams exponentially longer to get
    anything done. ”
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  79. @buritica
    velocity is good for
    planning work and bad
    for measuring teams

    View full-size slide

  80. @buritica
    hey boss, check this out


    View full-size slide

  81. @buritica
    Delivery Matters
    Engineering performance
    affects an organization's ability
    to achieve goals beyond profit
    and revenue.

    View full-size slide

  82. @buritica
    Delivery Matters
    Whatever the mission, engineering
    performance can predict overall
    organizational performance
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  83. @buritica
    How does Splice
    Engineering
    become high-
    performing?

    View full-size slide

  84. @buritica
    By driving the
    team towards a
    performance
    oriented culture

    View full-size slide

  85. @buritica
    CONTINUOUS
    DELIVERY
    PERFORMANCE
    ORIENTED CULTURE
    LEAN PRODUCT
    MANAGEMENT
    DELIVERY
    PERFORMANCE
    ORGANIZATIONAL
    PERFORMANCE
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”
    culture drivers culture outcomes

    View full-size slide

  86. @buritica
    To achieve this goal:
    We relentlessly pursue continuous
    improvement to drive the Tech Team's
    culture towards high-performance by
    using a capabilities model proven to
    predict high performance in
    organizations.

    View full-size slide

  87. @buritica
    Capability Categories
    Continuous Delivery
    Architecture
    Product & Process
    Lean management & monitoring
    Cultural
    Nicole Forsgren PhD, Jez Humble & Gene Kim. “Accelerate.”

    View full-size slide

  88. @buritica
    Continuous Delivery Architecture Product & Process
    Lean Mgt &
    Monitoring
    Culture
    Version control for all
    artifacts
    Loosely coupled
    architecture
    Customer feedback
    implementation
    Lightweight change
    approval
    Support a generative
    culture
    Automated deployment
    Architect for empowered
    teams
    Visibility into flow of work
    through value stream
    Infrastructure monitoring
    Encourage and support
    learning
    Continuous integration Small batched work
    Proactive system health
    checks
    Support and facilitate
    cross-team collaboration
    Trunk-based development Enable experimentation Work In Progress limits
    Provide resources and tools
    that make work meaningful
    Test automation
    Visualize work to monitor
    quality
    Transformational
    leadership
    Test data management
    Integrated security
    Continuous delivery
    Capabilities

    View full-size slide

  89. @buritica
    Continuous Delivery Architecture Product & Process
    Lean Mgt &
    Monitoring
    Culture
    Version control for all
    artifacts
    Loosely coupled
    architecture
    Customer feedback
    implementation
    Lightweight change
    approval
    Support a generative
    culture
    Automated deployment
    Architect for empowered
    teams
    Visibility into flow of work
    through value stream
    Infrastructure monitoring
    Encourage and support
    learning
    Continuous integration Small batched work
    Proactive system health
    checks
    Support and facilitate
    cross-team collaboration
    Trunk-based development Enable experimentation Work In Progress limits
    Provide resources and tools
    that make work meaningful
    Test automation
    Visualize work to monitor
    quality
    Transformational
    leadership
    Test data management
    Integrated security
    Continuous delivery
    Capabilities

    View full-size slide

  90. @buritica
    "I wish someone
    made a tool that
    helped me measure
    my team easily" -
    @buritica

    View full-size slide

  91. @buritica
    disclaimer: this is
    not a sponsored
    talk

    View full-size slide

  92. @buritica
    our working group has been
    successful, we'd like to invest more
    sounds good
    Gorsuch, you got this
    reeeeleaaase the

    View full-size slide

  93. @buritica
    Production Engineering
    Tooling, services, and expertise that enables teams to deliver and
    operate high quality, production-level software and services.


    Security Engineering

    Site Reliability Engineering

    Quality Reliability Engineering

    Developer Experience (DevX)

    View full-size slide

  94. @buritica
    Production Engineering
    Primary Metric: Cycle Time (commit to production)
    Secondary Metrics: 

    Mean Time to Restore, Deploy Frequency, Time to First Commit

    View full-size slide

  95. @buritica
    where are we now?

    View full-size slide

  96. wtf cycle time

    View full-size slide

  97. an accelerating organization

    View full-size slide

  98. 118
    a steady rhythm

    View full-size slide

  99. 119
    a performance oriented culture
    We have created a
    culture of continuous
    systems and process
    improvement. We also
    have a language and
    a framework to
    measure this change.

    View full-size slide

  100. @buritica
    what changes
    caused most
    impact?

    View full-size slide

  101. @buritica
    Notable Changes
    • Quality Expansion & Acceleration
    • Focused Quality Engs
    • Transferred Quality Ownership to Eng
    • New & Improved Testing Infra & Tools
    • Decentralized Staging Environment
    • Sub 2 min API Deploys
    • Stabilized Search Infrastructure
    • Decoupled Desktop UI
    • Feature Flags

    View full-size slide

  102. @buritica
    what I learned

    View full-size slide

  103. @buritica
    success: trust &
    change management

    View full-size slide

  104. @buritica
    we can take a
    scientific approach
    as we build our teams

    View full-size slide

  105. @buritica
    I can use cycle time
    to strategize about
    investments

    View full-size slide

  106. what's next?

    View full-size slide

  107. Thank you
    @buritica

    View full-size slide