Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery at scale - social and technical practices for fast flow - Conflux - Agile Manchester 2023

Continuous Delivery at scale - social and technical practices for fast flow - Conflux - Agile Manchester 2023

Fast flow delivers vital business outcomes:

* Improved efficiency due to removed dependencies and reduced rework
* Shorter time from hypothesis to user value
* Greater transparency and traceability in service development and operation
* Quicker problem diagnosis and service restoration
* Happier teams

But what are the social and technical practices that enable fast flow?

At the heart of fast flow is the practice of Continuous Delivery (CD). CD practices have increasingly become “table stakes” for effective software delivery since the publication of the book ‘Continuous Delivery’ by Jez Humble and Dave Farley in 2010. The combination of sound technical practices and a scientific approach to testing and feature development has led to significant success with software delivery with organizations around the world.

However, in large organizations with many teams and many different suppliers, there are also many ideas about what Continuous Delivery is. This variety of approaches and assumptions can lead to conflicts around deployments, testing, releasing, and operations, resulting in a macro-level sub-optimal flow of change.

In this talk, Matthew Skelton will share some insights from his time spent as Engineering Lead at a large GOV.UK department during most of 2018. There he championed Continuous Delivery practices across 70+ teams and 7 locations, helping to raise standards for software operability, Developer Experience (DevEx), testing, deployments, and inter-team communications. He will share some practical techniques for getting Continuous Delivery working at scale.

---
From a keynote talk at Agile Manchester conference on 10 May 2023 - https://agilemanchester.net/programme/continuous-delivery-scale-social-and-technical-practices-fast-flow

Matthew Skelton
PRO

May 10, 2023
Tweet

More Decks by Matthew Skelton

Other Decks in Business

Transcript

  1. Continuous Delivery at scale
    Social and technical practices for fast flow
    Matthew Skelton, Founder at Conflux
    K03
    Keynote at Agile Manchester | 10 May 2023

    View Slide

  2. Photo goes here
    Matthew Skelton
    Founder at Conflux
    Co-author of Team Topologies
    2

    View Slide

  3. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    3

    View Slide

  4. 4

    View Slide

  5. 5
    Fast flow: multiple ongoing
    streams of changes to
    software-enriched services
    done safely and sustainably
    https://confluxhq.com/fast-flow

    View Slide

  6. Fast flow delivers vital
    business outcomes
    6

    View Slide

  7. 7
    Improved efficiency due to
    removed dependencies and
    reduced rework

    View Slide

  8. 8
    Shorter time from
    hypothesis to user value

    View Slide

  9. 9
    Greater transparency and
    traceability in service
    development and operation

    View Slide

  10. 10
    Quicker problem diagnosis
    and service restoration

    View Slide

  11. 11
    Happier teams

    View Slide

  12. 12
    Improved efficiency due to removed
    dependencies and reduced rework
    Shorter time from hypothesis to user value
    Greater transparency and traceability in
    service development and operation
    Quicker problem diagnosis and service
    restoration
    Happier teams

    View Slide

  13. What social and technical
    practices are needed for fast
    flow at scale?
    13

    View Slide

  14. 14
    Make space for learning/sharing
    Define the Platform
    Address operability early on
    Good logging is foundational
    Empower teams to improve

    View Slide

  15. Case study: context & challenges
    Guiding principles for fast flow
    What we did to help fast flow
    Results and key takeaways
    15

    View Slide

  16. Case study:
    context & challenges
    16

    View Slide

  17. Case Study
    UK Home Office
    (GOV.UK)
    17

    View Slide

  18. Case Study
    18

    View Slide

  19. Case Study
    19
    700+ people in total
    70+ engineering teams, 7 locations
    Time-critical delivery (#Brexit)
    Most of 2018

    View Slide

  20. Case Study
    20
    Increase speed & safety of delivery
    Multi-year programme
    New Brexit-related work
    Track & control infra costs

    View Slide

  21. Case Study
    21
    (Traditional DC + VMs + JVM)
    AWS + Kubernetes + JVM
    ELK for logs and transaction traces
    [ ]

    View Slide

  22. Case Study
    Scale existing approach
    to 7x size
    22

    View Slide

  23. Case Study
    Spread awareness of
    effective practices
    23

    View Slide

  24. Case Study
    Cross internal and
    external boundaries
    24

    View Slide

  25. Case Study
    “Continuous Delivery”
    25

    View Slide

  26. Case Study
    Avoid “one-size-fits-all”
    26

    View Slide

  27. Case Study
    Define the Platform
    27

    View Slide

  28. Case Study
    Improve logging quality
    28

    View Slide

  29. Case Study
    Multi-supplier delivery
    model
    29

    View Slide

  30. What do we aim for in the
    context of fast flow?
    30

    View Slide

  31. 31
    Continuously ‘untangle’
    business concepts

    View Slide

  32. 32
    Find and adjust team &
    system boundaries for flow

    View Slide

  33. 33
    Minimize hand-offs

    View Slide

  34. 34
    Avoid blocking
    dependencies

    View Slide

  35. 35
    Move some decision-making
    to teams

    View Slide

  36. 36
    ➔ Untangle business concepts
    ➔ Adjust boundaries for flow
    ➔ Minimize hand-offs
    ➔ Remove blocking dependencies
    ➔ Move decision-making to teams

    View Slide

  37. 37
    Example: deployment pipeline
    architecture

    View Slide

  38. 38

    View Slide

  39. 39

    View Slide

  40. 40

    View Slide

  41. 41

    View Slide

  42. 42
    The architecture of
    deployment pipelines is
    a key consideration for
    fast flow

    View Slide

  43. 43
    “Show me your
    deployment pipelines and
    I will predict your team
    boundaries” 💥

    View Slide

  44. Guiding principles for
    fast flow
    45

    View Slide

  45. 46
    Reliable Software Releases
    Through Build, Test, and
    Deployment Automation
    Jez Humble & Dave Farley,
    2010
    Continuous
    Delivery

    View Slide

  46. Continuous Delivery with
    Windows and .NET
    47
    Matthew Skelton & Chris O’Dell
    O’Reilly, 2016
    cdwithwindows.net
    ** Now a key text for MSc in
    Software Engineering at UCL **

    View Slide

  47. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    48

    View Slide

  48. 49
    Re-aligned
    architecture

    View Slide

  49. 50
    Fast feedback via
    deployment pipelines

    View Slide

  50. 51
    Good technical
    practices (TDD, …)

    View Slide

  51. 52
    Team ownership of
    software & services

    View Slide

  52. 53
    Configuration in
    version control (Git)

    View Slide

  53. 54
    Cloud-native:
    transparent in
    operation

    View Slide

  54. 55
    Cloud-native:
    designed for
    automation

    View Slide

  55. 56
    Continuous
    testing
    performance
    scanning
    deployment
    monitoring
    right-sizing
    integration

    View Slide

  56. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    57

    View Slide

  57. What we did to help fast flow
    58

    View Slide

  58. Case Study
    New flow-based
    operating model
    59

    View Slide

  59. 60
    Flow of change

    View Slide

  60. Case Study
    Focus on the
    "operator experience"
    61

    View Slide

  61. Case Study
    Define the platform
    62

    View Slide

  62. Case Study
    Engineering Guild
    +
    Engineering Working
    Group
    64

    View Slide

  63. Case Study
    Weekly lunchtime tech
    talks
    65

    View Slide

  64. Case Study
    ‘Continuous Delivery for
    people who don’t write
    code’
    Slides:
    https://www.slideshare.net/matthewskelton/continuous-delivery-for-p
    eople-who-do-not-write-code-matthew-skelton-conflux
    66

    View Slide

  65. Case Study
    Team-first techniques
    Example:
    Run Book Dialogue Sheet
    67

    View Slide

  66. 68
    runbooktemplate.info
    Run Book dialogue sheets

    View Slide

  67. Case Study
    Engineering Standards
    69

    View Slide

  68. 70

    View Slide

  69. Engineering Standards
    71
    Minimal but important constraints
    Dynamic for reviewing and updating
    Emphasise team engagement

    View Slide

  70. Case Study
    Engineering Assessments
    72

    View Slide

  71. Engineering Assessments
    73
    6 dimensions [in 2018]:
    Team Health, Deployment, Continuous
    Delivery, Flow, Operability, Testing
    SoftwareDeliveryAssessment.com

    View Slide

  72. Engineering Assessments
    74
    Criteria taken from existing sources:
    Spotify Team Health Check, DevOps for the Modern
    Enterprise, Continuous Delivery, Accelerate, Team
    Guide to Software Operability, Team Guide to
    Software Testability

    View Slide

  73. 75

    View Slide

  74. 76

    View Slide

  75. 77

    View Slide

  76. Engineering Assessments
    78
    Team self-assessment
    Facilitated to guide discussions
    “Viral” facilitation model
    Results collated across teams

    View Slide

  77. 79

    View Slide

  78. 80
    SoftwareDeliveryAssessment.com

    View Slide

  79. Many of these principles
    and practices appear in the
    Team Topologies book 💡
    82

    View Slide

  80. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    83

    View Slide

  81. Results and key takeaways
    84

    View Slide

  82. Case Study
    85
    Software releases
    From 2 successful release candidate
    builds per week to 8 builds per day
    Multiple, independent routes to live

    View Slide

  83. Case Study
    86
    Operability
    Ops teams love the new "operator
    experience" with logging &
    dashboards
    No major operational problems with
    new releases

    View Slide

  84. Case Study
    87
    Collaboration
    Logging beginning to be seen as a
    key observability tool
    Better cross-team awareness of how
    systems work through telemetry

    View Slide

  85. Case Study
    Moving towards
    Continuous Delivery at
    DDaT
    See public blog post at
    https://hodigital.blog.gov.uk/2018/08/10/moving-tow
    ards-continuous-delivery-in-ddat/
    88

    View Slide

  86. Case Study
    89
    (Many other people were
    also involved)

    View Slide

  87. Case Study
    90
    Ideas and patterns from
    Team Topologies now
    being used in this
    department

    View Slide

  88. What social and technical
    practices are needed for fast
    flow at scale?
    91

    View Slide

  89. 92
    Make space for learning/sharing
    Promote good work
    Help to develop skills in speaking
    Lunchtime talks (food)
    Interplay of working group + guild

    View Slide

  90. 93
    Define the Platform
    Service Level Expectations
    Who owns what? What's missing?
    Use DevEx to drive improvements
    Thinnest Viable Platform

    View Slide

  91. 94
    Address operability early on
    Operational aspects are also features
    Add operational checks into the
    deployment pipeline
    Team-first: RunBookTemplate.info

    View Slide

  92. 95
    Good logging is foundational
    Use a well-defined event space
    Searchable EventIDs increase context
    Enables collaboration between teams
    Invest in logging infra (Saas/people/&c.)

    View Slide

  93. 96
    Empower teams to improve
    Co-create the engineering standards
    Look for organisational blockers
    “No-blame” is paramount
    SoftwareDeliveryAssessment.com

    View Slide

  94. 98
    The architecture of
    deployment pipelines is
    a key consideration for
    fast flow

    View Slide

  95. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    100

    View Slide

  96. expert-led
    group learning
    ❏ P602 - Software Delivery
    Assessment
    ❏ P616 - software reliability for
    product managers
    ❏ P630 - Find possible streams
    with Independent Service
    Heuristics
    ❏ P634 - Peer Learning - Writing
    Tech Talks
    ❏ P641f - Preparing to adopt
    Team Topologies
    ❏ P641i - Skills paths for fast
    flow - leaders and HR
    101
    online workshops for up to
    20 people

    View Slide

  97. 102
    expert-led group learning for adopting
    fast flow and Team Topologies
    confluxhq.com

    View Slide

  98. thank you
    confluxhq.com
    Copyright (c) 2017-2023 Conflux group of companies. All Rights Reserved.
    The name “Conflux” and the filled C device are Registered Trademarks ® in multiple jurisdictions.
    103

    View Slide