Continuous Delivery at scale - social and technical practices for fast flow - Conflux - Agile Manchester 2023

Fast flow delivers vital business outcomes:

* Improved efficiency due to removed dependencies and reduced rework
* Shorter time from hypothesis to user value
* Greater transparency and traceability in service development and operation
* Quicker problem diagnosis and service restoration
* Happier teams

But what are the social and technical practices that enable fast flow?

At the heart of fast flow is the practice of Continuous Delivery (CD). CD practices have increasingly become “table stakes” for effective software delivery since the publication of the book ‘Continuous Delivery’ by Jez Humble and Dave Farley in 2010. The combination of sound technical practices and a scientific approach to testing and feature development has led to significant success with software delivery with organizations around the world.

However, in large organizations with many teams and many different suppliers, there are also many ideas about what Continuous Delivery is. This variety of approaches and assumptions can lead to conflicts around deployments, testing, releasing, and operations, resulting in a macro-level sub-optimal flow of change.

In this talk, Matthew Skelton will share some insights from his time spent as Engineering Lead at a large GOV.UK department during most of 2018. There he championed Continuous Delivery practices across 70+ teams and 7 locations, helping to raise standards for software operability, Developer Experience (DevEx), testing, deployments, and inter-team communications. He will share some practical techniques for getting Continuous Delivery working at scale.

---
From a keynote talk at Agile Manchester conference on 10 May 2023 - https://agilemanchester.net/programme/continuous-delivery-scale-social-and-technical-practices-fast-flow

Transcript

1. Continuous Delivery at scale
   Social and technical practices for fast flow
   Matthew Skelton, Founder at Conflux
   K03
   Keynote at Agile Manchester | 10 May 2023
   

   View Slide

2. Photo goes here
   Matthew Skelton
   Founder at Conflux
   Co-author of Team Topologies
   2
   

   View Slide

3. Team Topologies
   Organizing business and
   technology teams for fast flow
   Matthew Skelton & Manuel Pais
   IT Revolution Press, September 2019
   Order via stores worldwide:
   teamtopologies.com/book
   3
   

   View Slide

4. View Slide

5. 5
   Fast flow: multiple ongoing
   streams of changes to
   software-enriched services
   done safely and sustainably
   https://confluxhq.com/fast-flow
   

   View Slide

6. Fast flow delivers vital
   business outcomes
   6
   

   View Slide

7. 7
   Improved efficiency due to
   removed dependencies and
   reduced rework
   

   View Slide

8. 8
   Shorter time from
   hypothesis to user value
   

   View Slide

9. 9
   Greater transparency and
   traceability in service
   development and operation
   

   View Slide

10. 10
    Quicker problem diagnosis
    and service restoration
    

    View Slide

11. 11
    Happier teams
    

    View Slide

12. 12
    Improved efficiency due to removed
    dependencies and reduced rework
    Shorter time from hypothesis to user value
    Greater transparency and traceability in
    service development and operation
    Quicker problem diagnosis and service
    restoration
    Happier teams
    

    View Slide

13. What social and technical
    practices are needed for fast
    flow at scale?
    13
    

    View Slide

14. 14
    Make space for learning/sharing
    Define the Platform
    Address operability early on
    Good logging is foundational
    Empower teams to improve
    

    View Slide

15. Case study: context & challenges
    Guiding principles for fast flow
    What we did to help fast flow
    Results and key takeaways
    15
    

    View Slide

16. Case study:
    context & challenges
    16
    

    View Slide

17. Case Study
    UK Home Office
    (GOV.UK)
    17
    

    View Slide

18. Case Study
    18
    

    View Slide

19. Case Study
    19
    700+ people in total
    70+ engineering teams, 7 locations
    Time-critical delivery (#Brexit)
    Most of 2018
    

    View Slide

20. Case Study
    20
    Increase speed & safety of delivery
    Multi-year programme
    New Brexit-related work
    Track & control infra costs
    

    View Slide

21. Case Study
    21
    (Traditional DC + VMs + JVM)
    AWS + Kubernetes + JVM
    ELK for logs and transaction traces
    [ ]
    

    View Slide

22. Case Study
    Scale existing approach
    to 7x size
    22
    

    View Slide

23. Case Study
    Spread awareness of
    effective practices
    23
    

    View Slide

24. Case Study
    Cross internal and
    external boundaries
    24
    

    View Slide

25. Case Study
    “Continuous Delivery”
    25
    

    View Slide

26. Case Study
    Avoid “one-size-fits-all”
    26
    

    View Slide

27. Case Study
    Define the Platform
    27
    

    View Slide

28. Case Study
    Improve logging quality
    28
    

    View Slide

29. Case Study
    Multi-supplier delivery
    model
    29
    

    View Slide

30. What do we aim for in the
    context of fast flow?
    30
    

    View Slide

31. 31
    Continuously ‘untangle’
    business concepts
    

    View Slide

32. 32
    Find and adjust team &
    system boundaries for flow
    

    View Slide

33. 33
    Minimize hand-offs
    

    View Slide

34. 34
    Avoid blocking
    dependencies
    

    View Slide

35. 35
    Move some decision-making
    to teams
    

    View Slide

36. 36
    ➔ Untangle business concepts
    ➔ Adjust boundaries for flow
    ➔ Minimize hand-offs
    ➔ Remove blocking dependencies
    ➔ Move decision-making to teams
    

    View Slide

37. 37
    Example: deployment pipeline
    architecture
    

    View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. 42
    The architecture of
    deployment pipelines is
    a key consideration for
    fast flow
    

    View Slide

43. 43
    “Show me your
    deployment pipelines and
    I will predict your team
    boundaries” 💥
    

    View Slide

44. Guiding principles for
    fast flow
    45
    

    View Slide

45. 46
    Reliable Software Releases
    Through Build, Test, and
    Deployment Automation
    Jez Humble & Dave Farley,
    2010
    Continuous
    Delivery
    

    View Slide

46. Continuous Delivery with
    Windows and .NET
    47
    Matthew Skelton & Chris O’Dell
    O’Reilly, 2016
    cdwithwindows.net
    ** Now a key text for MSc in
    Software Engineering at UCL **
    

    View Slide

47. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    48
    

    View Slide

48. 49
    Re-aligned
    architecture
    

    View Slide

49. 50
    Fast feedback via
    deployment pipelines
    

    View Slide

50. 51
    Good technical
    practices (TDD, …)
    

    View Slide

51. 52
    Team ownership of
    software & services
    

    View Slide

52. 53
    Configuration in
    version control (Git)
    

    View Slide

53. 54
    Cloud-native:
    transparent in
    operation
    

    View Slide

54. 55
    Cloud-native:
    designed for
    automation
    

    View Slide

55. 56
    Continuous
    testing
    performance
    scanning
    deployment
    monitoring
    right-sizing
    integration
    

    View Slide

56. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    57
    

    View Slide

57. What we did to help fast flow
    58
    

    View Slide

58. Case Study
    New flow-based
    operating model
    59
    

    View Slide

59. 60
    Flow of change
    

    View Slide

60. Case Study
    Focus on the
    "operator experience"
    61
    

    View Slide

61. Case Study
    Define the platform
    62
    

    View Slide

62. Case Study
    Engineering Guild
    +
    Engineering Working
    Group
    64
    

    View Slide

63. Case Study
    Weekly lunchtime tech
    talks
    65
    

    View Slide

64. Case Study
    ‘Continuous Delivery for
    people who don’t write
    code’
    Slides:
    https://www.slideshare.net/matthewskelton/continuous-delivery-for-p
    eople-who-do-not-write-code-matthew-skelton-conflux
    66
    

    View Slide

65. Case Study
    Team-first techniques
    Example:
    Run Book Dialogue Sheet
    67
    

    View Slide

66. 68
    runbooktemplate.info
    Run Book dialogue sheets
    

    View Slide

67. Case Study
    Engineering Standards
    69
    

    View Slide

68. View Slide

69. Engineering Standards
    71
    Minimal but important constraints
    Dynamic for reviewing and updating
    Emphasise team engagement
    

    View Slide

70. Case Study
    Engineering Assessments
    72
    

    View Slide

71. Engineering Assessments
    73
    6 dimensions [in 2018]:
    Team Health, Deployment, Continuous
    Delivery, Flow, Operability, Testing
    SoftwareDeliveryAssessment.com
    

    View Slide

72. Engineering Assessments
    74
    Criteria taken from existing sources:
    Spotify Team Health Check, DevOps for the Modern
    Enterprise, Continuous Delivery, Accelerate, Team
    Guide to Software Operability, Team Guide to
    Software Testability
    

    View Slide

73. View Slide

74. View Slide

75. View Slide

76. Engineering Assessments
    78
    Team self-assessment
    Facilitated to guide discussions
    “Viral” facilitation model
    Results collated across teams
    

    View Slide

77. View Slide

78. 80
    SoftwareDeliveryAssessment.com
    

    View Slide

79. Many of these principles
    and practices appear in the
    Team Topologies book 💡
    82
    

    View Slide

80. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    83
    

    View Slide

81. Results and key takeaways
    84
    

    View Slide

82. Case Study
    85
    Software releases
    From 2 successful release candidate
    builds per week to 8 builds per day
    Multiple, independent routes to live
    

    View Slide

83. Case Study
    86
    Operability
    Ops teams love the new "operator
    experience" with logging &
    dashboards
    No major operational problems with
    new releases
    

    View Slide

84. Case Study
    87
    Collaboration
    Logging beginning to be seen as a
    key observability tool
    Better cross-team awareness of how
    systems work through telemetry
    

    View Slide

85. Case Study
    Moving towards
    Continuous Delivery at
    DDaT
    See public blog post at
    https://hodigital.blog.gov.uk/2018/08/10/moving-tow
    ards-continuous-delivery-in-ddat/
    88
    

    View Slide

86. Case Study
    89
    (Many other people were
    also involved)
    

    View Slide

87. Case Study
    90
    Ideas and patterns from
    Team Topologies now
    being used in this
    department
    

    View Slide

88. What social and technical
    practices are needed for fast
    flow at scale?
    91
    

    View Slide

89. 92
    Make space for learning/sharing
    Promote good work
    Help to develop skills in speaking
    Lunchtime talks (food)
    Interplay of working group + guild
    

    View Slide

90. 93
    Define the Platform
    Service Level Expectations
    Who owns what? What's missing?
    Use DevEx to drive improvements
    Thinnest Viable Platform
    

    View Slide

91. 94
    Address operability early on
    Operational aspects are also features
    Add operational checks into the
    deployment pipeline
    Team-first: RunBookTemplate.info
    

    View Slide

92. 95
    Good logging is foundational
    Use a well-defined event space
    Searchable EventIDs increase context
    Enables collaboration between teams
    Invest in logging infra (Saas/people/&c.)
    

    View Slide

93. 96
    Empower teams to improve
    Co-create the engineering standards
    Look for organisational blockers
    “No-blame” is paramount
    SoftwareDeliveryAssessment.com
    

    View Slide

94. 98
    The architecture of
    deployment pipelines is
    a key consideration for
    fast flow
    

    View Slide

95. Team Topologies
    Organizing business and
    technology teams for fast flow
    Matthew Skelton & Manuel Pais
    IT Revolution Press, September 2019
    Order via stores worldwide:
    teamtopologies.com/book
    100
    

    View Slide

96. expert-led
    group learning
    ❏ P602 - Software Delivery
    Assessment
    ❏ P616 - software reliability for
    product managers
    ❏ P630 - Find possible streams
    with Independent Service
    Heuristics
    ❏ P634 - Peer Learning - Writing
    Tech Talks
    ❏ P641f - Preparing to adopt
    Team Topologies
    ❏ P641i - Skills paths for fast
    flow - leaders and HR
    101
    online workshops for up to
    20 people
    

    View Slide

97. 102
    expert-led group learning for adopting
    fast flow and Team Topologies
    confluxhq.com
    

    View Slide

98. thank you
    confluxhq.com
    Copyright (c) 2017-2023 Conflux group of companies. All Rights Reserved.
    The name “Conflux” and the filled C device are Registered Trademarks ® in multiple jurisdictions.
    103
    

    View Slide