Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Remote Scaled Trunk-Based Development (Build Stuff)

Richard
November 08, 2022

Remote Scaled Trunk-Based Development (Build Stuff)

Held at Build Stuff 2022
----
How do we coordinate teams of 20 or more developers and achieve consistent quality in a remote setting? Plenty scaling methods have been proposed but these models are usually based on Scrum and leave out the technical practices needed to achieve quality at scale.

In this talk we‘ll explore how my project grew from a local six person scrum team to a remote 20 person continuous flow team and beyond. We‘ll take a look at the iterations we needed until we achieved our flow, how we implemented Xp practices like trunk-based development and what practices we adopted in our all-remote setting.

By the end I hope to have provided you with all the information you need to achieve quality at scale in a remote setting.

Richard

November 08, 2022
Tweet

More Decks by Richard

Other Decks in Programming

Transcript

  1. Remote Scaled Trunk-
    Based Development*
    09.11.22
    * I hope I did not forget any buzzwords.
    IT-Tailor
    Richard Gross (he/him)
    Archeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works at maibornwolff.de/

    View Slide

  2. Slides by @arghrich
    Why do Big Software Projects
    fail?
    2

    View Slide

  3. Slides by @arghrich
    Big Projects fail because they are Big?
    3
    CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
    HBR 2011 https://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think
    McK https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value
    McK On average, large IT projects run
    45 percent over budget and 7 percent
    over time, while delivering 56
    percent less value than predicted.
    HBR one in six of the projects we
    studied was a black swan, with a
    cost overrun of 200%, on average,
    and a schedule overrun of almost
    70%.
    CHAOS‘15 size (is) the single most
    important factor in the (…) outcome.

    View Slide

  4. Slides by @arghrich
    How many Software Projects
    fail at your company?
    4
    IT Failure Rates— 70% or 10–15%?
    https://www.cs.uni.edu/~wallingf/teaching/172/resources/glass-on-it-failure.pdf
    If the reasearch is right it must be 50 to 70%...

    View Slide

  5. Slides by @arghrich
    CHAOS Report is highly controversial
    5
    CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
    IEEE https://www.cs.vu.nl/~x/the_rise_and_fall_of_the_chaos_report_figures.pdf
    AMB https://www.informationweek.com/executive-insights-and-innovation/the-non-existent-software-crisis-debunking-the-chaos-report
    AMB http://www.ambysoft.com/surveys/
    IEEE (the definitions are) solely
    based on estimation accuracy of
    cost, time, and functionality.
    AMB (independent study results) are
    much more positive than what the
    Standish Group claims.
    IEEE Our research shows that the
    Standish definitions (…) have 4
    major problems: they’re
    misleading, one-sided, pervert the
    estimation practice, and result in
    meaningless figures.
    CHAOS‘15 (in 2015) another major
    change is how we define success. Our
    new Modern definition is OnTime,
    OnBudget, with a satisfactory result.

    View Slide

  6. Slides by @arghrich
    Think big, act small
    6
    CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
    CHAOS is in agreement with how to succeed
    the secret to project success is to
    strongly recommend and enforce
    limits on size and complexity.
    These two factors trump all others
    factors.
    break down large projects into a
    sequence of smaller ones, prioritized
    on direct business value (…)
    Aka scale down the work

    View Slide

  7. Slides by @arghrich
    Success Factors for small projects
    7
    CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf

    View Slide

  8. Slides by @arghrich
    So why do Experts think Big
    Software Projects actually fail?
    8

    View Slide

  9. Slides by @arghrich
    Politics
    9
    Highly abbreviated from a short Twitter conversation with @TotherAlistair.

    View Slide

  10. Slides by @arghrich
    Succeeding with a Scaled
    Project*
    What we tried and avoided
    10
    * See if you can scale down the project first and deliver much smaller slices sooner

    View Slide

  11. Slides by @arghrich
    A customer approaches* …
    11
    * pre-Pandemic
    I need 4 teams

    View Slide

  12. Oh Boy

    View Slide

  13. Slides by @arghrich
    Complexity is …
    Theory
    13
    Power
    Sexy
    Easier

    View Slide

  14. Slides by @arghrich
    Scale Software via Evolutionary Design
    14 Theory
    https://www.industriallogic.com/blog/evolutionary-design/
    You can recognize the guitar
    Simple, easy to understand, makes music
    Can do more
    Still simple. Maybe that's enough for the
    customer. After all, time is money.
    This is a guitar
    Pretty good. Satisfies most people.
    Complex guitar
    Makes the best of the best happy. BUT,
    did we really have to serve them? How
    many normaler users did we lose with
    this complicated guitar?

    View Slide

  15. Slides by @arghrich
    Scale Teams by starting small
    15 Theory
    Hexagonal Architecture
    Bind Api to Database
    Clean Code is …
    I don’t like the way you write tests
    This is not good design
    Java!
    C#
    Spaces
    Tabs
    F#
    One Repo per Deployable Singe Repo
    Rust
    Should we use …

    View Slide

  16. Slides by @arghrich
    Fresh Teams have a lot …
    Theory
    16
    to discover
    of beef
    to agree on

    View Slide

  17. Slides by @arghrich
    Scale Teams via Evolutionary Division
    17 Theory
    Spring
    1 American Pizza
    Summer
    2 American Pizzas
    Winter
    4 American Pizzas
    Autumn
    3 American Pizzas

    View Slide

  18. Slides by @arghrich
    18
    Photo by Anne-sophie Parent from Pexels
    Slides by @arghrich
    Spring > Foundation

    View Slide

  19. Slides by @arghrich
    Try…1 Co-located team
    Spring > Foundation
    19
    Get to know each other

    View Slide

  20. Slides by @arghrich
    Try…Adopt a Process to Improve
    20 Spring > Foundation
    Retro
    2h Bi-Weekly
    • What to KEEP
    • What to TRY
    • Results often added to
    Team Charter
    Dev Talk
    1h Weekly
    • Present, Discuss,
    Agree on Tech
    • Results often ADRs
    • So many ADRs
    Slack Time

    View Slide

  21. Slides by @arghrich
    Try…Adopt and adapt Scrum1
    21 Winter > Scaled Remote
    1 If you mix it with Xp, Scrum is not a bad start at all.
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team

    View Slide

  22. Slides by @arghrich
    Try…Explicit1,2,3 Slack Time
    Spring > Foundation
    22
    1 Customers are super happy when tech stuff never shows up in their backlog. They don’t know how to prioritize it.
    2 Given devs ~2 days of slack time each sprint is a small price to pay when it stabilizes all estimation and planning.
    3 The backlog is also only about business features…
    To actually do Retro/Tech Items

    View Slide

  23. Slides by @arghrich
    Try…Trunk-Based
    Development1
    Spring > Foundation
    23
    1 Predictive of success à Accelerate State of DevOps 2021 https://cloud.google.com/resources/state-of-devops#section-4
    The simplest, nicest branching model

    View Slide

  24. Slides by @arghrich
    Avoid…Long-Lived Feature-Branches
    24 Spring > Foundation
    main

    View Slide

  25. Slides by @arghrich
    Avoid…Continuous
    Isolation
    Branches
    • Increase personal productivity
    • Discourage Refactoring
    • Increase Batch Size
    • Increase Risk
    Daily merges (aka integration)
    • Increase team productivity
    • Motivate Refactoring
    • Decrease Batch Size
    • Decrease Risk
    25 Spring > Foundation
    See also Feature Branches are Evil by Thierry de Pauw
    https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html
    https://thinkinglabs.io/articles/2022/09/17/the-practices-that-make-continuous-integration-team-working.html
    Try…Continuous
    Integration

    View Slide

  26. Slides by @arghrich
    Do…Pair TBD with maintained pipeline
    26 Spring > Foundation
    Behavior-changing, validated feature = upper-case F
    Behavior-preserving, safe refactor = lower-case r
    https://github.com/RefactoringCombos/ArlosCommitNotation
    main
    r r r r r F r r revert r
    Improve
    commit
    Always in a
    releasable state
    ▸ Lint
    ▸ Unit Test
    ▸ Int Test
    ▸ System Test
    ▸ PostDeploy
    Smoke Test
    ▸ PostDeploy
    Monitoring
    ▸ …
    Revert, if not
    fixable in 10min
    Small, focused
    commits

    View Slide

  27. Slides by @arghrich
    Our customer approaches again* …
    27 Late Spring
    * Still pre-Pandemic
    I want 4 teams
    I want scalable software

    View Slide

  28. Slides by @arghrich
    28 Summer
    Photo by Pixabay from Pexels
    Slides by @arghrich

    View Slide

  29. Slides by @arghrich
    Try…Start with a monolith
    • At project inception you have the least knowledge
    • Your modelling can and will change a lot in the beginning
    • Refactoring in a monolith has tool support
    • However: splitting up a monolith is not a freebie either
    29 Summer > Scale-Out

    View Slide

  30. Slides by @arghrich
    Try…Keep everything in one repo
    31 Summer > Scale-Out
    • All
    • Services
    • Shared Libraries
    • Documentation
    • Configuration
    • Provisioning
    • Onboarding.sh Scripts
    à Everything your team needs
    • You see everything
    everywhere, all at once
    • You filter what you need right
    now
    • Your (refactoring) tools help
    you with every change (in
    shared libraries)
    • You can unit test all services
    together J

    View Slide

  31. Slides by @arghrich
    Our customer approaches again again* …
    33 Late Summer
    * Still pre-Pandemic
    I want 4 teams
    I want scaledable software
    I want more, in parallel

    View Slide

  32. Slides by @arghrich
    34 Autumn
    Photo by Daniel Frank from Pexels
    Slides by @arghrich

    View Slide

  33. Slides by @arghrich
    Try…Dynamic Feature Teams1
    35 Autumn > Divide
    1 A more extreme version of this approach is FAST agile https://www.fastagile.io/method
    See also Hands-on Agile 45: FAST Scaling — James Shore [Youtube]
    Planning 1 Planning 1,5
    Topic A
    Topic B
    Present topics incl.
    Backlog
    Pick
    Topic
    Topic A
    Topic B
    Planning 2
    Task Breakdown

    View Slide

  34. Slides by @arghrich
    Try…Dynamic Feature Teams
    Each team can work anywhere in the repo
    36 Autumn > Divide
    Align via … the coffee machine
    the dev talk and ADRs
    Bi-weekly re-teaming

    View Slide

  35. Slides by @arghrich
    Our customer approaches remote* …
    38 Late Autumn
    *Pandemic!
    I want 4 teams
    I want scaledable software
    I want more, in parallel
    I want more speed I hate remote

    View Slide

  36. Slides by @arghrich
    39 Winter
    Photo by Riccardo Bresciani from Pexels
    Slides by @arghrich

    View Slide

  37. Slides by @arghrich
    Try…Divide & Conquer to increase
    Meeting Efficiency
    40 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team

    View Slide

  38. Slides by @arghrich
    Try…Stable Streams
    41 Winter > Scaled Remote
    Inception
    Pick
    Stream
    You already know who you can work with best.

    View Slide

  39. Slides by @arghrich
    Try…One Shared Backlog
    42 Winter > Scaled Remote
    Backlog

    View Slide

  40. Slides by @arghrich
    Try…Stable What’s Next1,2
    43 Winter > Scaled Remote
    1 Stream Backlogs should never run dry
    2 At this point you know your velocity to forecast feature completion dates
    What’s Next
    Backlog
    Task Breakdown
    Stream
    Backlogs

    View Slide

  41. Slides by @arghrich
    Try…Flexible Cadences
    44 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    Refinement
    30m | 3 per Week | 3 Ppl
    What’s Next
    1h | Stream
    Daily
    Retro
    20m | Stream
    Daily
    15m | Stream
    During Week Weekly Bi-Monthly
    Roadmap
    1h | Team
    Incl. Story Map
    On-Demand
    Retro
    2,3h | Stream or Team

    View Slide

  42. Slides by @arghrich
    Try…Response Team
    45 Winter > Scaled Remote
    Motivation
    • Reduce context switches
    Activities
    1. Keep Prod Stable
    2. Keep Dev Env Stable
    3. Proactively work on even more stability
    4. In that order

    View Slide

  43. Slides by @arghrich
    How do you find people that
    can help/enlighten/chat in a
    remote setting? 1,2
    Winter > Scaled Remote
    46
    1 Can meaning, they are not in focus mode.
    2 In a co-located setting you can easily see if someone does not want to be disturbed.

    View Slide

  44. Slides by @arghrich
    Try…Findable Remote Stream Rooms
    47 Winter > Scaled Remote
    Channel A
    We work on feature xyz
    Channel B
    We work on feature abc
    Channel Discovery
    We are discovering
    • Use team-internal channels/calendar events/etc. where the
    streams work
    No one there or all
    in focus mode.
    Someone is there,
    working on a feature
    Someone is there, working on
    the next features

    View Slide

  45. Slides by @arghrich
    48 Summary
    Photo by Irina Iriser from Pexels
    Slides by @arghrich

    View Slide

  46. Slides by @arghrich
    How far we got
    • 1 team with 6 devs
    • 2 week Sprints
    • 2h Planning
    • Dev Talk Decisions
    • Whole-team 2h retros every 2
    weeks
    • 1 team, 5 streams with 2-0 devs
    • Continuous
    • 1h What’s next meetings
    • Stream decisions
    + Dev Talk presentation
    • Daily 15min retros, On-demand
    large stream or team retros
    49 Summary

    View Slide

  47. Slides by @arghrich
    We improved as a team
    Spring > Foundation
    50

    View Slide

  48. Slides by @arghrich
    People issues determine a project’s speed
    Can they easily detect something needs attention?
    Will they care enough to do something about it?
    Can they effectively pass along the information?
    51 Summary
    Stolen from Alistair Cockburn „Why Agile Works“ at ~28m

    View Slide

  49. Slides by @arghrich
    Do you have a
    Process to
    Experiment
    and Improve?
    Summary
    52

    View Slide

  50. Slides by @arghrich
    Do you have
    the Mindset?
    Summary
    53

    View Slide

  51. Slides by @arghrich
    Try…Remote Trunk-Based Development
    Just don’t forget it’s not about the techniques
    54 Summary
    Pipeline
    Easy Hard
    Techniques
    Something you do
    Methods
    When you do it
    Mindset
    What you actually do

    View Slide

  52. Slides by @arghrich
    Thanks
    55
    IT-Tailor
    Richard Gross (he/him)
    Archaeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works at maibornwolff.de/
    Tailored Software by People
    who Care.
    DE TN ES

    View Slide

  53. Slides by @arghrich
    Backup
    56

    View Slide

  54. Slides by @arghrich
    57
    Twitter https://twitter.com/ArghRich/status/1569009015123808258 ?cxt=HHwWhMC4vdzYnsYrAAAA

    View Slide