Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Remote Scaled Trunk-Based Development (IPC)

Richard
October 25, 2022

Remote Scaled Trunk-Based Development (IPC)

Held at IPC Fall 2022
-----
How do we coordinate teams of 20 or more developers and achieve consistent quality in a remote setting? Plenty scaling methods have been proposed but these models are usually based on Scrum and leave out the technical practices needed to achieve quality at scale.

In this talk we‘ll explore how my project grew from a local six person scrum team to a remote 20 person continuous flow team and beyond. We‘ll take a look at the iterations we needed until we achieved our flow, how we implemented Xp practices like trunk-based development and what practices we adopted in our all-remote setting.

By the end I hope to have provided you with all the information you need to achieve quality at scale in a remote setting.

---

Talk held at https://phpconference.com/mixed/from-the-trenches-remote-scaled-trunk-based-development/

Richard

October 25, 2022
Tweet

More Decks by Richard

Other Decks in Programming

Transcript

  1. Remote Scaled Trunk-
    Based Development*
    25.10.22
    * I hope I did not forget any buzzwords. Richard Gross (he/him)
    Coder, Archaeologist, Auditor.
    Works at maibornwolff.de/
    richargh.de/
    speakerdeck.com/richargh
    @arghrich

    View full-size slide

  2. Slides by @arghrich
    Why do Big Software Projects
    fail?
    2

    View full-size slide

  3. Slides by @arghrich
    Big Projects fail because they are Big?
    3
    CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
    HBR 2011 https://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think
    McK https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value
    McK On average, large IT projects run
    45 percent over budget and 7 percent
    over time, while delivering 56
    percent less value than predicted.
    HBR one in six of the projects we
    studied was a black swan, with a
    cost overrun of 200%, on average,
    and a schedule overrun of almost
    70%.
    CHAOS‘15 size (is) the single most
    important factor in the (…) outcome.

    View full-size slide

  4. Slides by @arghrich
    How many Software Projects
    fail at your company?
    4
    IT Failure Rates— 70% or 10–15%?
    https://www.cs.uni.edu/~wallingf/teaching/172/resources/glass-on-it-failure.pdf
    If the reasearch is right it must be 50 to 70%...

    View full-size slide

  5. Slides by @arghrich
    CHAOS Report is highly controversial
    5
    CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
    IEEE https://www.cs.vu.nl/~x/the_rise_and_fall_of_the_chaos_report_figures.pdf
    AMB https://www.informationweek.com/executive-insights-and-innovation/the-non-existent-software-crisis-debunking-the-chaos-report
    AMB http://www.ambysoft.com/surveys/
    IEEE (the definitions are) solely
    based on estimation accuracy of
    cost, time, and functionality.
    AMB (independent study results) are
    much more positive than what the
    Standish Group claims.
    IEEE Our research shows that the
    Standish definitions (…) have 4
    major problems: they’re
    misleading, one-sided, pervert the
    estimation practice, and result in
    meaningless figures.
    CHAOS‘15 (in 2015) another major
    change is how we define success. Our
    new Modern definition is OnTime,
    OnBudget, with a satisfactory result.

    View full-size slide

  6. Slides by @arghrich
    Think big, act small
    6
    CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
    CHAOS is in agreement with how to succeed
    the secret to project success is to
    strongly recommend and enforce
    limits on size and complexity.
    These two factors trump all others
    factors.
    break down large projects into a
    sequence of smaller ones, prioritized
    on direct business value (…)
    Aka scale down the work

    View full-size slide

  7. Slides by @arghrich
    Success Factors for small projects
    7
    CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf

    View full-size slide

  8. Slides by @arghrich
    So why do Experts think Big
    Software Projects actually fail?
    8

    View full-size slide

  9. Slides by @arghrich
    Politics
    9
    Highly abbreviated from a short Twitter conversation.

    View full-size slide

  10. Slides by @arghrich
    Succeeding with a Scaled
    Project*
    What we tried and avoided
    10
    * See if you can scale down the project first and deliver much smaller slices sooner

    View full-size slide

  11. Slides by @arghrich
    A customer approaches* …
    11
    * pre-Pandemic
    I need 4 teams

    View full-size slide

  12. Slides by @arghrich
    Complexity is …
    Theory
    13
    Power
    Sexy
    Easier

    View full-size slide

  13. Slides by @arghrich
    Scale Software via Evolutionary Design
    14 Theory
    https://www.industriallogic.com/blog/evolutionary-design/
    You can recognize the guitar
    Simple, easy to understand, makes music
    Can do more
    Still simple. Maybe that's enough for the
    customer. After all, time is money.
    This is a guitar
    Pretty good. Satisfies most people.
    Complex guitar
    Makes the best of the best happy. BUT,
    did we really have to serve them? How
    many normaler users did we lose with
    this complicated guitar?

    View full-size slide

  14. Slides by @arghrich
    Scale Teams by starting small
    15 Theory
    Hexagonal Architecture
    Bind Api to Database
    Clean Code is …
    I don’t like the way you write tests
    This is not good design
    Java!
    C#
    Spaces
    Tabs
    F#
    One Repo per Deployable Singe Repo
    Rust
    Should we use …

    View full-size slide

  15. Slides by @arghrich
    Fresh Teams have a lot …
    Theory
    16
    to discover
    of beef
    to agree on

    View full-size slide

  16. Slides by @arghrich
    Scale Teams via Evolutionary Division
    17 Theory
    Spring
    1 American Pizza
    Summer
    2 American Pizzas
    Winter
    4 American Pizzas
    Autumn
    3 American Pizzas

    View full-size slide

  17. Slides by @arghrich
    18
    Photo by Anne-sophie Parent from Pexels
    Slides by @arghrich
    Spring > Foundation

    View full-size slide

  18. Slides by @arghrich
    Try…1 Co-located team
    Spring > Foundation
    19
    Get to know each other

    View full-size slide

  19. Slides by @arghrich
    Try…Adopt a Process to Improve
    20 Spring > Foundation
    Retro
    2h Bi-Weekly
    • What to KEEP
    • What to TRY
    • Results often added to
    Team Charter
    Dev Talk
    1h Weekly
    • Present, Discuss,
    Agree on Tech
    • Results often ADRs
    • So many ADRs
    Slack Time

    View full-size slide

  20. Slides by @arghrich
    Try…Adopt and adapt Scrum1
    21 Winter > Scaled Remote
    1 If you mix it with Xp, Scrum is not a bad start at all.
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team

    View full-size slide

  21. Slides by @arghrich
    Try…Explicit1,2,3 Slack Time
    Spring > Foundation
    22
    1 Customers are super happy when tech stuff never shows up in their backlog. They don’t know how to prioritize it.
    2 Given devs ~2 days of slack time each sprint is a small price to pay when it stabilizes all estimation and planning.
    3 The backlog is also only about business features…
    To actually do Retro/Tech Items

    View full-size slide

  22. Slides by @arghrich
    Try…Trunk-Based
    Development1
    Spring > Foundation
    23
    1 Predictive of success à Accelerate State of DevOps 2021 https://cloud.google.com/resources/state-of-devops#section-4
    The simplest, nicest branching model

    View full-size slide

  23. Slides by @arghrich
    Avoid…Long-Lived Feature-Branches
    24 Spring > Foundation
    main

    View full-size slide

  24. Slides by @arghrich
    Avoid…Continuous
    Isolation
    Branches
    • Increase personal productivity
    • Discourage Refactoring
    • Increase Batch Size
    • Increase Risk
    Daily merges (aka integration)
    • Increase team productivity
    • Motivate Refactoring
    • Decrease Batch Size
    • Decrease Risk
    25 Spring > Foundation
    See also Feature Branches are Evil by Thierry de Pauw
    https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html
    https://thinkinglabs.io/articles/2022/09/17/the-practices-that-make-continuous-integration-team-working.html
    Try…Continuous
    Integration

    View full-size slide

  25. Slides by @arghrich
    Do…Pair TBD with maintained pipeline
    26 Spring > Foundation
    Behavior-changing, validated feature = upper-case F
    Behavior-preserving, safe refactor = lower-case r
    https://github.com/RefactoringCombos/ArlosCommitNotation
    main
    r r r r r F r r revert r
    Improve
    commit
    Always in a
    releasable state
    ▸ Lint
    ▸ Unit Test
    ▸ Int Test
    ▸ System Test
    ▸ PostDeploy
    Smoke Test
    ▸ PostDeploy
    Monitoring
    ▸ …
    Revert, if not
    fixable in 10min
    Small, focused
    commits

    View full-size slide

  26. Slides by @arghrich
    Our customer approaches again* …
    28 Late Spring
    * Still pre-Pandemic
    I want 4 teams
    I want scalable software

    View full-size slide

  27. Slides by @arghrich
    29 Summer
    Photo by Pixabay from Pexels
    Slides by @arghrich

    View full-size slide

  28. Slides by @arghrich
    Try…Start with a monolith
    • At project inception you have the least knowledge
    • Your modelling can and will change a lot in the beginning
    • Refactoring in a monolith has tool support
    • However: splitting up a monolith is not a freebie either
    30 Summer > Scale-Out

    View full-size slide

  29. Slides by @arghrich
    Try…Event-Driven Communication
    31 Summer > Scale-Out
    Nothing to do with TBD, Remote or Scaled, just a cool approach for service integration.
    Service
    1
    Service

    Service
    n
    Kafka
    (RabbitMq etc. work nicely as well)

    View full-size slide

  30. Slides by @arghrich
    Try…Keep everything in one repo
    32 Summer > Scale-Out
    • All
    • Services
    • Shared Libraries
    • Documentation
    • Configuration
    • Provisioning
    • Onboarding.sh Scripts
    à Everything your team needs
    • You see everything
    everywhere, all at once
    • You filter what you need right
    now
    • Your (refactoring) tools help
    you with every change (in
    shared libraries)
    • You can unit test all services
    together J

    View full-size slide

  31. Slides by @arghrich
    Our customer approaches again again* …
    34 Late Summer
    * Still pre-Pandemic
    I want 4 teams
    I want scaled a software
    I want more, in parallel

    View full-size slide

  32. Slides by @arghrich
    35 Autumn
    Photo by Daniel Frank from Pexels
    Slides by @arghrich

    View full-size slide

  33. Slides by @arghrich
    Try…Dynamic Feature Teams1
    36 Autumn > Divide
    1 A more extreme version of this approach is FAST agile https://www.fastagile.io/method
    See also Hands-on Agile 45: FAST Scaling — James Shore [Youtube]
    Planning 1 Planning 1,5
    Topic A
    Topic B
    Present topics incl.
    Backlog
    Pick
    Topic
    Topic A
    Topic B
    Planning 2
    Task Breakdown

    View full-size slide

  34. Slides by @arghrich
    Try…Dynamic Feature Teams
    Each team can work anywhere in the repo
    37 Autumn > Divide
    Align via … the coffee machine
    the dev talk and ADRs
    Bi-weekly re-teaming

    View full-size slide

  35. Slides by @arghrich
    Our customer approaches remote* …
    40 Late Autumn
    *Pandemic!
    I want 4 teams
    I want scaled a software
    I want more, in parallel
    I want more speed I hate remote

    View full-size slide

  36. Slides by @arghrich
    41 Winter
    Photo by Riccardo Bresciani from Pexels
    Slides by @arghrich

    View full-size slide

  37. Slides by @arghrich
    Try…Divide & Conquer to increase
    Meeting Efficiency
    42 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team

    View full-size slide

  38. Slides by @arghrich
    Try…Stable Streams
    43 Winter > Scaled Remote
    Inception
    Pick
    Stream
    You already know who you can work with best.

    View full-size slide

  39. Slides by @arghrich
    Try…Stable What’s Next
    44 Winter > Scaled Remote
    Backlog

    View full-size slide

  40. Slides by @arghrich
    Try…Stable What’s Next1,2
    45 Winter > Scaled Remote
    1 Stream Backlogs should never run dry
    2 At this point you know your velocity to forecast feature completion dates
    What’s Next
    Backlog
    Task Breakdown
    Stream
    Backlogs

    View full-size slide

  41. Slides by @arghrich
    Try…Flexible Cadences
    46 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    Refinement
    30m | 3 per Week | 3 Ppl
    What’s Next
    1h | Stream
    Daily
    Retro
    20m | Stream
    Daily
    15m | Stream
    During Week Weekly Bi-Monthly
    Roadmap
    1h | Team
    Incl. Story Map
    On-Demand
    Retro
    2,3h | Stream or Team

    View full-size slide

  42. Slides by @arghrich
    Try…Response Team
    47 Winter > Scaled Remote
    Motivation
    • Reduce context switches
    Activities
    1. Keep Prod Stable
    2. Keep Dev Env Stable
    3. Proactively work on even more stability
    4. In that order

    View full-size slide

  43. Slides by @arghrich
    How do you find people that
    can help/enlighten/chat in a
    remote setting? 1,2
    Winter > Scaled Remote
    48
    1 Can meaning, they are not in focus mode.
    2 In a co-located setting you can easily see if someone does not want to be disturbed.

    View full-size slide

  44. Slides by @arghrich
    Try…Findable Remote Stream Rooms
    49 Winter > Scaled Remote
    Channel A
    We work on feature xyz
    Channel B
    We work on feature abc
    Channel Discovery
    We are discovering
    • Use team-internal channels/calendar events/etc. where the
    streams work
    Now one there or
    all in focus mode.
    Someone is there,
    working on a feature
    Someone is there, working on
    the next features

    View full-size slide

  45. Slides by @arghrich
    50 Summary
    Photo by Irina Iriser from Pexels
    Slides by @arghrich

    View full-size slide

  46. Slides by @arghrich
    How far we got
    • 1 team with 6 devs
    • 2 week Sprints
    • 2h Planning
    • Dev Talk Decisions
    • Whole-team 2h retros every 2
    weeks
    • 1 team, 5 streams with 2-0 devs
    • Continuous
    • 1h What’s next meetings
    • Stream decisions
    + Dev Talk presentation
    • Daily 15min retros, On-demand
    large stream or team retros
    51 Summary

    View full-size slide

  47. Slides by @arghrich
    People issues determine a project’s speed
    Can they easily detect something needs attention?
    Will they care enough to do something about it?
    Can they effectively pass along the information?
    52 Summary
    Stolen from Alistair Cockburn „Why Agile Works“ at ~28m

    View full-size slide

  48. Slides by @arghrich
    Do you have a
    Process to
    Experiment
    and Improve?
    Summary
    53

    View full-size slide

  49. Slides by @arghrich
    Do you have
    the Mindset?
    Summary
    54

    View full-size slide

  50. Slides by @arghrich
    Try…Remote Trunk-Based Development
    Just don’t forget it’s not about the techniques
    55 Summary
    Pipeline
    Easy Hard
    Techniques
    Something you do
    Methods
    When you do it
    Mindset
    What you actually do

    View full-size slide

  51. Slides by @arghrich
    Thanks
    56
    Richard Gross (he/him)
    Coder, Archaeologist, Auditor.
    Works at maibornwolff.de/
    richargh.de/
    speakerdeck.com/richargh
    @arghrich

    View full-size slide

  52. Slides by @arghrich
    Backup
    57

    View full-size slide

  53. Slides by @arghrich
    58
    Twitter https://twitter.com/ArghRich/status/1569009015123808258 ?cxt=HHwWhMC4vdzYnsYrAAAA

    View full-size slide