Remote Scaled Trunk-Based Development (Build Stuff)

Held at Build Stuff 2022
----
How do we coordinate teams of 20 or more developers and achieve consistent quality in a remote setting? Plenty scaling methods have been proposed but these models are usually based on Scrum and leave out the technical practices needed to achieve quality at scale.

In this talk we‘ll explore how my project grew from a local six person scrum team to a remote 20 person continuous flow team and beyond. We‘ll take a look at the iterations we needed until we achieved our flow, how we implemented Xp practices like trunk-based development and what practices we adopted in our all-remote setting.

By the end I hope to have provided you with all the information you need to achieve quality at scale in a remote setting.

Transcript

1. Remote Scaled Trunk-
   Based Development*
   09.11.22
   * I hope I did not forget any buzzwords.
   IT-Tailor
   Richard Gross (he/him)
   Archeologist Auditor
   richargh.de/
   speakerdeck.com/richargh
   @arghrich
   Works at maibornwolff.de/
   

   View Slide

2. Slides by @arghrich
   Why do Big Software Projects
   fail?
   2
   

   View Slide

3. Slides by @arghrich
   Big Projects fail because they are Big?
   3
   CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
   HBR 2011 https://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think
   McK https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value
   McK On average, large IT projects run
   45 percent over budget and 7 percent
   over time, while delivering 56
   percent less value than predicted.
   HBR one in six of the projects we
   studied was a black swan, with a
   cost overrun of 200%, on average,
   and a schedule overrun of almost
   70%.
   CHAOS‘15 size (is) the single most
   important factor in the (…) outcome.
   

   View Slide

4. Slides by @arghrich
   How many Software Projects
   fail at your company?
   4
   IT Failure Rates— 70% or 10–15%?
   https://www.cs.uni.edu/~wallingf/teaching/172/resources/glass-on-it-failure.pdf
   If the reasearch is right it must be 50 to 70%...
   

   View Slide

5. Slides by @arghrich
   CHAOS Report is highly controversial
   5
   CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
   IEEE https://www.cs.vu.nl/~x/the_rise_and_fall_of_the_chaos_report_figures.pdf
   AMB https://www.informationweek.com/executive-insights-and-innovation/the-non-existent-software-crisis-debunking-the-chaos-report
   AMB http://www.ambysoft.com/surveys/
   IEEE (the definitions are) solely
   based on estimation accuracy of
   cost, time, and functionality.
   AMB (independent study results) are
   much more positive than what the
   Standish Group claims.
   IEEE Our research shows that the
   Standish definitions (…) have 4
   major problems: they’re
   misleading, one-sided, pervert the
   estimation practice, and result in
   meaningless figures.
   CHAOS‘15 (in 2015) another major
   change is how we define success. Our
   new Modern definition is OnTime,
   OnBudget, with a satisfactory result.
   

   View Slide

6. Slides by @arghrich
   Think big, act small
   6
   CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
   CHAOS is in agreement with how to succeed
   the secret to project success is to
   strongly recommend and enforce
   limits on size and complexity.
   These two factors trump all others
   factors.
   break down large projects into a
   sequence of smaller ones, prioritized
   on direct business value (…)
   Aka scale down the work
   

   View Slide

7. Slides by @arghrich
   Success Factors for small projects
   7
   CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
   

   View Slide

8. Slides by @arghrich
   So why do Experts think Big
   Software Projects actually fail?
   8
   

   View Slide

9. Slides by @arghrich
   Politics
   9
   Highly abbreviated from a short Twitter conversation with @TotherAlistair.
   

   View Slide

10. Slides by @arghrich
    Succeeding with a Scaled
    Project*
    What we tried and avoided
    10
    * See if you can scale down the project first and deliver much smaller slices sooner
    

    View Slide

11. Slides by @arghrich
    A customer approaches* …
    11
    * pre-Pandemic
    I need 4 teams
    

    View Slide

12. Oh Boy
    

    View Slide

13. Slides by @arghrich
    Complexity is …
    Theory
    13
    Power
    Sexy
    Easier
    

    View Slide

14. Slides by @arghrich
    Scale Software via Evolutionary Design
    14 Theory
    https://www.industriallogic.com/blog/evolutionary-design/
    You can recognize the guitar
    Simple, easy to understand, makes music
    Can do more
    Still simple. Maybe that's enough for the
    customer. After all, time is money.
    This is a guitar
    Pretty good. Satisfies most people.
    Complex guitar
    Makes the best of the best happy. BUT,
    did we really have to serve them? How
    many normaler users did we lose with
    this complicated guitar?
    

    View Slide

15. Slides by @arghrich
    Scale Teams by starting small
    15 Theory
    Hexagonal Architecture
    Bind Api to Database
    Clean Code is …
    I don’t like the way you write tests
    This is not good design
    Java!
    C#
    Spaces
    Tabs
    F#
    One Repo per Deployable Singe Repo
    Rust
    Should we use …
    

    View Slide

16. Slides by @arghrich
    Fresh Teams have a lot …
    Theory
    16
    to discover
    of beef
    to agree on
    

    View Slide

17. Slides by @arghrich
    Scale Teams via Evolutionary Division
    17 Theory
    Spring
    1 American Pizza
    Summer
    2 American Pizzas
    Winter
    4 American Pizzas
    Autumn
    3 American Pizzas
    

    View Slide

18. Slides by @arghrich
    18
    Photo by Anne-sophie Parent from Pexels
    Slides by @arghrich
    Spring > Foundation
    

    View Slide

19. Slides by @arghrich
    Try…1 Co-located team
    Spring > Foundation
    19
    Get to know each other
    

    View Slide

20. Slides by @arghrich
    Try…Adopt a Process to Improve
    20 Spring > Foundation
    Retro
    2h Bi-Weekly
    • What to KEEP
    • What to TRY
    • Results often added to
    Team Charter
    Dev Talk
    1h Weekly
    • Present, Discuss,
    Agree on Tech
    • Results often ADRs
    • So many ADRs
    Slack Time
    

    View Slide

21. Slides by @arghrich
    Try…Adopt and adapt Scrum1
    21 Winter > Scaled Remote
    1 If you mix it with Xp, Scrum is not a bad start at all.
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team
    

    View Slide

22. Slides by @arghrich
    Try…Explicit1,2,3 Slack Time
    Spring > Foundation
    22
    1 Customers are super happy when tech stuff never shows up in their backlog. They don’t know how to prioritize it.
    2 Given devs ~2 days of slack time each sprint is a small price to pay when it stabilizes all estimation and planning.
    3 The backlog is also only about business features…
    To actually do Retro/Tech Items
    

    View Slide

23. Slides by @arghrich
    Try…Trunk-Based
    Development1
    Spring > Foundation
    23
    1 Predictive of success à Accelerate State of DevOps 2021 https://cloud.google.com/resources/state-of-devops#section-4
    The simplest, nicest branching model
    

    View Slide

24. Slides by @arghrich
    Avoid…Long-Lived Feature-Branches
    24 Spring > Foundation
    main
    

    View Slide

25. Slides by @arghrich
    Avoid…Continuous
    Isolation
    Branches
    • Increase personal productivity
    • Discourage Refactoring
    • Increase Batch Size
    • Increase Risk
    Daily merges (aka integration)
    • Increase team productivity
    • Motivate Refactoring
    • Decrease Batch Size
    • Decrease Risk
    25 Spring > Foundation
    See also Feature Branches are Evil by Thierry de Pauw
    https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html
    https://thinkinglabs.io/articles/2022/09/17/the-practices-that-make-continuous-integration-team-working.html
    Try…Continuous
    Integration
    

    View Slide

26. Slides by @arghrich
    Do…Pair TBD with maintained pipeline
    26 Spring > Foundation
    Behavior-changing, validated feature = upper-case F
    Behavior-preserving, safe refactor = lower-case r
    https://github.com/RefactoringCombos/ArlosCommitNotation
    main
    r r r r r F r r revert r
    Improve
    commit
    Always in a
    releasable state
    ▸ Lint
    ▸ Unit Test
    ▸ Int Test
    ▸ System Test
    ▸ PostDeploy
    Smoke Test
    ▸ PostDeploy
    Monitoring
    ▸ …
    Revert, if not
    fixable in 10min
    Small, focused
    commits
    

    View Slide

27. Slides by @arghrich
    Our customer approaches again* …
    27 Late Spring
    * Still pre-Pandemic
    I want 4 teams
    I want scalable software
    

    View Slide

28. Slides by @arghrich
    28 Summer
    Photo by Pixabay from Pexels
    Slides by @arghrich
    

    View Slide

29. Slides by @arghrich
    Try…Start with a monolith
    • At project inception you have the least knowledge
    • Your modelling can and will change a lot in the beginning
    • Refactoring in a monolith has tool support
    • However: splitting up a monolith is not a freebie either
    29 Summer > Scale-Out
    

    View Slide

30. Slides by @arghrich
    Try…Keep everything in one repo
    31 Summer > Scale-Out
    • All
    • Services
    • Shared Libraries
    • Documentation
    • Configuration
    • Provisioning
    • Onboarding.sh Scripts
    à Everything your team needs
    • You see everything
    everywhere, all at once
    • You filter what you need right
    now
    • Your (refactoring) tools help
    you with every change (in
    shared libraries)
    • You can unit test all services
    together J
    

    View Slide

31. Slides by @arghrich
    Our customer approaches again again* …
    33 Late Summer
    * Still pre-Pandemic
    I want 4 teams
    I want scaledable software
    I want more, in parallel
    

    View Slide

32. Slides by @arghrich
    34 Autumn
    Photo by Daniel Frank from Pexels
    Slides by @arghrich
    

    View Slide

33. Slides by @arghrich
    Try…Dynamic Feature Teams1
    35 Autumn > Divide
    1 A more extreme version of this approach is FAST agile https://www.fastagile.io/method
    See also Hands-on Agile 45: FAST Scaling — James Shore [Youtube]
    Planning 1 Planning 1,5
    Topic A
    Topic B
    Present topics incl.
    Backlog
    Pick
    Topic
    Topic A
    Topic B
    Planning 2
    Task Breakdown
    

    View Slide

34. Slides by @arghrich
    Try…Dynamic Feature Teams
    Each team can work anywhere in the repo
    36 Autumn > Divide
    Align via … the coffee machine
    the dev talk and ADRs
    Bi-weekly re-teaming
    

    View Slide

35. Slides by @arghrich
    Our customer approaches remote* …
    38 Late Autumn
    *Pandemic!
    I want 4 teams
    I want scaledable software
    I want more, in parallel
    I want more speed I hate remote
    

    View Slide

36. Slides by @arghrich
    39 Winter
    Photo by Riccardo Bresciani from Pexels
    Slides by @arghrich
    

    View Slide

37. Slides by @arghrich
    Try…Divide & Conquer to increase
    Meeting Efficiency
    40 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    3 Amigos
    30m | 2x | 3 Ppl
    Daily
    Daily
    15m | Team
    During Week Weekly Bi-Weekly
    Retro
    2h Bi-Weekly
    Review
    1h Bi-Weekly
    Planning
    2h Bi-Weekly
    Refinement
    30m | 2x | Team
    

    View Slide

38. Slides by @arghrich
    Try…Stable Streams
    41 Winter > Scaled Remote
    Inception
    Pick
    Stream
    You already know who you can work with best.
    

    View Slide

39. Slides by @arghrich
    Try…One Shared Backlog
    42 Winter > Scaled Remote
    Backlog
    

    View Slide

40. Slides by @arghrich
    Try…Stable What’s Next1,2
    43 Winter > Scaled Remote
    1 Stream Backlogs should never run dry
    2 At this point you know your velocity to forecast feature completion dates
    What’s Next
    Backlog
    Task Breakdown
    Stream
    Backlogs
    

    View Slide

41. Slides by @arghrich
    Try…Flexible Cadences
    44 Winter > Scaled Remote
    Original Scrum and Invention
    Dev Talk
    1,5h | Team
    Slack Time
    Refinement
    30m | 3 per Week | 3 Ppl
    What’s Next
    1h | Stream
    Daily
    Retro
    20m | Stream
    Daily
    15m | Stream
    During Week Weekly Bi-Monthly
    Roadmap
    1h | Team
    Incl. Story Map
    On-Demand
    Retro
    2,3h | Stream or Team
    

    View Slide

42. Slides by @arghrich
    Try…Response Team
    45 Winter > Scaled Remote
    Motivation
    • Reduce context switches
    Activities
    1. Keep Prod Stable
    2. Keep Dev Env Stable
    3. Proactively work on even more stability
    4. In that order
    

    View Slide

43. Slides by @arghrich
    How do you find people that
    can help/enlighten/chat in a
    remote setting? 1,2
    Winter > Scaled Remote
    46
    1 Can meaning, they are not in focus mode.
    2 In a co-located setting you can easily see if someone does not want to be disturbed.
    

    View Slide

44. Slides by @arghrich
    Try…Findable Remote Stream Rooms
    47 Winter > Scaled Remote
    Channel A
    We work on feature xyz
    Channel B
    We work on feature abc
    Channel Discovery
    We are discovering
    • Use team-internal channels/calendar events/etc. where the
    streams work
    No one there or all
    in focus mode.
    Someone is there,
    working on a feature
    Someone is there, working on
    the next features
    

    View Slide

45. Slides by @arghrich
    48 Summary
    Photo by Irina Iriser from Pexels
    Slides by @arghrich
    

    View Slide

46. Slides by @arghrich
    How far we got
    • 1 team with 6 devs
    • 2 week Sprints
    • 2h Planning
    • Dev Talk Decisions
    • Whole-team 2h retros every 2
    weeks
    • 1 team, 5 streams with 2-0 devs
    • Continuous
    • 1h What’s next meetings
    • Stream decisions
    + Dev Talk presentation
    • Daily 15min retros, On-demand
    large stream or team retros
    49 Summary
    

    View Slide

47. Slides by @arghrich
    We improved as a team
    Spring > Foundation
    50
    

    View Slide

48. Slides by @arghrich
    People issues determine a project’s speed
    Can they easily detect something needs attention?
    Will they care enough to do something about it?
    Can they effectively pass along the information?
    51 Summary
    Stolen from Alistair Cockburn „Why Agile Works“ at ~28m
    

    View Slide

49. Slides by @arghrich
    Do you have a
    Process to
    Experiment
    and Improve?
    Summary
    52
    

    View Slide

50. Slides by @arghrich
    Do you have
    the Mindset?
    Summary
    53
    

    View Slide

51. Slides by @arghrich
    Try…Remote Trunk-Based Development
    Just don’t forget it’s not about the techniques
    54 Summary
    Pipeline
    Easy Hard
    Techniques
    Something you do
    Methods
    When you do it
    Mindset
    What you actually do
    

    View Slide

52. Slides by @arghrich
    Thanks
    55
    IT-Tailor
    Richard Gross (he/him)
    Archaeologist Auditor
    richargh.de/
    speakerdeck.com/richargh
    @arghrich
    Works at maibornwolff.de/
    Tailored Software by People
    who Care.
    DE TN ES
    

    View Slide

53. Slides by @arghrich
    Backup
    56
    

    View Slide

54. Slides by @arghrich
    57
    Twitter https://twitter.com/ArghRich/status/1569009015123808258 ?cxt=HHwWhMC4vdzYnsYrAAAA
    

    View Slide