$30 off During Our Annual Pro Sale. View Details »

Remote Scaled Trunk-Based Development (IPC)

October 25, 2022

Remote Scaled Trunk-Based Development (IPC)

Held at IPC Fall 2022
How do we coordinate teams of 20 or more developers and achieve consistent quality in a remote setting? Plenty scaling methods have been proposed but these models are usually based on Scrum and leave out the technical practices needed to achieve quality at scale.

In this talk we‘ll explore how my project grew from a local six person scrum team to a remote 20 person continuous flow team and beyond. We‘ll take a look at the iterations we needed until we achieved our flow, how we implemented Xp practices like trunk-based development and what practices we adopted in our all-remote setting.

By the end I hope to have provided you with all the information you need to achieve quality at scale in a remote setting.


Talk held at https://phpconference.com/mixed/from-the-trenches-remote-scaled-trunk-based-development/


October 25, 2022

More Decks by Richard

Other Decks in Programming


  1. Remote Scaled Trunk- Based Development* 25.10.22 * I hope I

    did not forget any buzzwords. Richard Gross (he/him) Coder, Archaeologist, Auditor. Works at maibornwolff.de/ richargh.de/ speakerdeck.com/richargh @arghrich
  2. Slides by @arghrich Why do Big Software Projects fail? 2

  3. Slides by @arghrich Big Projects fail because they are Big?

    3 CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf HBR 2011 https://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think McK https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value McK On average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted. HBR one in six of the projects we studied was a black swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. CHAOS‘15 size (is) the single most important factor in the (…) outcome.
  4. Slides by @arghrich How many Software Projects fail at your

    company? 4 IT Failure Rates— 70% or 10–15%? https://www.cs.uni.edu/~wallingf/teaching/172/resources/glass-on-it-failure.pdf If the reasearch is right it must be 50 to 70%...
  5. Slides by @arghrich CHAOS Report is highly controversial 5 CHAOS’15

    https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf IEEE https://www.cs.vu.nl/~x/the_rise_and_fall_of_the_chaos_report_figures.pdf AMB https://www.informationweek.com/executive-insights-and-innovation/the-non-existent-software-crisis-debunking-the-chaos-report AMB http://www.ambysoft.com/surveys/ IEEE (the definitions are) solely based on estimation accuracy of cost, time, and functionality. AMB (independent study results) are much more positive than what the Standish Group claims. IEEE Our research shows that the Standish definitions (…) have 4 major problems: they’re misleading, one-sided, pervert the estimation practice, and result in meaningless figures. CHAOS‘15 (in 2015) another major change is how we define success. Our new Modern definition is OnTime, OnBudget, with a satisfactory result.
  6. Slides by @arghrich Think big, act small 6 CHAOS Manifesto

    ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf CHAOS is in agreement with how to succeed the secret to project success is to strongly recommend and enforce limits on size and complexity. These two factors trump all others factors. break down large projects into a sequence of smaller ones, prioritized on direct business value (…) Aka scale down the work
  7. Slides by @arghrich Success Factors for small projects 7 CHAOS

    Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
  8. Slides by @arghrich So why do Experts think Big Software

    Projects actually fail? 8
  9. Slides by @arghrich Politics 9 Highly abbreviated from a short

    Twitter conversation.
  10. Slides by @arghrich Succeeding with a Scaled Project* What we

    tried and avoided 10 * See if you can scale down the project first and deliver much smaller slices sooner
  11. Slides by @arghrich A customer approaches* … 11 * pre-Pandemic

    I need 4 teams
  12. Oh Boy

  13. Slides by @arghrich Complexity is … Theory 13 Power Sexy

  14. Slides by @arghrich Scale Software via Evolutionary Design 14 Theory

    https://www.industriallogic.com/blog/evolutionary-design/ You can recognize the guitar Simple, easy to understand, makes music Can do more Still simple. Maybe that's enough for the customer. After all, time is money. This is a guitar Pretty good. Satisfies most people. Complex guitar Makes the best of the best happy. BUT, did we really have to serve them? How many normaler users did we lose with this complicated guitar?
  15. Slides by @arghrich Scale Teams by starting small 15 Theory

    Hexagonal Architecture Bind Api to Database Clean Code is … I don’t like the way you write tests This is not good design Java! C# Spaces Tabs F# One Repo per Deployable Singe Repo Rust Should we use …
  16. Slides by @arghrich Fresh Teams have a lot … Theory

    16 to discover of beef to agree on
  17. Slides by @arghrich Scale Teams via Evolutionary Division 17 Theory

    Spring 1 American Pizza Summer 2 American Pizzas Winter 4 American Pizzas Autumn 3 American Pizzas
  18. Slides by @arghrich 18 Photo by Anne-sophie Parent from Pexels

    Slides by @arghrich Spring > Foundation
  19. Slides by @arghrich Try…1 Co-located team Spring > Foundation 19

    Get to know each other
  20. Slides by @arghrich Try…Adopt a Process to Improve 20 Spring

    > Foundation Retro 2h Bi-Weekly • What to KEEP • What to TRY • Results often added to Team Charter Dev Talk 1h Weekly • Present, Discuss, Agree on Tech • Results often ADRs • So many ADRs Slack Time
  21. Slides by @arghrich Try…Adopt and adapt Scrum1 21 Winter >

    Scaled Remote 1 If you mix it with Xp, Scrum is not a bad start at all. Original Scrum and Invention Dev Talk 1,5h | Team Slack Time 3 Amigos 30m | 2x | 3 Ppl Daily Daily 15m | Team During Week Weekly Bi-Weekly Retro 2h Bi-Weekly Review 1h Bi-Weekly Planning 2h Bi-Weekly Refinement 30m | 2x | Team
  22. Slides by @arghrich Try…Explicit1,2,3 Slack Time Spring > Foundation 22

    1 Customers are super happy when tech stuff never shows up in their backlog. They don’t know how to prioritize it. 2 Given devs ~2 days of slack time each sprint is a small price to pay when it stabilizes all estimation and planning. 3 The backlog is also only about business features… To actually do Retro/Tech Items
  23. Slides by @arghrich Try…Trunk-Based Development1 Spring > Foundation 23 1

    Predictive of success à Accelerate State of DevOps 2021 https://cloud.google.com/resources/state-of-devops#section-4 The simplest, nicest branching model
  24. Slides by @arghrich Avoid…Long-Lived Feature-Branches 24 Spring > Foundation main

  25. Slides by @arghrich Avoid…Continuous Isolation Branches • Increase personal productivity

    • Discourage Refactoring • Increase Batch Size • Increase Risk Daily merges (aka integration) • Increase team productivity • Motivate Refactoring • Decrease Batch Size • Decrease Risk 25 Spring > Foundation See also Feature Branches are Evil by Thierry de Pauw https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html https://thinkinglabs.io/articles/2022/09/17/the-practices-that-make-continuous-integration-team-working.html Try…Continuous Integration
  26. Slides by @arghrich Do…Pair TBD with maintained pipeline 26 Spring

    > Foundation Behavior-changing, validated feature = upper-case F Behavior-preserving, safe refactor = lower-case r https://github.com/RefactoringCombos/ArlosCommitNotation main r r r r r F r r revert r Improve commit Always in a releasable state ▸ Lint ▸ Unit Test ▸ Int Test ▸ System Test ▸ PostDeploy Smoke Test ▸ PostDeploy Monitoring ▸ … Revert, if not fixable in 10min Small, focused commits
  27. Slides by @arghrich Our customer approaches again* … 28 Late

    Spring * Still pre-Pandemic I want 4 teams I want scalable software
  28. Slides by @arghrich 29 Summer Photo by Pixabay from Pexels

    Slides by @arghrich
  29. Slides by @arghrich Try…Start with a monolith • At project

    inception you have the least knowledge • Your modelling can and will change a lot in the beginning • Refactoring in a monolith has tool support • However: splitting up a monolith is not a freebie either 30 Summer > Scale-Out
  30. Slides by @arghrich Try…Event-Driven Communication 31 Summer > Scale-Out Nothing

    to do with TBD, Remote or Scaled, just a cool approach for service integration. Service 1 Service … Service n Kafka (RabbitMq etc. work nicely as well)
  31. Slides by @arghrich Try…Keep everything in one repo 32 Summer

    > Scale-Out • All • Services • Shared Libraries • Documentation • Configuration • Provisioning • Onboarding.sh Scripts à Everything your team needs • You see everything everywhere, all at once • You filter what you need right now • Your (refactoring) tools help you with every change (in shared libraries) • You can unit test all services together J
  32. Slides by @arghrich Our customer approaches again again* … 34

    Late Summer * Still pre-Pandemic I want 4 teams I want scaled a software I want more, in parallel
  33. Slides by @arghrich 35 Autumn Photo by Daniel Frank from

    Pexels Slides by @arghrich
  34. Slides by @arghrich Try…Dynamic Feature Teams1 36 Autumn > Divide

    1 A more extreme version of this approach is FAST agile https://www.fastagile.io/method See also Hands-on Agile 45: FAST Scaling — James Shore [Youtube] Planning 1 Planning 1,5 Topic A Topic B Present topics incl. Backlog Pick Topic Topic A Topic B Planning 2 Task Breakdown
  35. Slides by @arghrich Try…Dynamic Feature Teams Each team can work

    anywhere in the repo 37 Autumn > Divide Align via … the coffee machine the dev talk and ADRs Bi-weekly re-teaming
  36. Slides by @arghrich Our customer approaches remote* … 40 Late

    Autumn *Pandemic! I want 4 teams I want scaled a software I want more, in parallel I want more speed I hate remote
  37. Slides by @arghrich 41 Winter Photo by Riccardo Bresciani from

    Pexels Slides by @arghrich
  38. Slides by @arghrich Try…Divide & Conquer to increase Meeting Efficiency

    42 Winter > Scaled Remote Original Scrum and Invention Dev Talk 1,5h | Team Slack Time 3 Amigos 30m | 2x | 3 Ppl Daily Daily 15m | Team During Week Weekly Bi-Weekly Retro 2h Bi-Weekly Review 1h Bi-Weekly Planning 2h Bi-Weekly Refinement 30m | 2x | Team
  39. Slides by @arghrich Try…Stable Streams 43 Winter > Scaled Remote

    Inception Pick Stream You already know who you can work with best.
  40. Slides by @arghrich Try…Stable What’s Next 44 Winter > Scaled

    Remote Backlog
  41. Slides by @arghrich Try…Stable What’s Next1,2 45 Winter > Scaled

    Remote 1 Stream Backlogs should never run dry 2 At this point you know your velocity to forecast feature completion dates What’s Next Backlog Task Breakdown Stream Backlogs
  42. Slides by @arghrich Try…Flexible Cadences 46 Winter > Scaled Remote

    Original Scrum and Invention Dev Talk 1,5h | Team Slack Time Refinement 30m | 3 per Week | 3 Ppl What’s Next 1h | Stream Daily Retro 20m | Stream Daily 15m | Stream During Week Weekly Bi-Monthly Roadmap 1h | Team Incl. Story Map On-Demand Retro 2,3h | Stream or Team
  43. Slides by @arghrich Try…Response Team 47 Winter > Scaled Remote

    Motivation • Reduce context switches Activities 1. Keep Prod Stable 2. Keep Dev Env Stable 3. Proactively work on even more stability 4. In that order
  44. Slides by @arghrich How do you find people that can

    help/enlighten/chat in a remote setting? 1,2 Winter > Scaled Remote 48 1 Can meaning, they are not in focus mode. 2 In a co-located setting you can easily see if someone does not want to be disturbed.
  45. Slides by @arghrich Try…Findable Remote Stream Rooms 49 Winter >

    Scaled Remote Channel A We work on feature xyz Channel B We work on feature abc Channel Discovery We are discovering • Use team-internal channels/calendar events/etc. where the streams work Now one there or all in focus mode. Someone is there, working on a feature Someone is there, working on the next features
  46. Slides by @arghrich 50 Summary Photo by Irina Iriser from

    Pexels Slides by @arghrich
  47. Slides by @arghrich How far we got • 1 team

    with 6 devs • 2 week Sprints • 2h Planning • Dev Talk Decisions • Whole-team 2h retros every 2 weeks • 1 team, 5 streams with 2-0 devs • Continuous • 1h What’s next meetings • Stream decisions + Dev Talk presentation • Daily 15min retros, On-demand large stream or team retros 51 Summary
  48. Slides by @arghrich People issues determine a project’s speed Can

    they easily detect something needs attention? Will they care enough to do something about it? Can they effectively pass along the information? 52 Summary Stolen from Alistair Cockburn „Why Agile Works“ at ~28m
  49. Slides by @arghrich Do you have a Process to Experiment

    and Improve? Summary 53
  50. Slides by @arghrich Do you have the Mindset? Summary 54

  51. Slides by @arghrich Try…Remote Trunk-Based Development Just don’t forget it’s

    not about the techniques 55 Summary Pipeline Easy Hard Techniques Something you do Methods When you do it Mindset What you actually do
  52. Slides by @arghrich Thanks 56 Richard Gross (he/him) Coder, Archaeologist,

    Auditor. Works at maibornwolff.de/ richargh.de/ speakerdeck.com/richargh @arghrich
  53. Slides by @arghrich Backup 57

  54. Slides by @arghrich 58 Twitter https://twitter.com/ArghRich/status/1569009015123808258 ?cxt=HHwWhMC4vdzYnsYrAAAA