Remote Scaled Trunk-Based Development (Build Stuff)

Held at Build Stuff 2022
----
How do we coordinate teams of 20 or more developers and achieve consistent quality in a remote setting? Plenty scaling methods have been proposed but these models are usually based on Scrum and leave out the technical practices needed to achieve quality at scale.

In this talk we‘ll explore how my project grew from a local six person scrum team to a remote 20 person continuous flow team and beyond. We‘ll take a look at the iterations we needed until we achieved our flow, how we implemented Xp practices like trunk-based development and what practices we adopted in our all-remote setting.

By the end I hope to have provided you with all the information you need to achieve quality at scale in a remote setting.

Transcript

1. Remote Scaled Trunk- Based Development* 09.11.22 * I hope I did not forget any buzzwords. IT-Tailor Richard Gross (he/him) Archeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich Works at maibornwolff.de/
2. Slides by @arghrich Why do Big Software Projects fail? 2
3. Slides by @arghrich Big Projects fail because they are Big? 3 CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf HBR 2011 https://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think McK https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/delivering-large-scale-it-projects-on-time-on-budget-and-on-value McK On average, large IT projects run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted. HBR one in six of the projects we studied was a black swan, with a cost overrun of 200%, on average, and a schedule overrun of almost 70%. CHAOS‘15 size (is) the single most important factor in the (…) outcome.
4. Slides by @arghrich How many Software Projects fail at your company? 4 IT Failure Rates— 70% or 10–15%? https://www.cs.uni.edu/~wallingf/teaching/172/resources/glass-on-it-failure.pdf If the reasearch is right it must be 50 to 70%...
5. Slides by @arghrich CHAOS Report is highly controversial 5 CHAOS’15 https://standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf IEEE https://www.cs.vu.nl/~x/the_rise_and_fall_of_the_chaos_report_figures.pdf AMB https://www.informationweek.com/executive-insights-and-innovation/the-non-existent-software-crisis-debunking-the-chaos-report AMB http://www.ambysoft.com/surveys/ IEEE (the definitions are) solely based on estimation accuracy of cost, time, and functionality. AMB (independent study results) are much more positive than what the Standish Group claims. IEEE Our research shows that the Standish definitions (…) have 4 major problems: they’re misleading, one-sided, pervert the estimation practice, and result in meaningless figures. CHAOS‘15 (in 2015) another major change is how we define success. Our new Modern definition is OnTime, OnBudget, with a satisfactory result.
6. Slides by @arghrich Think big, act small 6 CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf CHAOS is in agreement with how to succeed the secret to project success is to strongly recommend and enforce limits on size and complexity. These two factors trump all others factors. break down large projects into a sequence of smaller ones, prioritized on direct business value (…) Aka scale down the work
7. Slides by @arghrich Success Factors for small projects 7 CHAOS Manifesto ‘13 https://larlet.fr/static/david/stream/ChaosManifesto2013.pdf
8. Slides by @arghrich So why do Experts think Big Software Projects actually fail? 8
9. Slides by @arghrich Politics 9 Highly abbreviated from a short Twitter conversation with @TotherAlistair.
10. Slides by @arghrich Succeeding with a Scaled Project* What we tried and avoided 10 * See if you can scale down the project first and deliver much smaller slices sooner
11. Slides by @arghrich A customer approaches* … 11 * pre-Pandemic I need 4 teams
12. Oh Boy
13. Slides by @arghrich Complexity is … Theory 13 Power Sexy Easier
14. Slides by @arghrich Scale Software via Evolutionary Design 14 Theory https://www.industriallogic.com/blog/evolutionary-design/ You can recognize the guitar Simple, easy to understand, makes music Can do more Still simple. Maybe that's enough for the customer. After all, time is money. This is a guitar Pretty good. Satisfies most people. Complex guitar Makes the best of the best happy. BUT, did we really have to serve them? How many normaler users did we lose with this complicated guitar?
15. Slides by @arghrich Scale Teams by starting small 15 Theory Hexagonal Architecture Bind Api to Database Clean Code is … I don’t like the way you write tests This is not good design Java! C# Spaces Tabs F# One Repo per Deployable Singe Repo Rust Should we use …
16. Slides by @arghrich Fresh Teams have a lot … Theory 16 to discover of beef to agree on
17. Slides by @arghrich Scale Teams via Evolutionary Division 17 Theory Spring 1 American Pizza Summer 2 American Pizzas Winter 4 American Pizzas Autumn 3 American Pizzas
18. Slides by @arghrich 18 Photo by Anne-sophie Parent from Pexels Slides by @arghrich Spring > Foundation
19. Slides by @arghrich Try…1 Co-located team Spring > Foundation 19 Get to know each other
20. Slides by @arghrich Try…Adopt a Process to Improve 20 Spring > Foundation Retro 2h Bi-Weekly • What to KEEP • What to TRY • Results often added to Team Charter Dev Talk 1h Weekly • Present, Discuss, Agree on Tech • Results often ADRs • So many ADRs Slack Time
21. Slides by @arghrich Try…Adopt and adapt Scrum1 21 Winter > Scaled Remote 1 If you mix it with Xp, Scrum is not a bad start at all. Original Scrum and Invention Dev Talk 1,5h | Team Slack Time 3 Amigos 30m | 2x | 3 Ppl Daily Daily 15m | Team During Week Weekly Bi-Weekly Retro 2h Bi-Weekly Review 1h Bi-Weekly Planning 2h Bi-Weekly Refinement 30m | 2x | Team
22. Slides by @arghrich Try…Explicit1,2,3 Slack Time Spring > Foundation 22 1 Customers are super happy when tech stuff never shows up in their backlog. They don’t know how to prioritize it. 2 Given devs ~2 days of slack time each sprint is a small price to pay when it stabilizes all estimation and planning. 3 The backlog is also only about business features… To actually do Retro/Tech Items
23. Slides by @arghrich Try…Trunk-Based Development1 Spring > Foundation 23 1 Predictive of success à Accelerate State of DevOps 2021 https://cloud.google.com/resources/state-of-devops#section-4 The simplest, nicest branching model
24. Slides by @arghrich Avoid…Long-Lived Feature-Branches 24 Spring > Foundation main
25. Slides by @arghrich Avoid…Continuous Isolation Branches • Increase personal productivity • Discourage Refactoring • Increase Batch Size • Increase Risk Daily merges (aka integration) • Increase team productivity • Motivate Refactoring • Decrease Batch Size • Decrease Risk 25 Spring > Foundation See also Feature Branches are Evil by Thierry de Pauw https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html https://thinkinglabs.io/articles/2022/09/17/the-practices-that-make-continuous-integration-team-working.html Try…Continuous Integration
26. Slides by @arghrich Do…Pair TBD with maintained pipeline 26 Spring > Foundation Behavior-changing, validated feature = upper-case F Behavior-preserving, safe refactor = lower-case r https://github.com/RefactoringCombos/ArlosCommitNotation main r r r r r F r r revert r Improve commit Always in a releasable state ▸ Lint ▸ Unit Test ▸ Int Test ▸ System Test ▸ PostDeploy Smoke Test ▸ PostDeploy Monitoring ▸ … Revert, if not fixable in 10min Small, focused commits
27. Slides by @arghrich Our customer approaches again* … 27 Late Spring * Still pre-Pandemic I want 4 teams I want scalable software
28. Slides by @arghrich 28 Summer Photo by Pixabay from Pexels Slides by @arghrich
29. Slides by @arghrich Try…Start with a monolith • At project inception you have the least knowledge • Your modelling can and will change a lot in the beginning • Refactoring in a monolith has tool support • However: splitting up a monolith is not a freebie either 29 Summer > Scale-Out
30. Slides by @arghrich Try…Keep everything in one repo 31 Summer > Scale-Out • All • Services • Shared Libraries • Documentation • Configuration • Provisioning • Onboarding.sh Scripts à Everything your team needs • You see everything everywhere, all at once • You filter what you need right now • Your (refactoring) tools help you with every change (in shared libraries) • You can unit test all services together J
31. Slides by @arghrich Our customer approaches again again* … 33 Late Summer * Still pre-Pandemic I want 4 teams I want scaledable software I want more, in parallel
32. Slides by @arghrich 34 Autumn Photo by Daniel Frank from Pexels Slides by @arghrich
33. Slides by @arghrich Try…Dynamic Feature Teams1 35 Autumn > Divide 1 A more extreme version of this approach is FAST agile https://www.fastagile.io/method See also Hands-on Agile 45: FAST Scaling — James Shore [Youtube] Planning 1 Planning 1,5 Topic A Topic B Present topics incl. Backlog Pick Topic Topic A Topic B Planning 2 Task Breakdown
34. Slides by @arghrich Try…Dynamic Feature Teams Each team can work anywhere in the repo 36 Autumn > Divide Align via … the coffee machine the dev talk and ADRs Bi-weekly re-teaming
35. Slides by @arghrich Our customer approaches remote* … 38 Late Autumn *Pandemic! I want 4 teams I want scaledable software I want more, in parallel I want more speed I hate remote
36. Slides by @arghrich 39 Winter Photo by Riccardo Bresciani from Pexels Slides by @arghrich
37. Slides by @arghrich Try…Divide & Conquer to increase Meeting Efficiency 40 Winter > Scaled Remote Original Scrum and Invention Dev Talk 1,5h | Team Slack Time 3 Amigos 30m | 2x | 3 Ppl Daily Daily 15m | Team During Week Weekly Bi-Weekly Retro 2h Bi-Weekly Review 1h Bi-Weekly Planning 2h Bi-Weekly Refinement 30m | 2x | Team
38. Slides by @arghrich Try…Stable Streams 41 Winter > Scaled Remote Inception Pick Stream You already know who you can work with best.
39. Slides by @arghrich Try…One Shared Backlog 42 Winter > Scaled Remote Backlog
40. Slides by @arghrich Try…Stable What’s Next1,2 43 Winter > Scaled Remote 1 Stream Backlogs should never run dry 2 At this point you know your velocity to forecast feature completion dates What’s Next Backlog Task Breakdown Stream Backlogs
41. Slides by @arghrich Try…Flexible Cadences 44 Winter > Scaled Remote Original Scrum and Invention Dev Talk 1,5h | Team Slack Time Refinement 30m | 3 per Week | 3 Ppl What’s Next 1h | Stream Daily Retro 20m | Stream Daily 15m | Stream During Week Weekly Bi-Monthly Roadmap 1h | Team Incl. Story Map On-Demand Retro 2,3h | Stream or Team
42. Slides by @arghrich Try…Response Team 45 Winter > Scaled Remote Motivation • Reduce context switches Activities 1. Keep Prod Stable 2. Keep Dev Env Stable 3. Proactively work on even more stability 4. In that order
43. Slides by @arghrich How do you find people that can help/enlighten/chat in a remote setting? 1,2 Winter > Scaled Remote 46 1 Can meaning, they are not in focus mode. 2 In a co-located setting you can easily see if someone does not want to be disturbed.
44. Slides by @arghrich Try…Findable Remote Stream Rooms 47 Winter > Scaled Remote Channel A We work on feature xyz Channel B We work on feature abc Channel Discovery We are discovering • Use team-internal channels/calendar events/etc. where the streams work No one there or all in focus mode. Someone is there, working on a feature Someone is there, working on the next features
45. Slides by @arghrich 48 Summary Photo by Irina Iriser from Pexels Slides by @arghrich
46. Slides by @arghrich How far we got • 1 team with 6 devs • 2 week Sprints • 2h Planning • Dev Talk Decisions • Whole-team 2h retros every 2 weeks • 1 team, 5 streams with 2-0 devs • Continuous • 1h What’s next meetings • Stream decisions + Dev Talk presentation • Daily 15min retros, On-demand large stream or team retros 49 Summary
47. Slides by @arghrich We improved as a team Spring > Foundation 50
48. Slides by @arghrich People issues determine a project’s speed Can they easily detect something needs attention? Will they care enough to do something about it? Can they effectively pass along the information? 51 Summary Stolen from Alistair Cockburn „Why Agile Works“ at ~28m
49. Slides by @arghrich Do you have a Process to Experiment and Improve? Summary 52
50. Slides by @arghrich Do you have the Mindset? Summary 53
51. Slides by @arghrich Try…Remote Trunk-Based Development Just don’t forget it’s not about the techniques 54 Summary Pipeline Easy Hard Techniques Something you do Methods When you do it Mindset What you actually do
52. Slides by @arghrich Thanks 55 IT-Tailor Richard Gross (he/him) Archaeologist Auditor richargh.de/ speakerdeck.com/richargh @arghrich Works at maibornwolff.de/ Tailored Software by People who Care. DE TN ES
53. Slides by @arghrich Backup 56
54. Slides by @arghrich 57 Twitter https://twitter.com/ArghRich/status/1569009015123808258 ?cxt=HHwWhMC4vdzYnsYrAAAA