The reusability fallacy

This slide deck discusses reusability, a topic that is sort of a "holy grail" in software architecture. Unfortunately, reusability often is discussed in the wrong places and for the wrong reasons, especially if it is used to sell a new architectural paradigm ("via reuse, the initial investment will amortize very soon and it will pay for itself").

I start with the origins of the topic in the early 1970s (and the different conditions how software was developed back then). Then I discuss and debunk the most widespread fallacies that are typically used to press for reusability.

With that knowledge I present a few advices (not dogmas!) how to deal better with reusability - when not to use it, what to do instead of those places, and: When to use it!

Reusability has a huge value in software development and we need to make sure to harvest it. But pressing for it in the wrong places for the wrong reasons only causes problems and harm. We need to avoid that and the intention of the talk is to have better arguments if you encounter such a situation.

This is the unabridged slide deck containing all slides of the talk. For conferences, I usually leave out several slides to meet the session length constraints.

I also accompanied this slide deck with a blog series that can be found at https://ufried.com/blog/reusability_fallacy_1/. The series consists of 4 posts, covers the contents of this talk and provides a few more ideas and pointers.

Edit: Uploaded an updated version of the slide deck on Jun-30-2023

Transcript

1. The reusability fallacy Killing a beloved dogma Uwe Friedrichsen – codecentric AG – 2008-2023
2. Uwe Friedrichsen Works @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://ufried.com/blog/reusability_fallacy_1/
3. The “holy grail” of architecture
4. The promise (the dream) • Create software module once • Reuse as often as you like at no reproduction cost • “Invest once, harvest forever” • Big efficiency lever • Huge cost saver
5. The origins
6. Structured design by W. P. Stevens, G. J. Myers and L. L. Constantine [Ste 1974]
7. “Considerations and techniques are proposed that reduce the complexity of programs by dividing them into functional modules. This can make it possible to create complex systems from simple, independent, reusable modules. […] And, as the module library grows, increasingly sophisticated programs can be implemented using less and less new code.” [Ste 1974]
8. The origins • Reusability discussion started ~1970 • Most companies and projects started at scratch • Barely more than 3GL compiler and linker available • No big standard libraries bundled with language • No big ecosystems available around language • Even most basic functionalities needed to be developed • Reusability could increase productivity significantly
9. Reusability today (the fallacy) • Became an end in itself • “Reusability is needed. Period!” • Demanded by managers to improve cost efficiency • “I want reusability to improve the IT cost ratio” • Used by architects to sell new architectural paradigms • “Required investment will amortize soon due to reuse” à Never worked this way. Never will work this way.
10. Still, reusability is widely popular. Thus, where is the misconception?
11. Actually, it is a mix of misconceptions
12. Misconception #1 Industrial production analogies
13. “Let us apply the reusability approaches from house building or car manufacturing” -- The unknown “industrialization of IT” advocate
14. Reuse ≈ Standardization
15. Standardization • Standardization helps building at reduced price • Same part can be reused in many products • Saves production costs over time • Design efforts only once • Production process design and set up only once • Required tooling created and installed only once • Cost savings are realized in production • Spreads design and set up costs over parts produced
16. Software development has nothing in common with producing a physical good
17. The products we build are not source code, but executable programs
18. Our assembly lines are so efficient for more than 50 years that most people forgot they even exist We call them “compiler” * * or CI/CD pipelines in the settings of today
19. Software development is about designing the product The design is only complete after the last line of code is written * * Building the product from the design (source code) is merely pushing a button
20. We cannot make software production more cost-efficient by building more parts from the same design
21. Our production process is already unbelievably cheap * and we can produce multiple copies from a design at zero cost We call it the “copy command” * compared to the production process of any physical goods
22. All the metaphors comparing software development to house building or car assembly are just plain wrong
23. Conclusion Potential cost or efficiency improvements of reusability are limited to design time
24. New (industrially minted) idea ”The more reusable parts we use in our design process (a.k.a. software development), the more time, costs and effort we can save”
25. No silver bullet by Frederick P. Brooks, Jr. [Bro 1986]
26. “Software entities are more complex for their size than perhaps any other human construct, because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into one, a subroutine, open or closed. In this respect software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound. [...] Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger size; it is necessarily an increase in the number of different elements.” [Bro 1986]
27. Reusing the same part repeatedly does not work
28. Reuse is limited to lower-level functionalities supporting higher-level functionalities, still resulting in a significant increase of different parts
29. Misconception #2 Reusability comes for free
30. “Each reusable part saves me a lot of money!” -- The unknown decision maker
31. Well, ever considered the cost of creating a reusable asset?
32. The tar pit by Frederick P. Brooks, Jr. (taken from the “The mythical man-month”) [Bro 1995]
33. “One occasionally reads newspaper accounts of how two programmers in a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams. Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.” [Bro 1995]
34. 1x Program The original module, suitable for the context it was created for 3x Programming Product A generalized module, suitable for multiple contexts A module, ready to be used in an ecosystem of interacting modules A (re-)usable module, that provides a general solution for a problem 3x Programming System 9x Programming Systems Product
35. 1x Program 9x Programming Systems Product Completeness of functionality • Handling of edge cases • Hardening implementation • Thorough functional testing • Design Documentation Completeness of accessibility • Precise interface definition • Clear behavioral contract • Thorough integration (or alike) testing • API Documentation
36. Reusability rarely pays! Creating a reusable asset is a lot harder and costs a lot more than most people think
37. Shirking the costs • Asset not implemented for being reused • Each “reusing” party needs to • Understand the functionality of the asset • Understand if it covers the given use case • Modify the asset or adopt the using parts • Test the resulting functionality • … • Increases costs and impedes reliability • Often leads to avoiding the asset and reimplementing it
38. Conclusion Creating a reusable asset costs 9 times as much as creating a regular project asset. You only start saving money the 10th time you reuse the asset.
39. Misconception #3 Reuse requires new architectural paradigms
40. “To foster reusability, we need to introduce <new architectural paradigm>” -- The unknown architect
41. The long version of the reasoning IT person (typically an architect) to business sponsor: “We definitely need to invest in implementing <cool new architectural paradigm>. I know this does not create any business value now, but it allows us to create reusable <architectural paradigm building blocks>! This way, we will have to pay a lot less for future projects, they will be a lot faster and we will be a lot more flexible. In short: the investment will pay for itself very soon.” Never worked this way, never will work this way!
42. Everything we need for decoupling and reusability is already in place for more than 60 years We call it “modularization” * * being available ubiquitously in IT, new paradigms are not needed to foster reuse
43. If you were not able to foster reuse with the means you have, what makes you think a new architectural paradigm will?
44. Reusability suitability Poor Great Fine-grained Stable Simple Universal Technical Coarse-grained Volatile Complicated Specific Business Reusability promoters Reusability inhibitors Collection classes Technical framework Embedded HTTP server Currency Account & Transaction Ledger Order handling Available through programming language ecosystems, OSS solutions and alike Subject of “reusability via new paradigm” discussions Could be provided via internal company-specific libraries or alike
45. All assets providing good reusability suitability properties already exist. Just download (or buy) and use them.
46. The assets worth being made reusable inside a company scope are usually ignored.
47. The new architectural paradigms usually target an area where aiming for reusability is not worth the effort
48. Conclusion You do not need to introduce a new architectural paradigm to foster reusability
49. Misconception #4 Distributed paradigms as reusability enabler
50. “To foster reusability, we need to introduce <new distributed architectural paradigm>” -- The unknown architect
51. The long version again … IT person (typically an architect) to business sponsor: “We definitely need to invest in implementing <cool new distributed architectural paradigm>. I know this does not create any business value now, but it allows us to create reusable <architectural paradigm building blocks>! This way, we will have to pay a lot less for future projects, they will be a lot faster and we will be a lot more flexible. In short: the investment will pay for itself very soon.” Now distributed hell breaks loose …
52. Distributed architectural paradigms • DCE • CORBA • EJB / DCOM • SOA • Microservices
53. Distributed systems in a nutshell
54. Everything fails, all the time. -- Werner Vogels
55. If X then Y What we learned in our IT education If X then maybe Y This changes everything! What we need for distributed systems We are good at this (due to how our brains work) Inside process thinking Reasoning about deterministic behavior Designing a complicated system We are poor at that (due to how our brains work) Reasoning about non-deterministic behavior Across process thinking Designing a complex system
56. Yet, we usually use deterministic thinking to reason about distributed systems
57. Failures in distributed systems ... • Crash failure • Omission failure • Timing failure • Response failure • Byzantine failure
58. ... lead to a variety of effects … • Lost messages • Incomplete messages • Duplicate messages • Distorted messages • Delayed messages • Out-of-order message arrival • Partial, out-of-sync local memory • ...
59. ... turning seemingly simple issues into very hard ones Time & Ordering Leslie Lamport "Time, clocks, and the ordering of events in distributed systems" Consensus Leslie Lamport ”The part-time parliament” (Paxos) CAP Eric A. Brewer "Towards robust distributed systems" Faulty processes Leslie Lamport, Robert Shostak, Marshall Pease "The Byzantine generals problem" Consensus Michael J. Fischer, Nancy A. Lynch, Michael S. Paterson "Impossibility of distributed consensus with one faulty process” (FLP) Impossibility Nancy A. Lynch ”A hundred impossibility proofs for distributed computing"
60. These effects are based on the non-determinism introduced by remote communication due to distributed failure modes
61. Understand that remote communication points are predetermined breaking points of your application
62. To minimize the non-deterministic effects of distributed systems you need to minimize coupling between distributed parts
63. The coupling dilemma • Reusability is tightest form of coupling possible • If a reused part fails, reusing parts fail, too • Availability requires avoiding cascading failures • If a part fails, the failure must not spread to other parts • Reusability means cascading failures by definition • Due to the tight coupling between reused and reusing part • Technical measures cannot avoid it • The tight coupling is at the functional level
64. Conclusion Aiming for reusability in a distributed system compromises system availability and response times by design
65. The value of reuse unfolds inside a process boundary Across process boundaries it should be avoided as it maximizes coupling
66. Misconception #5 Use means reuse
67. “But every time I call a remote service, I reuse that service” -- The unknown software engineer
68. Well, it depends …
69. Reusable asset Usable asset Asset Problem Solution Composition • Asset is part of the solution • Solution cannot work without the asset Collaboration • Asset is not part of the solution • Solution can work without the asset (if designed right) Asset Problem Solution
70. Usability is typically a result of horizontal decomposition, reusability a result of vertical decomposition
71. <uses> Functionality vertical decomposition (layer design, composition) In practice, you typically use a combination of both approaches Core functional decomposition approaches horizontal decomposition (pillar design, segregation)
72. Vertical decomposition • Based on functional decomposition • Typical drivers are reuse and avoidance of redundancy • Creates strong coupling (high functional dependence) • Often useful pattern inside a process boundary • Due to deterministic communication behavior • Problematic across process boundaries à Should be avoided in distributed design
73. Horizontal decomposition • Based on functional segregation • Typical drivers are autonomy and independence • Creates low coupling (high functional independence) • Useful pattern across process boundaries • Can also be useful inside a process boundary • Less accidental "ripple" effects due to changes à Should be preferred in distributed design
74. Watch out! • Vertical decomposition is our default design approach • We’ve learned it in our CS education (divide and conquer, ...) • It’s emphasized in our daily work (DRY, reusability, ...) • Even our IDEs support it (“Extract method”) • It's everywhere! It's predominant! • It takes energy not to apply vertical decomposition • Most people never learned horizontal decomposition
75. While the distinction between “use” and “reuse” seems to be irrelevant at first sight, mastering it is essential in practice
76. Conclusion Aim for usability across process boundaries (collaborating, functionally independent parts) * * and learn the associated design approaches (see, e.g. https://speakerdeck.com/ufried/getting-service-design-right)
77. Doing it better Recommendations
78. When not to go for reusability
79. Advice #1 Do not use reusability to justify the introduction of a new architectural paradigm
80. Reusability as business case • Neither works nor makes sense • Modularization is all you need, exists for more than 60 years • Usually leads to bad designs • Many tight couplings • Especially in conjunction with distributed paradigms • Bad for all people affected • Leads to cost savings pressure (the amortization promise) • Does not work as promised • Increased pressure – reinforces problems
81. Advice #2 Be wary of reusability if the problem you try to solve does not exhibit good reusability suitability properties
82. What to do instead
83. Advice #3 At a business case level, better argue with benefits like "increased business agility" or "better technology obsolescence management” * * especially if you want to introduce a a distributed modularization paradigm like microservices
84. Corollary to advice #3 Do not try to introduce a distributed modularization paradigm like microservices if cost reduction is your sole driver
85. Advice #4 In distributed systems prefer "usability" (collaboration) over "reusability" (composition)
86. Avoid dogmatism
87. Corollary to advices #2 and #4 Design advices are heuristics, not a dogma They are not a replacement for thinking
88. Going for reusability
89. Advice #5 Try to use existing reusable assets to increase your productivity Do not try to reinvent the wheel
90. Advice #6 If you solve a problem that has good reusability suitability properties and is needed in many places, consider creating a reusable asset * * if you can afford to pay the extra costs
91. Bonus advice Think about reusability in other areas of software engineering, e.g., on the level of ideas
92. Moving on
93. This was not an argument against reusability
94. This was an argument against using reusability in the wrong places and for the wrong reasons
95. Use reusability to continuously improve productivity
96. Problem 3 GL development base level Self-development base level New reusable assets created Reusability improves productivity and the scope of solvable problems over time Problem 3 GL development base level Self-development base level Recurring solution parts emerge Problem 3 GL development base level Reusable assets Self-developed solution part Self-development base level 3 GL development base level Self-development base level Less self- development needed Problem Self-developed solution part Reusable assets
97. Wrap-up
98. Wrap-up • Do not use reusability as sales argument for new paradigms • All you need for reusability is modularization • Avoid reusability across process boundaries • Learn the difference between “use” and “reuse” • Be willing to pay the price for creating reusable assets • But first consider reusability suitability • Use reusability as productivity driver • Reuse has a huge value used in the right places
99. References [Bro 1986] Frederick P. Brooks, jr., “No silver bullet”, from H.-J. Kugler (ed.), "Information Processing 1986: World Congress Proceedings (IFIP congress series, vol 10)", 1986 [Bro 1995] Frederick P. Brooks, jr., “The tar pit”, from “The mythical man-month”, anniversary edition 1995 [Ste 1974] W. P. Stevens, G. J. Myers, and L. L. Constantine, “Structured design”, IBM Systems Journal, Vol 13, No. 2, 1974
100. Uwe Friedrichsen Works @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://ufried.com/blog/reusability_fallacy_1/