Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The reusability fallacy

The reusability fallacy

This slide deck discusses reusability, a topic that is sort of a "holy grail" in software architecture. Unfortunately, reusability often is discussed in the wrong places and for the wrong reasons, especially if it is used to sell a new architectural paradigm ("via reuse, the initial investment will amortize very soon and it will pay for itself").

I start with the origins of the topic in the early 1970s (and the different conditions how software was developed back then). Then I discuss and debunk the most widespread fallacies that are typically used to press for reusability.

With that knowledge I present a few advices (not dogmas!) how to deal better with reusability - when not to use it, what to do instead of those places, and: When to use it!

Reusability has a huge value in software development and we need to make sure to harvest it. But pressing for it in the wrong places for the wrong reasons only causes problems and harm. We need to avoid that and the intention of the talk is to have better arguments if you encounter such a situation.

This is the unabridged slide deck containing all slides of the talk. For conferences, I usually leave out several slides to meet the session length constraints.

I also accompanied this slide deck with a blog series that can be found at https://ufried.com/blog/reusability_fallacy_1/. The series consists of 4 posts, covers the contents of this talk and provides a few more ideas and pointers.

E698a765c9d04ae52d5e1815b2007cfe?s=128

Uwe Friedrichsen

November 04, 2020
Tweet

Transcript

  1. The reusability fallacy Killing a beloved dogma Uwe Friedrichsen –

    codecentric AG – 2008-2020
  2. Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://ufried.com/blog/reusability_fallacy_1/

  3. The “holy grail” of architecture

  4. The promise (the dream) • Create software module once •

    Reuse as often as you like at no reproduction cost • “Invest once, harvest forever” • Big efficiency lever • Huge cost saver
  5. The origins

  6. Structured design by W. P. Stevens, G. J. Myers and

    L. L. Constantine [Ste 1974]
  7. “Considerations and techniques are proposed that reduce the complexity of

    programs by dividing them into functional modules. This can make it possible to create complex systems from simple, independent, reusable modules. […] And, as the module library grows, increasingly sophisticated programs can be implemented using less and less new code.” [Ste 1974]
  8. The origins • Reusability discussion started ~1970 • Most companies

    and projects started at scratch • Barely more than 3GL compiler and linker available • No big standard libraries bundled with language • No big ecosystems available around language • Even most basic functionalities needed to be developed • Reusability could increase productivity significantly
  9. Reusability today (the fallacy) • Became an end in itself

    • “Reusability is needed. Period!” • Demanded by managers to improve cost efficiency • “I want reusability to improve the IT cost ratio” • Used by architects to sell new architectural paradigms • “Required investment will amortize soon due to reuse” à Never worked this way. Never will work this way.
  10. Still, reusability is widely popular. Thus, where is the misconception?

  11. Actually, it is a mix of misconceptions

  12. Misconception #1 Industrial production analogies

  13. “Let us apply the reusability approaches from house building or

    car manufacturing” -- The unknown “industrialization of IT” advocate
  14. Reuse ≈ Standardization

  15. Standardization • Standardization helps building at reduced price • Same

    part can be reused in many products • Saves production costs over time • Design efforts only once • Production process design and set up only once • Required tooling created and installed only once • Cost savings are realized in production • Spreads design and set up costs over more parts produced
  16. Software development has nothing in common with producing a physical

    good
  17. The products we build are not source code, but executable

    programs
  18. Our assembly lines are so efficient for more than 50

    years that most people forgot they even exist We call them “compiler” * * or CI/CD pipelines in the settings of today
  19. Software development is about designing the product The design is

    only complete after the last line of code is written * * Building the product from the design (source code) is merely pushing a button
  20. We cannot make software production more cost-efficient by building more

    parts from the same design
  21. Our production process is already unbelievably cheap * and we

    can produce multiple copies from a design at zero cost We call it the “copy command” * compared to the production process of any physical goods
  22. All the metaphors comparing software development to house building or

    car assembly are just plain wrong
  23. Conclusion Potential cost or efficiency improvements of reusability are limited

    to design time
  24. New (industrially minted) idea ”The more reusable parts we use

    in our design process (a.k.a. software development), the more time, costs and effort we can save”
  25. No silver bullet by Frederick P. Brooks, Jr. [Bro 1986]

  26. “Software entities are more complex for their size than perhaps

    any other human construct, because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into one, a subroutine, open or closed. In this respect software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound. [...] Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger size; it is necessarily an increase in the number of different elements.” [Bro 1986]
  27. Reusing the same part repeatedly does not work

  28. Reuse is limited to lower-level functionalities supporting higher-level functionalities, still

    resulting in a significant increase of different parts
  29. Misconception #2 Reusability comes for free

  30. “Each reusable part saves me a lot of money!” --

    The unknown decision maker
  31. Well, ever considered the cost of creating a reusable asset?

  32. Reusability rarely pays! Creating a reusable asset is a lot

    harder and costs a lot more than most people think
  33. The tar pit by Frederick P. Brooks, Jr. (taken from

    the “The mythical man-month”) [Bro 1995]
  34. “One occasionally reads newspaper accounts of how two programmers in

    a remodeled garage have built an important program that surpasses the best efforts of large teams. And every programmer is prepared to believe such tales, for he knows that he could build any program much faster than the 1000 statements/year reported for industrial teams. Why then have not all industrial programming teams been replaced by dedicated garage duos? One must look at what is being produced.” [Bro 1995]
  35. 1x Program The original module, suitable for the context it

    was created for 3x Programming Product A generalized module, suitable for multiple contexts A module, ready to be used in an ecosystem of interacting modules A (re-)usable module, that provides a general solution for a problem 3x Programming System 9x Programming Systems Product
  36. 1x Program 9x Programming Systems Product Completeness of functionality •

    Handling of edge cases • Hardening implementation • Thorough functional testing • Design Documentation Completeness of accessibility • Precise interface definition • Clear behavioral contract • Thorough integration (or alike) testing • API Documentation
  37. Shirking the costs • Asset not implemented for being reused

    • Each “reusing” party needs to • Understand the functionality of the asset • Understand if it covers the given use case • Modify the asset or adopt the using parts • Test the resulting functionality • … • Increases costs and impedes reliability • Often leads to avoiding the asset and reimplementing it
  38. Conclusion Creating a reusable asset costs 9 times as much

    as creating a regular project asset. You only start saving money the 10th time you reuse the asset.
  39. Misconception #3 Reuse requires new architectural paradigms

  40. “To foster reusability we need to introduce <new architectural paradigm>”

    -- The unknown architect
  41. The long version of the reasoning IT person (typically an

    architect) to business sponsor: “We definitely need to invest in implementing <cool new architectural paradigm>. I know this does not create any business value now, but it allows us to create reusable <architectural paradigm building blocks>! This way, we will have to pay a lot less for future projects, they will be a lot faster and we will be a lot more flexible. In short: the investment will pay for itself very soon.” Never worked this way, never will work this way!
  42. Everything we need for decoupling and reusability is already in

    place for more than 60 years We call it “modularization” * * being available ubiquitously in IT, new paradigms are not needed to foster reuse
  43. If you were not able to foster reuse with the

    means you have, what makes you think a new architectural paradigm will?
  44. Reusability suitability Poor Great Fine-grained Stable Simple Universal Technical Coarse-grained

    Volatile Complicated Specific Business Reusability promoters Reusability inhibitors Collection classes Technical framework Embedded HTTP server Currency Account & Transaction Ledger Order handling Available through programming language ecosystems, OSS solutions and alike Subject of “reusability via new paradigm” discussions Could be provided via internal company-specific libraries or alike
  45. All assets providing good reusability suitability properties already exist. Just

    download (or buy) and use them.
  46. The assets worth being made reusable inside a company scope

    are usually ignored.
  47. The new architectural paradigms usually target an area where aiming

    for reusability is not worth the effort
  48. Conclusion You do not need to introduce a new architectural

    paradigm to foster reusability
  49. Misconception #4 Distributed paradigms as reusability enabler

  50. “To foster reusability we need to introduce <new distributed architectural

    paradigm>” -- The unknown architect
  51. The long version again … IT person (typically an architect)

    to business sponsor: “We definitely need to invest in implementing <cool new distributed architectural paradigm>. I know this does not create any business value now, but it allows us to create reusable <architectural paradigm building blocks>! This way, we will have to pay a lot less for future projects, they will be a lot faster and we will be a lot more flexible. In short: the investment will pay for itself very soon.” Now distributed hell breaks loose …
  52. Distributed architectural paradigms • DCE • CORBA • EJB /

    DCOM • SOA • Microservices
  53. Distributed systems in a nutshell

  54. Everything fails, all the time. -- Werner Vogels

  55. If X then Y What we learned in our IT

    education If X then maybe Y This changes everything! What we need for distributed systems We are good at this (due to how our brains work) Inside process thinking Reasoning about deterministic behavior Designing a complicated system We are poor at that (due to how our brains work) Reasoning about non-deterministic behavior Across process thinking Designing a complex system
  56. Yet, we usually use deterministic thinking to reason about distributed

    systems
  57. Failures in distributed systems ... • Crash failure • Omission

    failure • Timing failure • Response failure • Byzantine failure
  58. ... lead to a variety of effects … • Lost

    messages • Incomplete messages • Duplicate messages • Distorted messages • Delayed messages • Out-of-order message arrival • Partial, out-of-sync local memory • ...
  59. ... turning seemingly simple issues into very hard ones Time

    & Ordering Leslie Lamport "Time, clocks, and the ordering of events in distributed systems" Consensus Leslie Lamport ”The part-time parliament” (Paxos) CAP Eric A. Brewer "Towards robust distributed systems" Faulty processes Leslie Lamport, Robert Shostak, Marshall Pease "The Byzantine generals problem" Consensus Michael J. Fischer, Nancy A. Lynch, Michael S. Paterson "Impossibility of distributed consensus with one faulty process” (FLP) Impossibility Nancy A. Lynch ”A hundred impossibility proofs for distributed computing"
  60. These effects are based on the non-determinism introduced by remote

    communication due to distributed failure modes
  61. Understand that remote communication points are predetermined breaking points of

    your application
  62. To minimize the non-deterministic effects of distributed systems you need

    to minimize coupling between distributed parts
  63. The coupling dilemma • Reusability is tightest form of coupling

    possible • If a reused part fails, reusing parts fail, too • Availability requires avoiding cascading failures • If a part fails, the failure must not spread to other parts • Reusability means cascading failures by definition • Due to the tight coupling between reused and reusing part • Technical measures cannot avoid it • The tight coupling is on the functional level
  64. Conclusion Aiming for reusability in a distributed system compromises system

    availability and response times by design
  65. The value of reuse unfolds inside a process boundary Across

    process boundaries it should be avoided as it maximizes coupling
  66. Misconception #5 Use means reuse

  67. “But every time I call a remote service, I reuse

    that service” -- The unknown software engineer
  68. Well, it depends …

  69. Reusable asset Usable asset Asset Problem Solution Composition • Asset

    is part of the solution • Solution cannot work without the asset Collaboration • Asset is not part of the solution • Solution can work without the asset (if designed right) Asset Problem Solution
  70. Usability is typically a result of horizontal decomposition, Reusability a

    result of vertical decomposition
  71. <uses> Functionality vertical decomposition (layer design, composition) In practice, you

    typically use a combination of both approaches Core functional decomposition approaches horizontal decomposition (pillar design, segregation)
  72. Vertical decomposition • Based on functional decomposition • Typical drivers

    are reuse and avoidance of redundancy • Creates strong coupling (high functional dependence) • Often useful pattern inside a process boundary • Due to deterministic communication behavior • Problematic across process boundaries à Should be avoided in distributed design
  73. Horizontal decomposition • Based on functional segregation • Typical drivers

    are autonomy and independence • Creates low coupling (high functional independence) • Useful pattern across process boundaries • Can also be useful inside a process boundary • Less accidental "ripple" effects due to changes à Should be preferred in distributed design
  74. Watch out! • Vertical decomposition is our default design approach

    • We’ve learned it in our CS education (divide and conquer, ...) • It’s emphasized in our daily work (DRY, reusability, ...) • Even our IDEs support it (“Extract method”) • It's everywhere! It's predominant! • It takes energy not to apply vertical decomposition • Most people never learned horizontal decomposition
  75. While the distinction between “use” and “reuse” seems to be

    irrelevant at first sight, mastering it is essential in practice
  76. Conclusion Aim for usability across process boundaries (collaborating, functionally independent

    parts) * * and learn the associated design approach (“horizontal decomposition”)
  77. Doing it better Recommendations

  78. When not to go for reusability

  79. Advice #1 Do not use reusability to justify the introduction

    of a new architectural paradigm
  80. Reusability as business case • Neither works nor makes sense

    • Modularization is all you need, exists for more than 60 years • Usually leads to bad designs • Many tight couplings • Especially in conjunction with distributed paradigms • Bad for all people affected • Leads to cost savings pressure (the amortization promise) • Does not work as promised • Increased pressure – reinforces problems
  81. Advice #2 Be wary of reusability if the problem you

    try to solve does not exhibit good reusability suitability properties
  82. What to do instead

  83. Advice #3 On a business case level, better argue with

    benefits like "increased business agility" or "better technology obsolescence management” * * especially if you want to introduce a a distributed modularization paradigm like microservices
  84. Corollary to advice #3 Do not try to introduce a

    distributed modularization paradigm like microservices if cost reduction is your sole driver
  85. Advice #4 In distributed systems prefer "usability" (collaboration) over "reusability"

    (composition)
  86. Avoid dogmatism

  87. Corollary to advices #2 and #4 Design advices are heuristics,

    not a dogma They are not a replacement for thinking
  88. Going for reusability

  89. Advice #5 Try to use existing reusable assets to increase

    your productivity Do not try to reinvent the wheel
  90. Advice #6 If you solve a problem that has good

    reusability suitability properties and is needed in many places, consider creating a reusable asset * * if you can afford to pay the extra costs
  91. Bonus advice Think about reusability in other areas of software

    engineering, e.g., on the level of ideas
  92. Moving on

  93. This was not an argument against reusability

  94. This was an argument against using reusability in the wrong

    places and for the wrong reasons
  95. Use reusability to continuously improve productivity

  96. Problem 3 GL development base level Self-development base level New

    reusable assets created Reusability improves productivity and the scope of solvable problems over time Problem 3 GL development base level Self-development base level Recurring solution parts emerge Problem 3 GL development base level Reusable assets Self-developed solution part Self-development base level 3 GL development base level Self-development base level Less self- development needed Problem Self-developed solution part Reusable assets
  97. Wrap-up

  98. Wrap-up • Do not use reusability as sales argument for

    new paradigms • All you need for reusability is modularization • Avoid reusability across process boundaries • Learn the difference between “use” and “reuse” • Be willing to pay the price for creating reusable assets • But first consider reusability suitability • Use reusability as productivity driver • Reuse has a huge value used in the right places
  99. References [Bro 1986] Frederick P. Brooks, jr., “No silver bullet”,

    from H.-J. Kugler (ed.), "Information Processing 1986: World Congress Proceedings (IFIP congress series, vol 10)", 1986 [Bro 1995] Frederick P. Brooks, jr., “The tar pit”, from “The mythical man-month”, anniversary edition 1995 [Ste 1974] W. P. Stevens, G. J. Myers, and L. L. Constantine, “Structured design”, IBM Systems Journal, Vol 13, No. 2, 1974
  100. Uwe Friedrichsen CTO @ codecentric https://twitter.com/ufried https://www.speakerdeck.com/ufried https://ufried.com/blog/reusability_fallacy_1/