Averting Tragedy on the Architectural Commons (ScanDev 2013)

Much has been made of the need to establish software architectures to provide the firm foundations on which successful products are constructed, but if a product is successful over the long term its architecture will not only need to evolve, but must be actively defended against malignant forces. The fact that software architectures tend to outlive the tenure of developers, architects, management teams and even companies makes the maintenance of software architectures over the long term crucial for the ability of products to deliver ongoing value. What happens to the architecture of established systems in an environment where concurrent feature delivery projects make architectural ‘adjustments’ for their own ends? How can erosion of this shared architecture be managed across multiple teams? How can projects acting in their own best interests avoid over-exploiting the common architectural resource on which they all depend? Should ownership of the architecture be distributed and shared, or centralised and tightly controlled? Perhaps most importantly, can we detect when the architecture has been violated? This talk explores how shared resources in other fields are managed for the common good, and draws analogies and lessons which can be applied to the shared ‘resource’ of a software architecture. Examples garnered from over twelve years working with greenfield and legacy software systems illustrate how to diagnose, understand the causes of, and address the erosion of application architectures so that products can flourish and be productive for future generations.

Transcript

1. Avoiding Tragedy
   on the Architectural Commons
   1
   Robert Smallshire
   Roxar Software Solutions, Oslo
   @robsmallshire
   Tuesday, 5 March, 13
   

   View Slide

2. Systems and their architectures are long lived
   Lifetimes in the software industry
   2
   Developers
   Windows XP
   Applications
   CEOs
   Lines of code
   FTSE100
   Classes
   Modules
   0 15 30 45 60
   58
   37
   22
   13
   6.8
   6.2
   4.7
   3.1
   Sources: W3schools, Software Lifetime and its Evolution Process over Generations, CEO Succession Practices: 2012 Edition, Investors Chronicle,
   Half-lives of software related entities
   The number of years over which half the entities are replaced
   Tuesday, 5 March, 13
   

   View Slide

3. 3
   “Farming looks mighty easy
   when your plow is a pencil and
   you're a thousand miles from
   the corn field.” Dwight Eisenhower
   Tuesday, 5 March, 13
   

   View Slide

4. 4
   Tuesday, 5 March, 13
   

   View Slide

5. 5
   Sustainable Development
   Provision meets ongoing needs whilst preserving the supporting
   environment.
   Tuesday, 5 March, 13
   

   View Slide

6. 6
   Tuesday, 5 March, 13
   

   View Slide

7. 7
   The tragedy of the commons
   What is the tragedy and how does it relate to software architecture?
   The causes and effects of architectural erosion
   How to recognise and detect compromised architectures.
   Defending the architecture
   Managing the architecture and its stakeholders.
   1
   2
   3
   Tuesday, 5 March, 13
   

   View Slide

8. Garrett Hardin
   8
   “Resources held for common use by all will
   ultimately be destroyed.”
   Hardin, G. (1968) The Tragedy of the Commons. Science 162 1243-1248
   “There is no technical solution.”
   Tuesday, 5 March, 13
   

   View Slide

9. 9
   Tuesday, 5 March, 13
   

   View Slide

10. 10
    Tuesday, 5 March, 13
    

    View Slide

11. Potential and actual tragedies
    The Commons
    11
    Global
    Climate
    Fisheries
    Groundwater
    Grazing
    Habitat
    Forests
    Radio
    Spectrum
    Atmosphere
    The
    Internet
    Roads
    Parks
    Museums
    Silence
    Software
    Systems
    Welfare
    State
    Tuesday, 5 March, 13
    

    View Slide

12. 12
    The tragedy of the commons
    What is the tragedy and how does it relate to software architecture?
    The causes and effects of architectural erosion
    How to recognise and detect compromised architectures.
    Defending the architecture
    Managing the architecture and its stakeholders.
    1
    2
    3
    Tuesday, 5 March, 13
    

    View Slide

13. What takes us from this ...
    How does this happen?
    13
    Tuesday, 5 March, 13
    

    View Slide

14. ... to this?
    How does this happen?
    14
    Tuesday, 5 March, 13
    

    View Slide

15. The tragedy unfolds inexorably, one dependency at a time.
    How does this happen?
    15
    using App.Foo.Bar
    import App.Foo.Bar
    from App.Foo import Bar
    #include
    require “app/foo/bar.h”
    Tuesday, 5 March, 13
    

    View Slide

16. has second order effect on outcomes
    Organization
    16
    Command
    and Control
    Self
    Organising
    Tuesday, 5 March, 13
    

    View Slide

17. Large numbers of smart people - does smartness scale?
    17
    Organization
    Tuesday, 5 March, 13
    

    View Slide

18. Smartness is not an additive quantity.
    18
    Organization
    Tuesday, 5 March, 13
    

    View Slide

19. 100
    1000
    10000
    100000
    1000 10000 100000 1000000 10000000
    Productivity (Lines of Code / Year)
    Total Lines of Code
    Use published productivity
    data to forward model code
    size.
    Modelling team and code evolution
    19
    Sources: COCOMO II
    At any given system size we
    can predict a distribution for
    developer productivity.
    Tuesday, 5 March, 13
    

    View Slide

20. 20
    3 years
    54 kLoC
    Overall Employees : 4
    Team Size : 2
    LoC : 54 k
    Author present : 60%
    Tuesday, 5 March, 13
    

    View Slide

21. 21
    3 years
    152 kLoC
    Cumulative team size : 11 ± 2 @ 1σ
    Team Size : 7
    LoC : 157 k ± 23 k @ 1σ
    Author present : 70% ± 14% @ 1σ
    Tuesday, 5 March, 13
    

    View Slide

22. 22
    20 years
    1.8 MLoC
    Cumulative team size : 114 ± 9 @ 1σ
    Team Size : 21
    LoC : 1.8 M ± 0.08 M @ 1σ
    Author present : 19% ± 4% @ 1σ
    Tuesday, 5 March, 13
    

    View Slide

23. Most authors of your product quit way back when.
    Who can you still talk to?
    23
    days
    Proportion of code
    written by current
    colleagues.
    20% after
    20 years
    Tuesday, 5 March, 13
    

    View Slide

24. Step back and take in
    the big picture
    Detecting compromised architectures
    24
    Tuesday, 5 March, 13
    

    View Slide

25. 25
    Tuesday, 5 March, 13
    

    View Slide

26. How is complexity distributed in your system?
    Mapping complexity
    26
    Tuesday, 5 March, 13
    

    View Slide

27. How are defects distributed in your system?
    Mapping quality
    27
    Tuesday, 5 March, 13
    

    View Slide

28. Do you understand your actual dependencies?
    28
    Dependency Structure
    Tuesday, 5 March, 13
    

    View Slide

29. 29
    Tuesday, 5 March, 13
    

    View Slide

30. 30
    Tuesday, 5 March, 13
    

    View Slide

31. Reap all the immediate benefit but share the deferred cost
    31
    Introduced dependencies
    Tuesday, 5 March, 13
    

    View Slide

32. 32
    Tuesday, 5 March, 13
    

    View Slide

33. 33
    “any situation in which one person makes the
    decision about how much risk to take, while
    someone else bears the cost if things go badly”
    Paul Krugman, Economist
    Tuesday, 5 March, 13
    

    View Slide

34. 34
    The tragedy of the commons
    What is the tragedy and how does it relate to software architecture?
    The causes and effects of architectural erosion
    How to recognize and detect compromised architectures.
    Defending the architecture
    Managing the architecture and its stakeholders.
    1
    2
    3
    Tuesday, 5 March, 13
    

    View Slide

35. We have met the enemy and he is us.
    Tragedy on the Architectural Commons
    35
    Short Tenure
    Developer turnover outpaces
    change of the software system.
    Moral Hazard
    Developers acting rationally for
    local gain aren’t exposed to the
    deferred and global cost of their
    actions.
    The Tragedy
    The common environment of the
    software architecture erodes, making
    ongoing development more frustrating
    and costly. Productivity falls.
    1
    2
    3
    Tragic
    Cycle
    Tuesday, 5 March, 13
    

    View Slide

36. Often used in combination
    Solutions to the tragedy
    36
    Education Encourage ‘enlightened self-interest’. Foster cooperation through
    the architecture.
    Privatisation Create a ‘core’ team who own the architecture. Does this confuse
    architecture with infrastructure?
    Taxation Tax feature projects by requiring them to contribute beyond their
    narrow project scope.
    Regulation Give clear ownership to a central authority. Introduce regulating
    software.
    Tuesday, 5 March, 13
    

    View Slide

37. Education
    37
    ‣ Enlightened self interest
    • “do well by doing good”
    ‣ Deferred gratification
    • sacrifice short-term interests in favour of
    long-term interests
    Tuesday, 5 March, 13
    

    View Slide

38. The code is the design?
    “most complete, unambiguous, and
    irreducible expression of a design”
    38
    The code is the most,
    Maybe. But only the de facto design,
    not the intended design.
    Tuesday, 5 March, 13
    

    View Slide

39. Privatisation
    39
    ‣ Transfer ownership of the architecture
    to a single organisation
    • ownership obligates stewardship
    • acts on behalf of feature teams, who must
    rent or purchase services
    • does this confuse architecture with
    infrastructure?
    Tuesday, 5 March, 13
    

    View Slide

40. 40
    Creating a core team who owns
    the architecture usually results
    in another feature team.
    It’s hard to maintain stakeholder
    support for infrastructure or
    design artefacts.
    Tuesday, 5 March, 13
    

    View Slide

41. Taxation
    41
    ‣ Introduce transaction costs
    • attach an immediate cost to potentially
    detrimental changes
    • changes reflect their long-term, shared cost
    • discourage frivolous or merely convenient
    dependencies
    Tuesday, 5 March, 13
    

    View Slide

42. 42
    Not all changes should be
    treated equally
    Tuesday, 5 March, 13
    

    View Slide

43. 43
    Not all changes should be
    treated equally
    Have new dependencies require
    review from a wider constituency
    Tuesday, 5 March, 13
    

    View Slide

44. Regulation
    44
    ‣ Central authority
    • architects are easy to ignore
    • architects can’t be omniscient
    • architects should encode design rules
    in such a way that they can’t be ignored
    Tuesday, 5 March, 13
    

    View Slide

45. 45
    Encode design constraints into
    automated systems:
    • vigilant build systems
    • commit triggers
    • custom static analysis tools
    Tuesday, 5 March, 13
    

    View Slide

46. 46
    The microeconomics of change
    By understanding the incentives which drive architectural erosion
    we can avoid tragic decline.
    Tuesday, 5 March, 13
    

    View Slide

47. Dependencies and complexity are too cheap in the short term
    Confront the moral hazard
    47
    Short Tenure
    Developer turnover outpaces
    change of the software system.
    Moral Hazard
    Developers acting rationally for
    local gain aren’t exposed to the
    deferred cost of their actions, of
    technical debt.
    The Tragedy
    The common environment of the system
    erodes, making ongoing development
    more frustrating and costly. Productivity
    falls.
    1
    2
    3
    Tragic
    Cycle
    Tuesday, 5 March, 13
    

    View Slide

48. 48
    Tuesday, 5 March, 13
    

    View Slide

49. Thank you
    Questions?
    49
    Robert Smallshire
    Roxar Software Solutions, Oslo
    @robsmallshire
    starting in May
    Tuesday, 5 March, 13
    

    View Slide