Squaring the circle - Mastering the next level of architectural design

In this presentation, I discuss how architectural work changed in the last 10-15 years due to new and changed drivers of various sorts.

I start with architectural work as it was typically done in the early 2000s. Then I briefly touch how architectural work mostly stood still through the agile confusion due to some widespread misunderstandings regarding architecture and architectural work. After that, I discuss 12 drivers of the last 10-15 years that changed how we need to address architecture and architectural work. Finally, I provide a few recommendations how we could address the new challenges and hopefully turn them into opportunities.

As this a really big topic, the presentation only scratches the surface - creating awareness regarding the changed challenges in architectural work at best. Additionally, the voice track is lacking. Still, I hope it provides some food for thought.

Transcript

1. Squaring the circle
   Mastering the next level of architectural design
   Uwe Friedrichsen – codecentric AG – 2006-2022
   

   View Slide

2. Uwe Friedrichsen
   Works @ codecentric
   https://twitter.com/ufried
   https://www.speakerdeck.com/ufried
   https://ufried.com/
   

   View Slide

3. The “good ol’ times”
   Software architectural work until 2008
   

   View Slide

4. Typical setting (1/2)
   • Many internally used systems
   • Relatively few external systems
   • Mostly monolithic systems
   • Rich frontend apps or server-side rendered web pages
   • On-premises data centers
   

   View Slide

5. Typical setting (2/2)
   • Perimeter-based security
   • Pre-definable load patterns
   • “IT is a cost center” thinking
   • Effects of distribution handled by infrastructure
   

   View Slide

6. Typical mindset
   • “Single process, single thread” thinking
   • Availability demands handled by infrastructure
   • Security demands handled by infrastructure
   • Project-scope thinking
   • Integration as 2nd class citizen
   

   View Slide

7. Architectural work (1/3)
   • Design solution that satisfies project needs
   • Project cost efficiency as dominant design driver
   • Focus on system structure
   • Layered architecture was predominant
   

   View Slide

8. Architectural work (2/3)
   • Availability, security, scaling, … left to Ops
   • Standard, often mandatory solutions offered by Ops
   • Often respective NFRs determined by solutions offered
   • Performance as 2nd class citizen
   • Usability and architecture completely unrelated
   • Integration as 2nd class citizen
   • Either solved ad hoc or via centralized integration solution
   

   View Slide

9. Architectural work (3/3)
   • Core area of tension
   • Solution maintainability and changeability
   • Project cost efficiency
   • Speed of change usually constrained by project lead times
   • Typically measured in months or years
   

   View Slide

10. Architectural work summary
    • Dominated by project constraints
    • Focus on system structure
    • Required architectural work quite well understood
    • At least in theory
    • Still not easy in practice
    • Many solutions from that time as witnesses
    

    View Slide

11. Focus on structure
    Core area of tension
    Changeability
    Maintainability
    Flexibility
    Project efficiency
    Opposing goals
    Amplifying goals
    Usability
    Detached topic
    Performance
    Minor design
    consideration
    Availability
    Scalability
    Reliability
    Confidentiality Integrity
    Security
    “Ops problem”
    Dominant solution driver
    Portability
    Interoperability
    Minor design
    considerations
    

    View Slide

12. Architectural work typically meant
    „Create building block design and decide technology” *
    * within the predefined constraints
    

    View Slide

13. The age of confusion
    Software architectural work between 2008 and 2015
    

    View Slide

14. The age of confusion
    • The IT world went “Agile”
    • Developers were “The new kingmakers” *
    • Countermovement to the architecture-centric movement
    of the 1990s
    * https://www.oreilly.com/library/view/the-new-kingmakers/9781449368036/
    

    View Slide

15. “We do not need architects. The dev team is the architect.”
    “We do not need architectural work. Architecture emerges.”
    

    View Slide

16. IT changed a lot in those days
    

    View Slide

17. Architectural work mostly stood still
    

    View Slide

18. Focus on structure
    Core area of tension
    Changeability
    Maintainability
    Flexibility
    Project efficiency
    Opposing goals
    Amplifying goals
    Usability
    Detached topic
    Performance
    Minor design
    consideration
    Availability
    Scalability
    Reliability
    Confidentiality Integrity
    Security
    “Ops problem”
    Dominant solution driver
    Portability
    Interoperability
    Minor design
    considerations
    We were still here
    

    View Slide

19. Or sometimes even worse
    “From 0 to legacy in just 6 months”
    

    View Slide

20. The present
    Software architectural work since 2015
    

    View Slide

21. Novel drivers (1/3)
    • Cloud revolution *
    • Completely new solution options
    • Pay per use
    • Elasticity
    • Moving data around has a price tag
    • Availability and reliability on non-HA hardware
    * https://www.ufried.com/blog/public_cloud_revolution_1/
    

    View Slide

22. Changeability
    Maintainability
    Usability
    Performance
    Availability
    Scalability
    Reliability
    Confidentiality Integrity
    Security
    Portability
    Interoperability
    Flexibility
    Cloud revolution
    Project efficiency
    Opposing goals
    Amplifying goals
    Suitability
    New solution options
    Pay per use
    Interoperability
    Data flow
    Economic efficiency
    Data flow
    Elasticity
    Elasticity
    Scalability
    Resilience
    Availability
    Reliability
    Scale out
    on non-HA
    hardware
    Project efficiency
    

    View Slide

23. Novel drivers (1/3)
    • Cloud revolution
    • Big data & ML/AI
    • Huge amounts of data to store, move and process
    • Data flow becoming 1st class citizen of system design
    • Polyglot persistence
    • MLaaS/AIaaS
    

    View Slide

24. Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    Big data & ML/AI
    Project efficiency
    Suitability
    Data flow
    Resilience
    Project efficiency
    Economic efficiency
    Move, store &
    process data
    Mlaas/AIaas
    Economic efficiency
    Availability
    Scalability
    Reliability
    Interoperability
    Performance
    Process &
    move data
    Polyglot persistence
    Suitability
    Processing options
    Interoperability
    Move data
    Scalability
    Move, store &
    process data
    Store data
    Reliability
    Availability
    

    View Slide

25. Novel drivers (1/3)
    • Cloud revolution
    • Big data & ML/AI
    • Mobile devices
    • Completely new interaction patterns
    • Various novel device classes
    • Less reliable and often limited connectivity
    • Unpredictable load patterns
    

    View Slide

26. Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    Mobile devices
    Project efficiency
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Reliability
    Interoperability
    Performance
    Limited battery
    Scalability
    Unpredictable
    usage patterns
    Suitability
    New interaction patterns
    Usability
    Increased user expectations
    Adaptability
    Novel device classes
    Portability
    Novel attack vectors
    Security
    Confidentiality Integrity
    Availability
    Dependability
    Less reliable
    connectivity
    Versatility
    Manifold subclasses
    Interoperability
    Less reliable
    connectivity
    Changeability
    Increased user expectations
    

    View Slide

27. Novel drivers (1/3)
    • Cloud revolution
    • Big data & ML/AI
    • Mobile devices
    • IoT & Edge
    • Most of the effects of mobile devices “to the square”
    • Solutions enter safety-critical domains
    

    View Slide

28. Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    IoT & Edge
    Project efficiency
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Reliability
    Interoperability
    Performance
    Limited battery
    Suitability
    New interaction patterns
    Usability
    Novel device classes
    Portability
    Novel attack vectors Security
    Confidentiality Integrity
    Availability
    Dependability
    Less reliable
    connectivity
    Versatility
    Manifold subclasses
    Interoperability
    Less reliable
    connectivity
    Adaptability
    Safety Safety-critical
    applications
    

    View Slide

29. Novel drivers (2/3)
    • Distributed systems
    • Highly connected, distributed system landscape
    • Service-based applications, …
    • Intricate new failure types need to be handled
    • Effects of distribution penetrate to the application level
    • Near-realtime response expectations intensify effects
    

    View Slide

30. Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    Distributed systems
    Project efficiency
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Performance
    Latency
    Confidentiality Integrity
    Adaptability
    Safety
    Reliability
    Remote
    communication
    Security
    Safety
    Availability
    Dependability
    Distributed
    failure modes
    Reliability
    Resilience
    Interoperability
    Suitability
    Communication flow
    Usability
    Suitability
    Communication flow
    Interoperability
    Observability
    Distributed
    failure modes
    

    View Slide

31. Novel drivers (2/3)
    • Distributed systems
    • Near-realtime
    • Batch updates turn into online updates
    • Complex web of continuously communicating systems
    • Integration becomes first-class citizen
    • Near-realtime insights feedback loop
    • Intensifies effects of system distribution
    

    View Slide

32. Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    Near-realtime
    Project efficiency
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Performance
    Latency
    Confidentiality Integrity
    Adaptability
    Safety
    Reliability
    Availability
    Dependability
    Distribution
    failure modes
    Reliability
    Resilience
    Interoperability
    Integration as
    1st class citizen
    Observability
    

    View Slide

33. Novel drivers (2/3)
    • Distributed systems
    • Near-realtime
    • 24x7
    • Maintenance windows are gone
    • Overnight batch processing windows are gone
    • Upgrade/repair systems in-flight
    

    View Slide

34. Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality Integrity
    Security
    Portability
    Flexibility
    24x7
    Project efficiency
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality Integrity
    Adaptability
    Safety
    Reliability
    Availability
    Dependability
    In-flight system
    upgrade/repair
    Reliability
    Observability
    In-flight system
    upgrade/repair
    Observability
    Batch windows gone
    Performance
    

    View Slide

35. Novel drivers (2/3)
    • Distributed systems
    • Near-realtime
    • 24x7
    • Perimeter-less security
    • Trusted zones are gone: “The Internet is everywhere”
    • Remote work, BYOD, mobile devices, etc. intensify challenges
    • “Shift left security” mandatory
    

    View Slide

36. Observability
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Perimeter-less security
    Project efficiency
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    Observability
    Trusted zones gone
    Integrity
    Trusted zones gone
    Security
    Confidentiality
    

    View Slide

37. Novel drivers (3/3)
    • Digital transformation
    • “IT and business are the same side of the same coin;
    the other side are the market and your customers”
    • IT being integral part of all business offerings
    • IT being integral part of most customer interactions
    • IT enabling new, previously unthinkable business models
    • IT becoming indispensable part of business and private lives
    

    View Slide

38. Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Digital transformation
    Suitability
    Resilience
    Project efficiency
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    New business models Adaptability
    System lifecycle
    Products,
    not projects
    Economic efficiency
    Availability
    Dependability
    Integral part of
    business offerings
    Reliability
    Resilience
    Scalability
    Usability
    Suitability
    Integral part of
    customer interactions
    Integrity
    Security
    Confidentiality
    Safety
    Indispensable
    part of our lives
    Interoperability
    API-driven
    revenue streams
    Changeability
    New business models
    Flexibility
    Maintainability
    Human needs
    Socio-technical systems
    Affecting
    humans
    

    View Slide

39. Novel drivers (3/3)
    • Digital transformation
    • Post-industrial markets
    • Supply exceeds demand by far à customer-driven markets
    • Fast adoption to changing demands becomes vital
    • Lead and cycle times become essential
    • Continuous feedback on all levels needed
    • More complex multidisciplinary collaboration needed
    

    View Slide

40. Human needs
    Socio-technical systems
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Post-industrial markets
    Suitability
    Resilience
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    Fast adoption Adaptability
    Human needs
    Socio-technical systems
    Multidisciplinary
    collaboration
    Changeability
    Fast adoption
    Flexibility
    Maintainability
    Observability
    Observability Continuous feedback
    Usability
    Suitability
    Customer-driven market
    System lifecycle
    

    View Slide

41. Novel drivers (3/3)
    • Digital transformation
    • Post-industrial markets
    • Exploding complexity
    • Probably the 2nd biggest challenge of the upcoming years
    • IT drowns in complexity and every day it becomes worse
    • Due to shortsighted thinking
    • Due to important decisions not made
    

    View Slide

42. Gregor’s Law: Excessive complexity is nature’s punishment
    for organizations that are unable to make decisions.
    -- Gregor Hohpe
    Source: https://architectelevator.com/architecture/it-complexity/
    

    View Slide

43. Human needs
    Socio-technical systems
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Exploding complexity
    Suitability
    Resilience
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    System lifecycle
    Simplicity
    Tackling complexity
    System empathy
    System viability
    and collaboration
    

    View Slide

44. Novel drivers (3/3)
    • Digital transformation
    • Post-industrial markets
    • Exploding complexity
    • Ecological sustainability
    • Probably the biggest challenge of the upcoming years
    • Making sustainability 1st class citizen of system design
    • Counterforce to many other forces
    

    View Slide

45. Human needs
    Socio-technical systems
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Ecological sustainability
    Suitability
    Resilience
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    System lifecycle
    Fostering sustainability
    Sustainability
    Simplicity
    System empathy
    

    View Slide

46. Finally, let us add the amplifying and opposing forces
    

    View Slide

47. Human needs
    Socio-technical systems
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Opposing goals
    Amplifying goals
    Suitability
    Resilience
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    System lifecycle
    Simplicity
    System empathy
    Sustainability
    Project efficiency
    … and many more
    

    View Slide

48. All quality goals need to be balanced
    according to the given needs *
    * needs of different stakeholder groups, short-term and long-term needs, amplifying vs. opposing goals, …
    

    View Slide

49. All influencing forces and the derived quality goals
    vastly influence the resulting solution architecture
    

    View Slide

50. Observations
    What has changed in the last decade?
    

    View Slide

51. Human needs
    Socio-technical systems
    Integrity
    Integrity
    Dependability
    Versatility
    Changeability
    Maintainability
    Usability
    Performance
    Confidentiality
    Security
    Portability
    Flexibility
    Opposing goals
    Amplifying goals
    Suitability
    Resilience
    Economic efficiency
    Availability
    Scalability
    Interoperability
    Confidentiality
    Adaptability
    Safety
    Reliability
    Observability
    System lifecycle
    Simplicity
    System empathy
    Sustainability
    Project efficiency
    Focus on structure
    not sufficient anymore
    Data flow
    Communication flow
    Limited view on projects
    not sufficient anymore
    Economic perspective
    over lifecycle as 1st
    class citizen
    Prevalent complexity of
    IT system landscapes
    requires new design
    approaches
    Exploding number
    of device classes
    Integration as
    1st class citizen
    Thinking in
    feedback loops
    Holistic view required.
    Focus on software alone
    not sufficient anymore
    Novel drivers of future system
    design. Counterforces to many
    other design forces
    Enterprise software
    affects safety-critical
    systems more and more
    Security as 1st class
    citizen in design
    (“Shift left security”)
    Huge new
    challenges due
    to highly
    interconnected
    distributed
    system
    landscapes
    (“microservices”)
    Unpredictable
    load patterns
    Massively increased user
    expectations influence
    system design
    Technology evolution provides
    new options and changes what
    a feasible solution is
    

    View Slide

52. Lots of new forces. Lots of new tasks and challenges.
    Partially completely novel forces and resulting tasks.
    

    View Slide

53. Reinforces long know, yet still unsolved problems
    

    View Slide

54. Unsolved problems
    • Getting functional design right (“modularization”)
    • Pondering behavior and data flow, not just structure
    • Systemic thinking instead of local optimizations
    • Balancing long-term and short-term effects
    • Understanding distributed systems
    • …
    

    View Slide

55. This is a completely
    different game ..
    … than that
    

    View Slide

56. Useful side effect
    It is acceptable again to talk about architecture *
    * and use words like “architect” or “architectural work” … ;)
    

    View Slide

57. Moving on
    Addressing the new challenges
    

    View Slide

58. The collaboration perspective
    

    View Slide

59. “Emergent architecture” is dead
    

    View Slide

60. Tackling the challenges of today
    requires hard work and careful decision making
    

    View Slide

61. Does that mean the “all-knowing” architect is back?
    

    View Slide

62. No!
    But you need the skills and someone who drives the architectural work that needs to be done.
    

    View Slide

63. Even the smartest architect imaginable will not be able
    to address all these challenges alone
    

    View Slide

64. In the end, architectural work always is a team effort
    

    View Slide

65. Does that mean the BDUF * is back?
    * Big Design UpFront
    

    View Slide

66. No!
    But you need to have an idea where the goal is before you start running.
    

    View Slide

67. Implementation without any
    upfront architectural work
    Implementation with some
    upfront architectural work
    

    View Slide

68. View Slide

69. View Slide

70. Sometimes, a new or previously undetected requirement
    may take us by surprise
    

    View Slide

71. View Slide

72. But basically, your architecture will evolve over time
    

    View Slide

73. View Slide

74. View Slide

75. View Slide

76. Never forget Gall’s Law
    

    View Slide

77. “A complex system designed from scratch never works
    and cannot be made to work. You have to start over,
    beginning with a working simple system.”
    -- John Gall
    Source: John Gall, "Systemantics", 1975
    

    View Slide

78. The skills perspective
    

    View Slide

79. Learn about architectural work
    • Understand the problem from all perspectives
    • Find alternative solution options
    • Evaluate the trade-offs
    • Help stakeholders make best decisions possible
    

    View Slide

80. Improve your RE skills
    • Need to understand demands and consequences
    • Impeded by widespread division-of-labor approach
    • Help find requirements that do not harm the design
    • Often, alternatives with same outcome are possible
    • If not, you need to change the design of course
    • Uncover the actual “needs” behind the “wants”
    • Helps to avoid lots of accidental complexity
    

    View Slide

81. Learn better functional design
    • Especially master the foundations of good design
    • “Loose coupling, high cohesion” & “separation of concerns”
    • “Information hiding” especially with respect to APIs
    • Domain-driven design may be a useful starting point
    • Good design skills are more relevant than ever
    • Affect understandability, changeability and extensibility
    • Affect usability, stability and acceptance at API level
    • Affect robustness, availability and scalability at runtime
    

    View Slide

82. “I would advise students to pay more attention to the
    fundamental ideas rather than the latest technology.
    The technology will be out-of-date before they graduate.
    Fundamental ideas never get out of date.”
    -- David L. Parnas (http://www.sigsoft.org/SEN/parnas.html)
    

    View Slide

83. Embrace distributed systems
    “(Almost) every system is a distributed system” -- Chas Emerick
    “Everything fails, all the time” -- Werner Vogels
    • Understand the non-determinism of distributed systems
    • “Memory, guesses and apologies” & promise theory
    • Understand that infrastructure cannot guarantee robustness
    

    View Slide

84. Help people understand software
    • Not comparable to physical goods
    • Needs to be changed after initial release to keep its value
    • Software development is design, not shop floor assembly
    • Extreme malleability of software can turn into a curse
    • Invisibility makes it very hard to grasp “software”
    

    View Slide

85. Understand the new challenges
    • Simplicity – tackling ever-growing complexity
    • System empathy – accepting heterogeneity
    • Dependability – making sure we all can rely on IT
    • Resilience – being prepared for the unexpected
    • Safety – dealing with the growing blast radius
    • Security – indispensable for dependable system
    • Sustainability – reducing the ecological footprint of IT
    

    View Slide

86. Wrap-up
    

    View Slide

87. Wrap-up
    • Architectural challenges have changed a lot
    • Complexity went up significantly
    • New counterforces have emerged
    • Decision making has become a lot more demanding
    • Updated perception of architectural work needed
    • New collaboration modes required
    • Different skills have become vital
    

    View Slide

88. "Ultimately, design is a tool to enhance our humanity”
    -- Ilse Crawford
    

    View Slide

89. Uwe Friedrichsen
    Works @ codecentric
    https://twitter.com/ufried
    https://www.speakerdeck.com/ufried
    https://ufried.com/
    

    View Slide