$30 off During Our Annual Pro Sale. View Details »

Architecting for Scale

SQUER Solutions
September 19, 2023
36

Architecting for Scale

Vienna Microservice Meetup

SQUER Solutions

September 19, 2023
Tweet

Transcript

  1. @duffleit
    The Art of
    Architecting for Scale
    @duffleit

    View Slide

  2. @duffleit
    DAVID LEITNER
    Coding Architect
    👋 [email protected]
    🌎 @duffleit

    View Slide

  3. @duffleit
    📦
    👧 🧑 👧 🧑 👧
    🔥
    Safely and sustainably reduce
    lead time to thank you.
    Daniel Terhorst-North
    Complex System

    View Slide

  4. @duffleit
    Simple System

    View Slide

  5. @duffleit
    Complicated System

    View Slide

  6. @duffleit
    Complicated System

    View Slide

  7. @duffleit
    Complicated System
    Module A Module B
    Module C

    View Slide

  8. @duffleit
    Complicated System
    Distributed
    Systems
    Service A Service B
    Service C

    View Slide

  9. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud

    View Slide

  10. @duffleit
    “Sadly, architecture has
    been undervalued for so
    long that many
    engineers regard life
    with a Big Ball of Mud
    as normal.“
    Foote & Yoder

    View Slide

  11. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    Distributed
    MOnolith
    WelL-Structured
    MOdulith

    View Slide

  12. @duffleit
    There is a multitude of reasons
    to go for Distributed Systems.
    Modularisation is none of them.

    View Slide

  13. @duffleit
    If you can’t build a well-structured
    monolith, what makes you think
    microservices are the answer?
    Simon Brown

    View Slide

  14. @duffleit
    Module A Module B
    Module C
    Single Deployment Unit

    View Slide

  15. @duffleit
    Service A Service B
    Serivice C

    View Slide

  16. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    10x Users
    Usual Traffic

    View Slide

  17. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    Technology
    Segmentation

    View Slide

  18. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    Technology
    Segmentation
    🇩🇪
    🇩🇪
    🇺🇸
    CO-Locating

    View Slide

  19. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    Technology
    Segmentation
    CO-Locating
    Technical
    Benefits

    View Slide

  20. @duffleit
    Service A Service B
    Serivice C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it

    View Slide

  21. @duffleit
    Single Deployment Unit
    MOdule A MOdule B
    MOdule C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it
    🔥

    View Slide

  22. @duffleit
    Single Deployment Unit
    MOdule A MOdule B
    MOdule C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it
    🔥
    🔥

    View Slide

  23. @duffleit
    Service A Service B
    Serivice C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Enable Teams to
    work autonomously

    View Slide

  24. @duffleit
    Technical
    Benefits
    Team
    Autonomoy
    Scale Engineering
    CO-Locating
    Technology
    Segmentation
    Individual
    Scaling Demand
    80%

    View Slide

  25. @duffleit
    We optimize for Autonomy.

    View Slide

  26. @duffleit
    Responsibility
    The ops will
    hate it Zone
    The Devs will
    Quit Zone
    Autonomy
    You build it
    you Own it
    You build it
    you run it
    You build it
    you care it

    View Slide

  27. @duffleit
    Responsibility
    The ops will
    hate it Zone
    The Devs will
    Quit Zone
    Autonomy
    You build it
    you Own it
    You build it
    you run it
    You build it
    you care it

    View Slide

  28. @duffleit
    We optimize for Responsibility.
    Therefore, teams strive for Autonomy.

    View Slide

  29. @duffleit
    👧 🧑
    🧑 👧 🧑
    🧑
    👧 🧑
    🧑
    📦

    View Slide

  30. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑

    View Slide

  31. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS

    View Slide

  32. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Decomposition Fallacy

    View Slide

  33. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisation
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Decomposition Fallacy

    View Slide

  34. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Decomposition Fallacy

    View Slide

  35. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisation
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Decomposition Fallacy

    View Slide

  36. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Decomposition Fallacy

    View Slide

  37. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisation
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Change in Cause of Failure
    Reliabilty

    View Slide

  38. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Change in Cause of Failure
    Reliabilty Availability

    View Slide

  39. @duffleit
    What’s the difference between a
    method call within a single deployment unit

    View Slide

  40. @duffleit
    What’s the difference between a
    method call within a single deployment unit
    Deployment Unit
    Module A Module B
    moduleB.createUser() fun createUser() { /*...*/ }

    View Slide

  41. @duffleit
    What’s the difference between a
    method call within a single deployment unit,
    and a method call across the network.
    Deployment Unit
    Deployment Unit
    Service A Service B
    restClient.user() fun createUser() { /*...*/ }

    View Slide

  42. @duffleit
    What’s the difference between a
    method call within a single deployment unit,
    and a method call across the network.
    Everything.

    View Slide

  43. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴

    View Slide

  44. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

    View Slide

  45. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑

    View Slide

  46. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔥

    View Slide

  47. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    🤯

    View Slide

  48. @duffleit
    To enable teams to strive for Autonomy.
    We need to Optimize for Availability.

    View Slide

  49. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Change in Cause of Failure
    Reliabilty Availability

    View Slide

  50. @duffleit
    Deployment Units
    Monolithic Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    SYSTEMS
    Change in Cause of Failure
    Reliabilty Availability

    View Slide

  51. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    🔴
    Complex Distributed System
    👧 🧑
    🧑
    Loan
    Team
    Lending System

    View Slide

  52. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    🔴
    🔥
    Complex Distributed System
    👧 🧑
    🧑
    🟢
    Account Team
    Loan
    Team
    Lending System

    View Slide

  53. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    Complex Distributed System
    👧 🧑
    🧑
    🔴
    Loan
    Team
    Lending System
    Account Team

    View Slide

  54. @duffleit
    We strive for a stable environment.

    View Slide

  55. @duffleit
    There is no stable environment.

    View Slide

  56. @duffleit
    There is no stable environment.
    * The sooner we accept this, the better.

    View Slide

  57. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Account Service

    View Slide

  58. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    🔥
    Users[]
    🔥

    View Slide

  59. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    Users[]
    🔥

    View Slide

  60. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    Users[]
    cache

    View Slide

  61. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    Users[]
    stream
    UserChanged

    View Slide

  62. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    readmodel
    stream
    Users[]
    UserChanged

    View Slide

  63. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Account Service
    readmodel
    stream
    Users[]
    UserChanged

    View Slide

  64. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Account Service
    readmodel
    stream
    🔥🔥
    🔥
    🔥
    Users[]

    View Slide

  65. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Account Service
    🔥
    🚀
    Caching
    LBs
    CBs

    View Slide

  66. @duffleit
    Chaotic Environments
    Architect for
    🔥

    View Slide

  67. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Account Service
    🔥
    🔥
    🔥
    🔥

    View Slide

  68. @duffleit
    we need to
    start dacining
    with the beast
    and embrace Choas.

    View Slide

  69. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑
    Gamedays
    🤡
    🤡
    🤡

    View Slide

  70. @duffleit
    A Services
    that can run in a
    stable environment
    can run only there.

    View Slide

  71. @duffleit
    A Services
    that can run in a
    chaotic environment
    can run everywhere.

    View Slide

  72. @duffleit
    Chaotic Environments
    Architect for
    🔥

    View Slide

  73. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Team
    🧑
    Account Team
    Execute
    Payment
    Payment
    Gateway
    Team

    View Slide

  74. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    Payment Team
    🧑
    Account Team
    🤯

    View Slide

  75. @duffleit
    <
    👧 🧑
    🧑
    SEPA Payment Team
    Account Team
    CREDITCARD
    Payment
    Team
    <
    👧 🧑
    🧑
    🧑
    👧 🧑

    View Slide

  76. @duffleit
    <
    👧 🧑
    🧑
    End2End Responsibility
    SEPA Payment Team
    Account Team
    CREDITCARD
    Payment
    Team
    <
    End2End Responsibility
    👧 🧑
    🧑
    🧑
    👧 🧑
    😌

    View Slide

  77. @duffleit
    <
    👧 🧑
    🧑
    End2End Responsibility
    SEPA Payment Team
    Account Team
    CREDITCARD
    Payment
    Team
    <
    End2End Responsibility
    👧 🧑
    🧑
    🧑
    👧 🧑
    😌
    We need to keep
    depenedencies in mind
    Accounts

    View Slide

  78. @duffleit
    Customer Journey and Entity Data
    Mismatch Problem

    View Slide

  79. @duffleit
    Customer
    Journey
    Entities
    User Accounts Transactions
    User
    Onboarding
    Payment
    Execution
    Fraud
    Detection

    View Slide

  80. @duffleit
    <
    👧 🧑
    🧑
    End2End Responsibility
    Account Team
    <
    End2End Responsibility
    👧 🧑
    🧑
    🧑
    👧 🧑
    Accounts

    View Slide

  81. @duffleit
    <
    👧 🧑
    🧑
    End2End Responsibility
    Account Team
    <
    End2End Responsibility
    👧 🧑
    🧑
    🧑
    👧 🧑
    Accounts
    cache
    🔥

    View Slide

  82. @duffleit
    <
    👧 🧑
    🧑
    End2End Responsibility
    Account Team
    <
    End2End Responsibility
    👧 🧑
    🧑
    🧑
    👧 🧑
    Accounts
    Account
    Changed
    readmodel
    🔥

    View Slide

  83. @duffleit
    End-2-End Responsibilities.
    Architect for
    🛶

    View Slide

  84. @duffleit
    End-to-End
    Autonomy

    Dependencies
    Decrease
    Cognitive Load

    View Slide

  85. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    🧑
    👧 🧑

    View Slide

  86. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    🧑
    👧 🧑
    💁🙋

    View Slide

  87. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    🧑
    👧 🧑
    Availability
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    MTBF

    View Slide

  88. @duffleit
    Can our stream aligned team still have a
    99,99% availability
    if our complex subsystem dependency has a
    99,9% availability?

    View Slide

  89. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    🧑
    👧 🧑
    Availability
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    rescheduler

    View Slide

  90. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    🧑
    👧 🧑
    Availability
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    🔄
    rescheduler

    View Slide

  91. @duffleit
    <
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Check
    Payment
    Complicated
    Subsystem
    👧 🧑
    🧑
    Availability
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    🔄
    rescheduler
    Platform Teams
    Developer Experience
    SLOs

    View Slide

  92. @duffleit
    Pain = Strength x Distance x Volatility
    Vladik Khononov
    Dependencies
    their three main aspects:

    View Slide

  93. @duffleit
    Pain = Strength x Distance x Volatility
    Vladik Khononov
    Dependencies

    View Slide

  94. @duffleit
    Dependencies as a Service
    Architect for
    9⃣

    View Slide

  95. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Legacy
    System
    👧 🧑
    🧑
    👧 🧑
    🧑

    View Slide

  96. @duffleit
    <
    SEPA Payment Team
    Account Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Legacy
    System
    👧 🧑
    🧑

    View Slide

  97. @duffleit
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers

    View Slide

  98. @duffleit
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API

    View Slide

  99. @duffleit
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API
    Downtime

    View Slide

  100. @duffleit
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API
    Downtime

    View Slide

  101. @duffleit
    Readers
    Writers
    Readers
    Writers
    API
    User Fraud
    Data
    🔥

    View Slide

  102. @duffleit
    Readers
    Writers
    Readers
    Writers
    API
    User Fraud
    Data
    🔥

    View Slide

  103. @duffleit
    User Fraud
    Data
    Readers
    Writers
    API
    🔥
    Readers
    Writers

    View Slide

  104. @duffleit
    User Fraud
    Data
    Readers
    Writers
    API
    Readers
    Writers

    View Slide

  105. @duffleit
    Event
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API
    User Fraud
    Data
    CDC

    View Slide

  106. @duffleit
    Event
    User Fraud
    Data
    Readers
    Readers
    Writers
    API
    User Fraud
    Data
    Event
    Writers

    View Slide

  107. @duffleit
    Event
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API
    User Fraud
    Data
    Event
    🔥

    View Slide

  108. @duffleit
    Event
    User Fraud
    Data
    Readers
    Readers
    Writers
    API
    User Fraud
    Data
    Event
    Writers
    👷

    View Slide

  109. @duffleit
    Event
    User Fraud
    Data
    Readers
    Writers
    Readers
    Writers
    API
    User Fraud
    Data
    Event

    View Slide

  110. @duffleit
    Event
    User Fraud
    Data
    Readers
    Writers
    API
    User Fraud
    Data
    Event
    Readers
    Writers

    View Slide

  111. @duffleit
    CDC
    User Fraud
    Data
    Readers
    Writers
    API
    User Fraud
    Data
    Event
    Readers
    Writers
    Canary Releases
    Zero Downtime
    Migration
    Dark Launching
    Scientist Pattern

    View Slide

  112. @duffleit
    Change Anxiety
    Architect against
    🦜

    View Slide

  113. @duffleit
    So many thing to consider,
    Let’s sum up.

    View Slide

  114. @duffleit
    Modularisation is no
    reason to go for
    Distributed Systems.
    Deployment Units
    Monolithic Distributed
    Reliabilty Availability
    Autonomy is.

    View Slide

  115. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🧑

    View Slide

  116. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🧑
    Chaotic Environments
    Architect for
    🔥

    View Slide

  117. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🧑
    Chaotic Environments
    Architect for
    🔥
    End-2-End Responsibilities
    Architect for
    🛶

    View Slide

  118. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🧑
    Chaotic Environments
    Architect for
    🔥
    End-2-End Responsibilities
    Architect for
    🛶
    COMPLICATED
    Subsystem
    Teams
    Dependencies as a Service
    Architect for
    9⃣

    View Slide

  119. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🧑
    Chaotic Environments
    Architect for
    🔥
    End-2-End Responsibilities
    Architect for
    🛶
    COMPLICATED
    Subsystem
    Teams
    Change Anxiety
    Architect against
    🦜
    Dependencies as a Service
    Architect for
    9⃣

    View Slide

  120. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔥

    View Slide

  121. @duffleit
    📦
    👧 🧑
    🧑
    📦
    📦
    keep the flow of value high
    & the number of outages low.
    👧 🧑
    🧑
    👧 🧑
    🧑

    View Slide

  122. @duffleit
    Keep the lead time
    to thank you low,
    even at scale.

    View Slide

  123. @duffleit
    DAVID LEITNER
    Principal Engineer
    👋 [email protected]
    🌎 @duffleit

    View Slide