Architecting For Scale

CraftConf 2023

Transcript

1. @duffleit
   The Art of
   Architecting for Scale
   @duffleit
   

   View Slide

2. @duffleit
   DAVID LEITNER
   Principal Engineer
   👋 david@squer.io
   🌎 @duffleit
   

   View Slide

3. @duffleit
   📦
   👧 🧑 👧 🧑 👧
   🔥
   Safely and sustainably reduce
   lead time to thank you.
   Daniel Terhorst-North
   Complex System
   

   View Slide

4. @duffleit
   🔥
   🔴
   Simple System
   

   View Slide

5. @duffleit
   🥳
   🟢
   Simple System
   

   View Slide

6. @duffleit
   Simple System
   

   View Slide

7. @duffleit
   Complicated System
   🔥
   🔴
   

   View Slide

8. @duffleit
   Complicated System
   

   View Slide

9. @duffleit
   Complicated System
   Module A Module B
   Module C
   

   View Slide

10. @duffleit
    Complicated System
    Distributed
    Systems
    Service A Service B
    Service C
    

    View Slide

11. @duffleit
    Deployment Units
    Monolithical Distributed
    Modularisa3on
    Bad
    Well
    Big Ball
    Of Mud
    

    View Slide

12. @duffleit
    “Sadly, architecture has
    been undervalued for so
    long that many
    engineers regard life
    with a BIG BALL OF MUD
    as normal.“
    Foote & Yoder
    

    View Slide

13. @duffleit
    Deployment Units
    Monolithical Distributed
    Modularisation
    Bad
    Well
    Big Ball
    Of Mud
    Distributed
    MOnolith
    WelL-Structured
    MOdulith
    

    View Slide

14. @duffleit
    There is a multitude of reasons
    to go for Distributed Systems.
    Modularisation is none of them.
    

    View Slide

15. @duffleit
    Module A Module B
    Module C
    Single Deployment Unit
    

    View Slide

16. @duffleit
    Service A Service B
    Serivice C
    

    View Slide

17. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    10x Users
    Usual Traffic
    

    View Slide

18. @duffleit
    Service A Service B
    Serivice C
    Individual
    Scaling Demand
    Technology
    Segmentation
    

    View Slide

19. @duffleit
    Service A Service B
    Serivice C
    

    View Slide

20. @duffleit
    Service A Service B
    Serivice C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it
    

    View Slide

21. @duffleit
    Single Deployment Unit
    MOdule A MOdule B
    MOdule C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it
    🔥
    

    View Slide

22. @duffleit
    Single Deployment Unit
    MOdule A MOdule B
    MOdule C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Ownership
    You build it,
    You Own it
    You build it,
    You Run it
    🔥
    🔥
    

    View Slide

23. @duffleit
    Service A Service B
    Serivice C
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑 Take Ownership
    and Responsibility
    

    View Slide

24. @duffleit
    👧 🧑
    🧑👧 🧑
    🧑
    👧 🧑
    🧑
    📦
    

    View Slide

25. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    

    View Slide

26. @duffleit
    Deployment Units
    Monolithical Distributed
    Modularisation
    Bad
    Well
    Big Ball
    Of Mud
    WelL-Structured
    Modulith
    Distributed
    Monolith
    Autonomous
    Service-Based
    Architecture
    

    View Slide

27. @duffleit
    Deployment Units
    Monolithical Distributed
    Modularisation
    Bad
    Well
    Complex
    Distributed
    System
    Complicated
    Monolithical
    System
    

    View Slide

28. @duffleit
    👧 🧑
    🧑👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    🔥
    🟢
    Complicated Monolithical System
    Lending
    System
    Age
    Calculation
    

    View Slide

29. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    🔴
    Complex Distributed System
    Lending
    System
    Loan
    Team
    

    View Slide

30. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    🔴
    🔴
    Complex Distributed System
    Lending
    System
    Loan
    Team
    

    View Slide

31. @duffleit
    Rela%on Between
    Root Cause and Effect
    Complex
    Distributed
    System
    Complicated
    Monolithical
    System
    is usually
    given
    is usually
    not given
    

    View Slide

32. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    🔴
    🔴
    Complex Distributed System
    

    View Slide

33. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    Complex Distributed System
    

    View Slide

34. @duffleit
    What’s the difference between a
    method call within a single deployment unit
    

    View Slide

35. @duffleit
    What’s the difference between a
    method call within a single deployment unit
    Deployment Unit
    Module A Module B
    moduleB.createUser() fun createUser() { /*...*/ }
    

    View Slide

36. @duffleit
    What’s the difference between a
    method call within a single deployment unit,
    and a method call across the network.
    Deployment Unit
    Deployment Unit
    Service A Service B
    restClient.user() fun createUser() { /*...*/ }
    

    View Slide

37. @duffleit
    What’s the difference between a
    method call within a single deployment unit,
    and a method call across the network.
    Everything.
    

    View Slide

38. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    Complex Distributed System
    https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
    

    View Slide

39. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔴
    Complex Distributed System
    

    View Slide

40. @duffleit
    Rela%on Between
    Root Cause and Effect
    Complex
    Distributed
    System
    Complicated
    Monolithical
    System
    is usually
    given
    is usually
    not given
    Cause of Failures reliability availability
    

    View Slide

41. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    

    View Slide

42. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔥
    

    View Slide

43. @duffleit
    📦
    📦
    👧 🧑
    🧑
    📦
    👧 🧑
    🧑
    👧 🧑
    🧑
    🤯
    

    View Slide

44. @duffleit
    Relation Between
    Root Cause and Effect
    Complex
    Distributed
    System
    Complicated
    Monolithical
    System
    is usually
    given
    is usually
    not given
    Cause of Failures reliability availability
    

    View Slide

45. @duffleit
    Relation Between
    Root Cause and Effect
    Complex
    Distributed
    System
    Complicated
    Monolithical
    System
    is usually
    given
    is usually
    not given
    Cause of Failures reliability availability
    

    View Slide

46. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    🔴
    Complex Distributed System
    👧 🧑
    🧑
    Online
    Shopping System
    Store
    Team
    Warehouse Team
    

    View Slide

47. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    🔴
    🔴
    Complex Distributed System
    👧 🧑
    🧑
    🟢
    Store
    Team
    Warehouse Team
    

    View Slide

48. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    🧑
    Complex Distributed System
    👧 🧑
    🧑
    🔴
    Store
    Team
    Warehouse Team
    

    View Slide

49. @duffleit
    We strive for a stable production.
    But this is totally elusive
    in a complex environment.
    and sometimes chaotic
    The sooner we accept this,
    the better.
    

    View Slide

50. @duffleit
    We need to architect
    with Failure in Mind.
    

    View Slide

51. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Accout Service
    Online
    Shopping System
    

    View Slide

52. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    🔥
    Users[]
    🔥
    

    View Slide

53. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    Users[]
    🔥
    

    View Slide

54. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    Users[]
    cache
    

    View Slide

55. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    Users[]
    stream
    UserChanged
    

    View Slide

56. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    projection
    stream
    Users[]
    UserChanged
    

    View Slide

57. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Service
    🧑
    Accout Service
    projection
    stream
    Users[]
    UserChanged
    

    View Slide

58. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Accout Service
    projection
    stream
    🔥🔥
    🔥
    🔥
    Users[]
    

    View Slide

59. @duffleit
    👧 🧑
    🧑
    👧 🧑
    👧 🧑
    🧑
    Payment Service
    🧑
    Accout Service
    🔥
    🚀
    LBs
    CBs
    Caching
    

    View Slide

60. @duffleit
    We need to architect
    with Failure, and for Scale, in Mind.
    

    View Slide

61. @duffleit
    Services that can run in a
    chaotic environment can run everywhere.
    Services that can only run in
    stable environment can only run there.
    

    View Slide

62. @duffleit
    👧 🧑
    🧑
    🔥
    👧 🧑
    👧 🧑
    🧑
    🔴
    Payment Team
    🧑
    Accout Team
    Execute
    Payment
    Payment
    Gateway
    Team
    

    View Slide

63. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    Payment Team
    🧑
    Accout Team
    🤯
    

    View Slide

64. @duffleit
    <
    👧 🧑
    🧑
    🔥
    SEPA Payment Team
    Accout Team
    Execute
    Payment
    CREDITCARD
    Payment
    Team
    <
    👧 🧑
    🧑
    🧑
    👧 🧑
    

    View Slide

65. @duffleit
    <
    👧 🧑
    🧑
    🔥
    SEPA Payment Team
    Accout Team
    Execute
    Payment
    CREDITCARD
    Payment
    Team
    <
    COMPLICATED process
    👧 🧑
    🧑
    COMPLICATED process
    🧑
    👧 🧑
    😌
    

    View Slide

66. @duffleit
    👧 🧑
    🧑
    🔥
    🔴
    Payment Team
    Accout Team
    Execute
    Payment
    Payment
    Gateway
    Team
    COMPLEX process
    👧 🧑
    🧑
    👧 🧑
    🧑
    

    View Slide

67. @duffleit
    We need to architect
    for End-2-End Customer Journeys.
    

    View Slide

68. @duffleit
    <
    SEPA Payment Team
    Accout Team
    FRAUD
    CHECK
    <
    🧑
    👧 🧑
    👧 🧑
    🧑
    Core
    Team
    👧 🧑
    🧑
    

    View Slide

69. @duffleit
    <
    SEPA Payment Team
    Accout Team
    Check
    Payment
    FRAUD
    CHECK
    <
    Core
    Team
    🧑
    👧 🧑
    👧 🧑
    🧑
    👧 🧑
    🧑
    How many
    Nines?
    Availability?
    Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    A Planed Outage
    is an Outage
    

    View Slide

70. @duffleit
    Can our service have
    99,99% availability
    if our dependencies has a
    99,9% availability?
    

    View Slide

71. @duffleit
    <
    SEPA Payment Team
    Accout Team
    Check
    Payment
    FRAUD
    CHECK
    <
    🧑
    👧 🧑
    👧 🧑
    🧑
    Core
    Team
    👧 🧑
    🧑
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    A Planed Outage
    is an Outage
    rescheduler
    Availability?
    

    View Slide

72. @duffleit
    <
    SEPA Payment Team
    Accout Team
    Check
    Payment
    FRAUD
    CHECK
    <
    🧑
    👧 🧑
    👧 🧑
    🧑
    Core
    Team
    👧 🧑
    🧑
    How many
    Nines? Nines Percentage Yearly Outage
    1 Nine 90% 36,5 days
    2 Nines 99% 3,65 days
    3 Nines 99,9% 8,76 hours
    4 Nines 99,99% 52,56 minutes
    5 Nines 99,999% 5,26 minutes
    6 Nines 99,9999% 31,5 seconds
    🔴
    Availability?
    

    View Slide

73. @duffleit
    <
    SEPA Payment Team
    Accout Team
    Check
    Payment
    FRAUD
    CHECK
    <
    🧑
    👧 🧑
    👧 🧑
    🧑
    Core
    Team
    👧 🧑
    🧑
    MTTF?
    Platform teams
    SLAs
    

    View Slide

74. @duffleit
    We need to architect
    based on Availability.
    

    View Slide

75. @duffleit
    <
    SEPA Payment Team
    Accout Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Legacy
    System
    👧 🧑
    🧑
    👧 🧑
    🧑
    

    View Slide

76. @duffleit
    <
    SEPA Payment Team
    Accout Team
    FRAUD
    CHECK
    <
    👧 🧑
    🧑
    Legacy
    System
    👧 🧑
    🧑
    

    View Slide

77. @duffleit
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    User Fraud
    Data
    ⏳ Downtime Given
    

    View Slide

78. @duffleit
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    User Fraud
    Data
    🔥
    

    View Slide

79. @duffleit
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    User Fraud
    Data
    🔥
    

    View Slide

80. @duffleit
    CDC
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    User Fraud
    Data
    

    View Slide

81. @duffleit
    CDC
    Readers Writers
    Writers
    Readers
    API
    User Fraud
    Data
    Events
    User Fraud
    Data
    

    View Slide

82. @duffleit
    CDC
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    Events
    User Fraud
    Data
    🔥
    

    View Slide

83. @duffleit
    CDC
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    Events
    User Fraud
    Data
    👷
    

    View Slide

84. @duffleit
    CDC
    Readers
    Writers
    Writers
    Readers
    API
    User Fraud
    Data
    Events
    User Fraud
    Data
    Zero-Downtime
    Migration
    

    View Slide

85. @duffleit
    We need to architect
    with a non-negotiable Zero-Downtime Mindset.
    

    View Slide

86. @duffleit
    We need to
    Start Dacining
    with the Beast
    

    View Slide

87. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    Complex Distributed System
    👧 🧑
    🧑
    Gamedays
    🤡
    🤡
    🤡
    Payment Team
    

    View Slide

88. @duffleit
    We embrace Chaos
    in our architecture.
    

    View Slide

89. @duffleit
    So many thing to consider,
    Let’s sum up.
    

    View Slide

90. @duffleit
    Complex
    Distributed
    System
    Deployment Units
    Monolithical Distributed
    Complicated
    Monolithical
    System
    Modularisation is
    no reason to go
    for Distributed
    Systems.
    

    View Slide

91. @duffleit
    Failure and Scale in Mind.
    Architect with
    🔥
    End-2-End Customer Journeys
    Architect for
    🪢
    By the given Availability
    Architect driven
    🟢
    A non-negotiable Zero-Downtime Mindset.
    Architect around
    😌
    Embracing Chaos
    Architect by
    💣
    

    View Slide

92. @duffleit
    👧 🧑
    🧑
    👧 🧑
    🧑
    👧 🧑
    🧑
    🔥
    

    View Slide

93. @duffleit
    📦
    👧 🧑
    🧑
    📦
    📦
    Keep the Flow of Value high
    & the Amount of Outages low.
    👧 🧑
    🧑
    👧 🧑
    🧑
    

    View Slide

94. @duffleit
    Keep the lead time
    to Thank You low,
    even at scale.
    

    View Slide

95. @duffleit
    DAVID LEITNER
    Principal Engineer
    👋 david@squer.io
    🌎 @duffleit
    

    View Slide

96. @duffleit
    squer.link/arch-scale
    WORKSHOP
    Architecting for Scale
    Free Friends & Family DryRun
    

    View Slide