Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Chaos and Microservices Mayhem - presented at OOP Digital

Cloud Chaos and Microservices Mayhem - presented at OOP Digital

The cloud is just someone else's data center, but it has fundamentally changed how we design software and what we expect from our platforms. Our applications have gotten bigger, more distributed, and more complicated, and there are whole new categories of mistakes we can make. Some things that were a good idea ten years ago turn out to be a terrible idea in the cloud; and what used to be ‘good enough’ for testing really isn’t anymore. Managing microservices architecture demands a lot of us, to ensure observability, operational resiliency, and organisational agility. With a focus on Java, this talk will introduce some of the new tools, patterns, and best practices for modern distributed application development. It also gives a tour of some of the most painful anti-patterns Holly has seen as a cloud consultant.

Holly Cummins

February 07, 2023
Tweet

More Decks by Holly Cummins

Other Decks in Programming

Transcript

  1. cloud chaos and
    microservices mayhem




    Holly Cummins


    Red Hat


    @holly_cummins
    OOP Virtual

    View Slide

  2. View Slide

  3. View Slide

  4. #RedHat
    @holly_cummins
    things you
    need to do
    well in 2023:


    View Slide

  5. #RedHat
    @holly_cummins
    handwashing (still)
    things you
    need to do
    well in 2023:


    View Slide

  6. #RedHat
    @holly_cummins
    handwashing (still)
    apps
    things you
    need to do
    well in 2023:


    View Slide

  7. #RedHat
    @holly_cummins
    handwashing (still)
    apps
    ops
    things you
    need to do
    well in 2023:


    View Slide

  8. #RedHat
    @holly_cummins
    handwashing (still)
    apps
    ops
    devops
    things you
    need to do
    well in 2023:


    View Slide

  9. #RedHat
    @holly_cummins
    handwashing (still)
    apps
    ops
    devops
    devsecops
    things you
    need to do
    well in 2023:


    View Slide

  10. #RedHat
    @holly_cummins
    handwashing (still)
    apps
    ops
    devops
    devsecops
    finops
    things you
    need to do
    well in 2023:


    View Slide

  11. View Slide

  12. View Slide

  13. #RedHat
    @holly_cummins
    a few

    View Slide

  14. #RedHat
    @holly_cummins

    View Slide

  15. #RedHat
    @holly_cummins
    technology changes fast (duh)

    View Slide

  16. #RedHat
    @holly_cummins
    technology changes fast (duh)
    … but it’s taking us a while to catch up to cloud

    View Slide

  17. #RedHat
    @holly_cummins
    technology changes fast (duh)
    … but it’s taking us a while to catch up to cloud

    View Slide

  18. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    tracing

    View Slide

  19. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    observability

    View Slide

  20. #RedHat
    @holly_cummins
    hardware
    old way
    logs

    View Slide

  21. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs

    View Slide

  22. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs

    View Slide

  23. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    oops

    View Slide

  24. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs

    View Slide

  25. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs

    View Slide

  26. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs

    View Slide

  27. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs

    View Slide

  28. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs

    View Slide

  29. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    aggregation + correlation

    View Slide

  30. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    aggregation + correlation
    open telemetry
    (open tracing)

    View Slide

  31. #RedHat
    @holly_cummins

    View Slide

  32. #RedHat
    @holly_cummins
    good news: observability is ‘fixed’

    View Slide

  33. #RedHat
    @holly_cummins
    good news: observability is ‘fixed’
    bad news: not all of us are using it

    View Slide

  34. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    performance requirements

    View Slide

  35. #RedHat
    @holly_cummins
    memory footprint is money


    startup time is money

    View Slide

  36. #RedHat
    @holly_cummins
    all that dynamism we worked so
    hard is (now) a bad idea

    View Slide

  37. #RedHat
    @holly_cummins
    ahead of time optimisation
    is (now) a good idea

    View Slide

  38. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    performance optimisation

    View Slide

  39. #RedHat
    @holly_cummins

    View Slide

  40. #RedHat
    @holly_cummins
    garbage collection and microservices

    View Slide

  41. #RedHat
    @holly_cummins
    garbage collection and microservices

    View Slide

  42. #RedHat
    @holly_cummins
    garbage collection and microservices

    View Slide

  43. #RedHat
    @holly_cummins
    garbage collection and microservices

    View Slide

  44. #RedHat
    @holly_cummins
    garbage collection and microservices
    if you chain 100 microservices together,
    you are almost guaranteed a GC pause

    View Slide

  45. #RedHat
    @holly_cummins

    View Slide

  46. #RedHat
    @holly_cummins

    View Slide

  47. #RedHat
    @holly_cummins
    the JVM tunes itself differently on servers

    View Slide

  48. #RedHat
    @holly_cummins
    the JVM tunes itself differently on servers
    … but what’s a server?

    View Slide

  49. #RedHat
    @holly_cummins
    answer(ish):

    View Slide

  50. #RedHat
    @holly_cummins
    // This is the working definition of a server class machine:
    // >= 2 physical CPU's and >=2GB of memory
    answer(ish):
    actual hotspot source code

    View Slide

  51. #RedHat
    @holly_cummins

    View Slide

  52. #RedHat
    @holly_cummins
    takeaway: don’t shrink your containers too
    much or your JVM will make bad decisions
    advice accurate as of June 2022. all performance advice should be independently verified. do not try this at home without testing first. your mileage may vary.

    View Slide

  53. #RedHat
    @holly_cummins
    wait, isn’t that the
    opposite of the advice in the
    previous section? should my
    containers be big or
    small?

    View Slide

  54. #RedHat
    @holly_cummins
    this is the chaos part :)
    wait, isn’t that the
    opposite of the advice in the
    previous section? should my
    containers be big or
    small?

    View Slide

  55. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    management

    View Slide

  56. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    keeping track of hardware

    View Slide

  57. #RedHat
    @holly_cummins
    The cloud makes it so easy to
    provision hardware.

    View Slide

  58. #RedHat
    @holly_cummins
    That doesn’t mean
    the hardware is free.

    View Slide

  59. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.

    View Slide

  60. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.
    Or useful.

    View Slide

  61. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.
    Or useful.
    Or shuts itself off.

    View Slide

  62. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.

    View Slide

  63. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.
    I forgot it for 2 months.

    View Slide

  64. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.
    I forgot it for 2 months.
    … and it’s €1000 a month.

    View Slide

  65. Photo by daveynin, flickr

    View Slide

  66. #RedHat
    @holly_cummins
    2017 survey


    25%


    of 16,000 servers
    doing no useful work

    View Slide

  67. #RedHat
    @holly_cummins
    2017 survey


    25%


    of 16,000 servers
    doing no useful work
    “perhaps someone
    forgot to turn them off”

    View Slide

  68. #RedHat
    @holly_cummins
    finops


    figuring out who in
    your company forgot
    to turn off their cloud

    View Slide

  69. #RedHat
    @holly_cummins
    companies that went out of
    business for cloud cost accidents

    View Slide

  70. @holly_cummins #RedHat

    View Slide

  71. #RedHat
    @holly_cummins
    (See Troy tomorrow at GOTO !)

    View Slide

  72. #RedHat
    @holly_cummins
    (See Troy tomorrow at GOTO !)

    View Slide

  73. #RedHat
    @holly_cummins
    $11,448.30
    (See Troy tomorrow at GOTO !)

    View Slide

  74. #RedHat
    @holly_cummins
    “I did find a setting on your plan that
    set the max cacheable file size to
    15GB and it looks like your zipfile is
    18GB big”
    $11,448.30
    (See Troy tomorrow at GOTO !)

    View Slide

  75. #RedHat
    @holly_cummins

    View Slide

  76. #RedHat
    @holly_cummins
    maybe this is the chaos part

    View Slide

  77. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    releasing

    View Slide

  78. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    releasing

    View Slide

  79. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    microservices

    View Slide

  80. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    microservices

    View Slide

  81. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins

    View Slide

  82. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.

    View Slide

  83. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.
    we should modernise our COBOL
    application into microservices.

    View Slide

  84. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.
    we should modernise our COBOL
    application into microservices.
    … but our release board only meets twice a year.”

    View Slide

  85. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    modularity

    View Slide

  86. #RedHat
    @holly_cummins
    microservices are not the goal

    View Slide

  87. #RedHat
    @holly_cummins
    microservices are not the goal
    they are the means

    View Slide

  88. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    wishful mimicry

    View Slide

  89. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    cloud != microservices

    View Slide

  90. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    cloud native != microservices

    View Slide

  91. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “every time we touch one
    microservice, all the others break.”

    View Slide

  92. #RedHat
    @holly_cummins
    distributed monolith

    View Slide

  93. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking


    … or guaranteed function execution

    View Slide

  94. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking


    … or guaranteed function execution

    View Slide

  95. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking


    … or guaranteed function execution

    View Slide

  96. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking


    … or guaranteed function execution

    View Slide

  97. #RedHat
    @holly_cummins
    “each of our microservices has duplicated the
    same object model … with twenty classes and
    seventy fields”

    View Slide

  98. #RedHat
    @holly_cummins
    Microservice
    Domain

    View Slide

  99. #RedHat
    @holly_cummins
    Microservice
    Domain

    View Slide

  100. #RedHat
    @holly_cummins
    Microservice
    Domain
    (this is bad)

    View Slide

  101. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    distributed != decoupled

    View Slide

  102. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “ohhhh, we weren’t expecting your
    service to do that … “

    View Slide

  103. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “uh, what do you mean you
    corrected the typo in your json?”

    View Slide

  104. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    If you’re going to do microservices,
    you need to get good at automation.


    And testing.

    View Slide

  105. IBM Cloud © 2020 IBM Corporation

    View Slide

  106. IBM Cloud © 2020 IBM Corporation
    the test
    pyramid

    View Slide

  107. IBM Cloud © 2020 IBM Corporation
    the test
    pyramid end-to-end tests

    View Slide

  108. IBM Cloud © 2020 IBM Corporation
    the test
    pyramid end-to-end tests
    unit tests

    View Slide

  109. IBM Cloud © 2020 IBM Corporation
    the test
    pyramid end-to-end tests
    unit tests
    contract tests

    View Slide

  110. IBM Cloud © 2020 IBM Corporation
    the test
    pyramid
    (you can TDD
    at every level!)
    end-to-end tests
    unit tests
    contract tests

    View Slide

  111. #RedHat
    @holly_cummins
    How to test a fire alarm?

    View Slide

  112. IBM Cloud © 2020 IBM Corporation
    how not to test a fire alarm

    View Slide

  113. IBM Cloud © 2020 IBM Corporation
    how not to test a fire alarm

    View Slide

  114. #RedHat
    @holly_cummins

    View Slide

  115. #RedHat
    @holly_cummins
    unit testing a fire
    alarm

    View Slide

  116. #RedHat
    @holly_cummins
    uh … is that enough?

    View Slide

  117. #RedHat
    @holly_cummins
    contract testing a
    fire alarm

    View Slide

  118. #RedHat
    @holly_cummins
    contract testing a
    fire alarm

    View Slide

  119. @holly_cummins
    to the code!

    View Slide

  120. @holly_cummins #RedHat
    cold person knitter sheep farmer
    demo recap:


    View Slide

  121. @holly_cummins #RedHat
    cold person knitter sheep farmer
    “white sweater, please”
    demo recap:


    View Slide

  122. @holly_cummins #RedHat
    cold person knitter sheep farmer
    “white sweater, please”
    demo recap:


    View Slide

  123. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    demo recap:


    View Slide

  124. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    white


    wool
    demo recap:


    View Slide

  125. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    white


    wool
    demo recap:


    View Slide

  126. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    white


    wool
    demo recap:


    View Slide

  127. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    white


    wool
    black


    sweater
    demo recap:


    View Slide

  128. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    demo recap:


    View Slide

  129. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    demo recap:


    View Slide

  130. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    demo recap:


    View Slide

  131. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    pink


    sweater
    demo recap:


    View Slide

  132. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    pink


    sweater
    pink


    wool
    demo recap:


    View Slide

  133. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    pink


    sweater
    pink


    wool
    what?! pink sheep don’t exist
    demo recap:


    View Slide

  134. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    pink


    sweater
    pink


    wool
    should the knitter ask for white wool and bring in a dyeing service?
    what?! pink sheep don’t exist
    demo recap:


    View Slide

  135. @holly_cummins #RedHat
    cold person
    white


    sweater
    knitter sheep farmer
    “white sweater, please”
    “black sweater, please”
    “pink sweater, please”
    white


    wool
    black


    sweater
    black


    wool
    pink


    sweater
    pink


    wool
    should the knitter ask for white wool and bring in a dyeing service?
    should the farmer dye the wool?
    what?! pink sheep don’t exist
    demo recap:


    View Slide

  136. @holly_cummins #RedHat
    Photo of sheep at the Latitude Festival by Tim Parkinson, https://www.flickr.com/photos/timparkinson

    View Slide

  137. @holly_cummins #RedHat
    Photo of sheep at the Latitude Festival by Tim Parkinson, https://www.flickr.com/photos/timparkinson
    “pink sheep don’t exist”

    View Slide

  138. #RedHat
    @holly_cummins
    demo resources:


    https://github.com/holly-cummins/pact-quarkus-sweater-demo

    View Slide

  139. #RedHat
    @holly_cummins
    demo recap:


    • consumer-driven contract testing can help avoid disaster


    • pact is a mock for the consumer


    • pact is a functional test for the producer


    • shared json contracts aligns expectations across services

    View Slide

  140. #RedHat
    @holly_cummins
    so …


    don’t assume cloud is a solved problem


    don’t assume microservices will solve all your problems


    contract testing may solve some of your problems

    View Slide

  141. @holly_cummins
    slides

    View Slide