Cloud Chaos and Microservices Mayhem

The way we use Java has changed. Applets - or any browser-side Java - are a long, long, time ago. Java moved to the back end, and stayed there, but the back-end has gotten bigger, more distributed, and more complicated. Some things that were a good idea ten years ago turn out to be a terrible idea in the cloud; and what used to be ‘good enough’ for testing really isn’t anymore. Managing microservices architecture demands a lot of us, to ensure observability, operational resiliency, and organisational agility. This talk will introduce some of the new tools, patterns, and best practices for Java development in 2022. It also gives a tour of some of the most painful anti-patterns Holly has seen as a former consultant with the IBM Garage.

Transcript

1. cloud chaos and
   microservices mayhem
   
   
   
   
   Holly Cummins
   
   
   Red Hat
   
   
   @holly_cummins
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. #RedHat
   @holly_cummins
   things you
   need to do
   well in 2022:
   
   
   

   View Slide

6. #RedHat
   @holly_cummins
   handwashing
   things you
   need to do
   well in 2022:
   
   
   

   View Slide

7. #RedHat
   @holly_cummins
   handwashing
   apps
   things you
   need to do
   well in 2022:
   
   
   

   View Slide

8. #RedHat
   @holly_cummins
   handwashing
   apps
   ops
   things you
   need to do
   well in 2022:
   
   
   

   View Slide

9. #RedHat
   @holly_cummins
   handwashing
   apps
   ops
   devops
   things you
   need to do
   well in 2022:
   
   
   

   View Slide

10. #RedHat
    @holly_cummins
    handwashing
    apps
    ops
    devops
    devsecops
    things you
    need to do
    well in 2022:
    
    
    

    View Slide

11. #RedHat
    @holly_cummins
    handwashing
    apps
    ops
    devops
    devsecops
    finops
    things you
    need to do
    well in 2022:
    
    
    

    View Slide

12. View Slide

13. View Slide

14. #RedHat
    @holly_cummins
    a few
    

    View Slide

15. #RedHat
    @holly_cummins
    

    View Slide

16. #RedHat
    @holly_cummins
    technology changes fast (duh)
    

    View Slide

17. #RedHat
    @holly_cummins
    technology changes fast (duh)
    … but it’s taking us a while to catch up to cloud
    

    View Slide

18. #RedHat
    @holly_cummins
    technology changes fast (duh)
    … but it’s taking us a while to catch up to cloud
    

    View Slide

19. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    tracing
    

    View Slide

20. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    observability
    

    View Slide

21. #RedHat
    @holly_cummins
    hardware
    old way
    logs
    

    View Slide

22. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs
    

    View Slide

23. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    

    View Slide

24. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    oops
    

    View Slide

25. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs
    

    View Slide

26. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    container
    logs
    

    View Slide

27. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    

    View Slide

28. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    

    View Slide

29. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    

    View Slide

30. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    aggregation + correlation
    

    View Slide

31. #RedHat
    @holly_cummins
    hardware
    old way cloud way
    logs
    logs
    container
    logs
    container
    logs
    container
    logs
    container
    logs
    aggregation + correlation
    open telemetry
    open tracing
    

    View Slide

32. #RedHat
    @holly_cummins
    

    View Slide

33. #RedHat
    @holly_cummins
    good news: observability is ‘fixed’
    

    View Slide

34. #RedHat
    @holly_cummins
    good news: observability is ‘fixed’
    bad news: not all of us are using it
    

    View Slide

35. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    performance requirements
    

    View Slide

36. #RedHat
    @holly_cummins
    memory footprint is money
    
    
    startup time is money
    

    View Slide

37. #RedHat
    @holly_cummins
    ahead of time compilation is
    (now) a good idea
    

    View Slide

38. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    performance optimisation
    

    View Slide

39. #RedHat
    @holly_cummins
    

    View Slide

40. #RedHat
    @holly_cummins
    

    View Slide

41. #RedHat
    @holly_cummins
    the JVM tunes itself differently on servers
    

    View Slide

42. #RedHat
    @holly_cummins
    the JVM tunes itself differently on servers
    … but what’s a server?
    

    View Slide

43. #RedHat
    @holly_cummins
    answer(ish):
    

    View Slide

44. #RedHat
    @holly_cummins
    // This is the working definition of a server class machine:
    // >= 2 physical CPU's and >=2GB of memory
    answer(ish):
    actual hotspot source code
    

    View Slide

45. #RedHat
    @holly_cummins
    

    View Slide

46. #RedHat
    @holly_cummins
    takeaway: don’t shrink your containers too
    much or your JVM will make bad decisions
    advice accurate as of June 2022. all performance advice should be independently verified. do not try this at home without testing first. your mileage may vary.
    

    View Slide

47. #RedHat
    @holly_cummins
    wait, isn’t that the
    opposite of the advice in the
    previous section? should my
    containers be big or
    small?
    

    View Slide

48. #RedHat
    @holly_cummins
    this is the chaos part :)
    wait, isn’t that the
    opposite of the advice in the
    previous section? should my
    containers be big or
    small?
    

    View Slide

49. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    management
    

    View Slide

50. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    keeping track of hardware
    

    View Slide

51. #RedHat
    @holly_cummins
    The cloud makes it so easy to
    provision hardware.
    

    View Slide

52. #RedHat
    @holly_cummins
    That doesn’t mean
    the hardware is free.
    

    View Slide

53. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.
    

    View Slide

54. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.
    Or useful.
    

    View Slide

55. #RedHat
    @holly_cummins
    That doesn’t mean the hardware is
    free.
    Or useful.
    Or shuts itself off.
    

    View Slide

56. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.
    

    View Slide

57. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.
    I forgot it for 2 months.
    

    View Slide

58. #RedHat
    @holly_cummins
    Hey boss, I created a
    Kubernetes cluster.
    I forgot it for 2 months.
    … and it’s €1000 a month.
    

    View Slide

59. Photo by daveynin, flickr
    

    View Slide

60. #RedHat
    @holly_cummins
    2017 survey
    
    
    25%
    
    
    of 16,000 servers
    doing no useful work
    

    View Slide

61. #RedHat
    @holly_cummins
    2017 survey
    
    
    25%
    
    
    of 16,000 servers
    doing no useful work
    “perhaps someone
    forgot to turn them off”
    

    View Slide

62. #RedHat
    @holly_cummins
    finops
    
    
    figuring out who in
    your company forgot
    to turn off their cloud
    

    View Slide

63. #RedHat
    @holly_cummins
    companies that went out of
    business for cloud cost accidents
    

    View Slide

64. @holly_cummins #RedHat
    

    View Slide

65. #RedHat
    @holly_cummins
    

    View Slide

66. #RedHat
    @holly_cummins
    

    View Slide

67. #RedHat
    @holly_cummins
    $11,448.30
    

    View Slide

68. #RedHat
    @holly_cummins
    “I did find a setting on your plan that
    set the max cacheable file size to
    15GB and it looks like your zipfile is
    18GB big”
    $11,448.30
    

    View Slide

69. #RedHat
    @holly_cummins
    

    View Slide

70. #RedHat
    @holly_cummins
    maybe this is the chaos part
    

    View Slide

71. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    releasing
    

    View Slide

72. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    releasing
    

    View Slide

73. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    microservices
    

    View Slide

74. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    microservices
    

    View Slide

75. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    

    View Slide

76. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.
    

    View Slide

77. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.
    we should modernise our COBOL
    application into microservices.
    

    View Slide

78. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “we’re moving too slowly.
    we should modernise our COBOL
    application into microservices.
    … but our release board only meets twice a year.”
    

    View Slide

79. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    modularity
    

    View Slide

80. #RedHat
    @holly_cummins
    microservices are not the goal
    

    View Slide

81. #RedHat
    @holly_cummins
    microservices are not the goal
    they are the means
    

    View Slide

82. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    wishful mimicry
    

    View Slide

83. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    cloud != microservices
    

    View Slide

84. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    cloud native != microservices
    

    View Slide

85. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “every time we touch one
    microservice, all the others break.”
    

    View Slide

86. #RedHat
    @holly_cummins
    distributed monolith
    

    View Slide

87. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking
    
    
    … or guaranteed function execution
    

    View Slide

88. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking
    
    
    … or guaranteed function execution
    

    View Slide

89. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking
    
    
    … or guaranteed function execution
    

    View Slide

90. #RedHat
    @holly_cummins
    distributed monolith
    but without compile-time checking
    
    
    … or guaranteed function execution
    

    View Slide

91. #RedHat
    @holly_cummins
    “each of our microservices has duplicated the
    same object model … with twenty classes and
    seventy fields”
    

    View Slide

92. #RedHat
    @holly_cummins
    Microservice
    Domain
    

    View Slide

93. #RedHat
    @holly_cummins
    Microservice
    Domain
    

    View Slide

94. #RedHat
    @holly_cummins
    Microservice
    Domain
    (this is bad)
    

    View Slide

95. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    distributed != decoupled
    

    View Slide

96. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “ohhhh, we weren’t expecting your
    service to do that … “
    

    View Slide

97. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    “uh, what do you mean you
    corrected the typo in your json?”
    

    View Slide

98. #IBMGarage + IBM Cloud © 2020 IBM Corporation
    @holly_cummins
    If you’re going to do microservices,
    you need to get good at automation.
    
    
    And testing.
    

    View Slide

99. IBM Cloud © 2020 IBM Corporation
    

    View Slide

100. IBM Cloud © 2020 IBM Corporation
     the test
     pyramid
     

     View Slide

101. IBM Cloud © 2020 IBM Corporation
     the test
     pyramid end-to-end tests
     

     View Slide

102. IBM Cloud © 2020 IBM Corporation
     the test
     pyramid end-to-end tests
     unit tests
     

     View Slide

103. IBM Cloud © 2020 IBM Corporation
     the test
     pyramid end-to-end tests
     unit tests
     contract tests
     

     View Slide

104. IBM Cloud © 2020 IBM Corporation
     the test
     pyramid
     (you can TDD
     at every level!)
     end-to-end tests
     unit tests
     contract tests
     

     View Slide

105. #RedHat
     @holly_cummins
     How to test a fire alarm?
     

     View Slide

106. IBM Cloud © 2020 IBM Corporation
     how not to test a fire alarm
     

     View Slide

107. IBM Cloud © 2020 IBM Corporation
     how not to test a fire alarm
     

     View Slide

108. #RedHat
     @holly_cummins
     

     View Slide

109. #RedHat
     @holly_cummins
     unit testing a fire
     alarm
     

     View Slide

110. #RedHat
     @holly_cummins
     uh … is that enough?
     

     View Slide

111. #RedHat
     @holly_cummins
     contract testing a
     fire alarm
     

     View Slide

112. #RedHat
     @holly_cummins
     contract testing a
     fire alarm
     

     View Slide

113. #IBMGarage + IBM Cloud © 2020 IBM Corporation
     @holly_cummins
     to the code!
     

     View Slide

114. #RedHat
     @holly_cummins
     demo recap:
     
     
     https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample
     

     View Slide

115. #RedHat
     @holly_cummins
     demo recap:
     
     
     https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample
     

     View Slide

116. #RedHat
     @holly_cummins
     demo recap:
     
     
     • consumer-driven contract testing can save your bacon
     
     
     • pact is a mock for the consumer
     
     
     • pact is a functional test for the producer
     
     
     • shared json contracts aligns expectations across services
     

     View Slide

117. #RedHat
     @holly_cummins
     slides
     

     View Slide