Uh, Does This Thing Actually Work? Modern Microservices Testing

by Holly Cummins

Slide 1

Slide 1 text

Holly Cummins Red Hat #JSpring uh, does this thing actually work? modern microservices testing

Slide 2

Slide 2 text

Disclaimer: I am not a tester. … but that’s ok. (Testing is every developer’s job.)

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

@holly_cummins #RedHat what does testing mean?

Slide 5

Slide 5 text

@holly_cummins #RedHat what does testing mean?

Slide 6

Slide 6 text

@holly_cummins #RedHat writing tests?

Slide 7

Slide 7 text

@holly_cummins #RedHat writing lots of tests?

Slide 8

Slide 8 text

@holly_cummins #RedHat 95% test coverage!

Slide 9

Slide 9 text

@holly_cummins #RedHat actually having assertions in tests

Slide 10

Slide 10 text

@holly_cummins #RedHat confidence the thing works

Slide 11

Slide 11 text

@holly_cummins #RedHat confidence to make changes

Slide 12

Slide 12 text

@holly_cummins #RedHat “we are moving to microservices so we can change things more easily.”

Slide 13

Slide 13 text

@holly_cummins #RedHat “If your microservice only does one thing, it can’t have an error. That would be an additional thing.” – Matt Stratton (not being serious!)

Slide 14

Slide 14 text

@holly_cummins #RedHat if you’re going to do microservices, you need to get good at automation. … and testing.

Slide 15

Slide 15 text

@holly_cummins #RedHat fallacies of microservices

Slide 16

Slide 16 text

@holly_cummins #RedHat fallacies of microservices “things we wish were true”

Slide 17

Slide 17 text

#Quarkus @holly_cummins “microservices makes testing easy! the thing I have to test is so small!”

Slide 18

Slide 18 text

#Quarkus @holly_cummins Netflix microservice architecture “microservices makes testing easy! the thing I have to test is so small!”

Slide 19

Slide 19 text

#Quarkus @holly_cummins Netflix microservice architecture “microservices makes testing easy! the thing I have to test is so small!” if you only test the points, and not the lines … does the thing work?

Slide 20

Slide 20 text

@holly_cummins #RedHat “my microservice is perfectly tested, even though the system doesn’t actually work.”

Slide 21

Slide 21 text

@holly_cummins #RedHat “we don’t need to test the lines… we’re decoupled!”

Slide 22

Slide 22 text

@holly_cummins #RedHat “we don’t need to test the lines… we’re decoupled!”

Slide 23

Slide 23 text

@holly_cummins #RedHat distributed != decoupled

Slide 24

Slide 24 text

@holly_cummins #RedHat “uh, what do you mean you corrected the typo in your json?”

Slide 25

Slide 25 text

@holly_cummins #RedHat “ohhhh, we weren’t expecting your service to do that … “

Slide 26

Slide 26 text

@holly_cummins #RedHat

Slide 27

Slide 27 text

@holly_cummins #RedHat “you’re putting a value in the wrong field when you call our API.

Slide 28

Slide 28 text

@holly_cummins #RedHat “you’re putting a value in the wrong field when you call our API. yes, I know there are 15 fields and the names aren’t particularly descriptive.”

Slide 29

Slide 29 text

@holly_cummins #RedHat how do you know if you’re decoupled?

Slide 30

Slide 30 text

can you deploy independently? v5.1.4 v5.1.4 v5.1.4 v5.1.4

Slide 31

Slide 31 text

can you deploy independently? v6.2.8 v65.7 v5.1.4 v1.0.5

Slide 32

Slide 32 text

@holly_cummins #RedHat TDD

Slide 33

Slide 33 text

@holly_cummins #RedHat TDD: improves quality

Slide 34

Slide 34 text

@holly_cummins #RedHat TDD: an excellent design tool

Slide 35

Slide 35 text

@holly_cummins #RedHat TDD: an excellent design tool (if testing is hard, that’s a hint the design is wrong)

Slide 36

Slide 36 text

@holly_cummins #RedHat

Slide 37

Slide 37 text

@holly_cummins #RedHat the real reason to do TDD:

Slide 38

Slide 38 text

@holly_cummins #RedHat the real reason to do TDD:

Slide 39

Slide 39 text

@holly_cummins #RedHat the real reason to do TDD: gamified testing

Slide 40

Slide 40 text

@holly_cummins #RedHat the test pyramid

Slide 41

Slide 41 text

@holly_cummins #RedHat the test pyramid (you can TDD at every level)

Slide 42

Slide 42 text

@holly_cummins #RedHat the test pyramid

Slide 43

Slide 43 text

@holly_cummins #RedHat the test pyramid end-to-end tests

Slide 44

Slide 44 text

@holly_cummins #RedHat the test pyramid end-to-end tests integration tests

Slide 45

Slide 45 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests

Slide 46

Slide 46 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort

Slide 47

Slide 47 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort high realism

Slide 48

Slide 48 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort high realism high value

Slide 49

Slide 49 text

@holly_cummins #RedHat all models are wrong some are useful

Slide 50

Slide 50 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism

Slide 51

Slide 51 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server

Slide 52

Slide 52 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints

Slide 53

Slide 53 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints

Slide 54

Slide 54 text

@holly_cummins #RedHat unit tests low effort

Slide 55

Slide 55 text

@holly_cummins #RedHat “why have I spent half a day writing mocks just so I can write a unit test?” the limitations of unit tests

Slide 56

Slide 56 text

@holly_cummins #RedHat the limitations of unit tests “why do my colleagues tell me my code is broken, even though my unit tests all pass?”

Slide 57

Slide 57 text

@holly_cummins #RedHat their code our code why unit tests aren’t enough

Slide 58

Slide 58 text

@holly_cummins #RedHat their code our code why mocks aren’t enough

Slide 59

Slide 59 text

@holly_cummins #RedHat their code our code why stubs aren’t enough

Slide 60

Slide 60 text

@holly_cummins #RedHat our code our mock why mocks aren’t enough

Slide 61

Slide 61 text

@holly_cummins #RedHat our code our mock tests ✔ why mocks aren’t enough

Slide 62

Slide 62 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ why mocks aren’t enough

Slide 63

Slide 63 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ reality ✘ why mocks aren’t enough

Slide 64

Slide 64 text

@holly_cummins #RedHat what needs integration tests? our service

Slide 65

Slide 65 text

@holly_cummins #RedHat what needs integration tests? our service all code that serialises or deserialises data

Slide 66

Slide 66 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers all code that serialises or deserialises data

Slide 67

Slide 67 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls all code that serialises or deserialises data

Slide 68

Slide 68 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls queues ETL all code that serialises or deserialises data

Slide 69

Slide 69 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 70

Slide 70 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls external applications colleague’s services other services filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 71

Slide 71 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls external applications colleague’s services filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 72

Slide 72 text

Slide 73

Slide 73 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 74

Slide 74 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 75

Slide 75 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 76

Slide 76 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism

Slide 77

Slide 77 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism

Slide 78

Slide 78 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism in staging

Slide 79

Slide 79 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism in staging ?

Slide 80

Slide 80 text

@holly_cummins #RedHat “why do I keep getting failures in production even though everything looked fine in staging?” Photo by Eggs&Beer, Flickr

Slide 81

Slide 81 text

@holly_cummins #RedHat staging staging ✔ why staging is not enough our service other services other services other services other services

Slide 82

Slide 82 text

@holly_cummins #RedHat staging staging ✔ production ✘ why staging is not enough our service other services other services other services other services our service production other services other services other services other services

Slide 83

Slide 83 text

@holly_cummins #RedHat “test in prod or live a lie” - Charity Majors

Slide 84

Slide 84 text

@holly_cummins #RedHat testing in production, done well: observability ability to deploy quickly (& safely)

Slide 85

Slide 85 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism

Slide 86

Slide 86 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism testing in production

Slide 87

Slide 87 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism static tests testing in production

Slide 88

Slide 88 text

@holly_cummins #RedHat confidence effort

Slide 89

Slide 89 text

@holly_cummins #RedHat confidence effort quality

Slide 90

Slide 90 text

@holly_cummins #RedHat confidence effort quality expense

Slide 91

Slide 91 text

@holly_cummins #RedHat confidence effort quality expense cheap and cheerful

Slide 92

Slide 92 text

@holly_cummins #RedHat confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 93

Slide 93 text

@holly_cummins #RedHat winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 94

Slide 94 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 95

Slide 95 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 96

Slide 96 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests

Slide 97

Slide 97 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests

Slide 98

Slide 98 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests

Slide 99

Slide 99 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests static tests

Slide 100

Slide 100 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests

Slide 101

Slide 101 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 102

Slide 102 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 103

Slide 103 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 104

Slide 104 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 105

Slide 105 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests

Slide 106

Slide 106 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests constantly question the value of your tests

Slide 107

Slide 107 text

@holly_cummins #RedHat integration tests

Slide 108

Slide 108 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests

Slide 109

Slide 109 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests

Slide 110

Slide 110 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal

Slide 111

Slide 111 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal

Slide 112

Slide 112 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal integration tests

Slide 113

Slide 113 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal integration tests integration tests

Slide 114

Slide 114 text

Slide 115

Slide 115 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal integration tests contract tests integration tests integration tests

Slide 116

Slide 116 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal contract tests sandbox tests integration tests integration tests

Slide 117

Slide 117 text

Slide 118

Slide 118 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications

Slide 119

Slide 119 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications mock all the things

Slide 120

Slide 120 text

@holly_cummins #RedHat “why have I spent half a day writing mocks just so I can write a cleanly-scoped integration test?”

Slide 121

Slide 121 text

@holly_cummins #RedHat approaches to integration tests our service rest api db calls external applications filesystem config, etc queues ETL colleague’s services colleague’s services colleague’s services external applications

Slide 122

Slide 122 text

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Slide 125

Slide 125 text

@holly_cummins #RedHat

Slide 126

Slide 126 text

@holly_cummins #RedHat “I can’t bring up my dev stack

Slide 127

Slide 127 text

@holly_cummins #RedHat “I can’t bring up my dev stack … on my brand new Apple laptop with a M1 chip and 64GB of RAM.” - fintech developer

Slide 128

Slide 128 text

@holly_cummins #RedHat “I can’t bring up my dev stack … on my brand new Apple laptop with a M1 chip and 64GB of RAM.” digression: if this is you • maybe testing is only part of your problem • … and Quarkus could help :) - fintech developer

Slide 129

Slide 129 text

@holly_cummins #RedHat “I have to do all this shell scripting and manual process and orchestration before I can write a test or do development.”

Slide 130

Slide 130 text

@holly_cummins #RedHat testing is a useful design tool

Slide 131

Slide 131 text

@holly_cummins #RedHat testing is a useful design tool (if testing your micro service is hard, that’s a hint the domain boundary is wrong)

Slide 132

Slide 132 text

Slide 133

Slide 133 text

@holly_cummins #RedHat “I miss developing on my actual laptop.”

Slide 134

Slide 134 text

Slide 135

Slide 135 text

Slide 136

Slide 136 text

Slide 137

Slide 137 text

@holly_cummins #RedHat pitfalls: • all services need to be configured to forward headers • relies on rest; will not play well with kafka queues or other protocols • can be tricky to get to work

Slide 138

Slide 138 text

Slide 139

Slide 139 text

@holly_cummins #RedHat pitfalls: • it all works great – until a test changes some state, eg • writing to a db • adding a user to a security provider • getting IP whitelists when running locally is tricky

Slide 140

Slide 140 text

@holly_cummins #RedHat

Slide 141

Slide 141 text

@holly_cummins #RedHat “could I be … a bit less realistic? and a bit more offline?”

Slide 142

Slide 142 text

@holly_cummins #RedHat networks: just say no

Slide 143

Slide 143 text

Slide 144

Slide 144 text

Slide 145

Slide 145 text

Slide 146

Slide 146 text

Slide 147

Slide 147 text

to the code! @holly_cummins #RedHat

Slide 148

Slide 148 text

@holly_cummins #RedHat

Slide 149

Slide 149 text

@holly_cummins #RedHat drawback:

Slide 150

Slide 150 text

@holly_cummins #RedHat drawback: • the service still uses CPU

Slide 151

Slide 151 text

@holly_cummins #RedHat drawback: • the service still uses CPU • only works for well-known libraries

Slide 152

Slide 152 text

@holly_cummins #RedHat bonus: •programmatic control of dependencies •seamless with some frameworks (esp. Quarkus)

Slide 153

Slide 153 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications

Slide 154

Slide 154 text

Slide 155

Slide 155 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ why mocks aren’t enough

Slide 156

Slide 156 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ reality ✘ why mocks aren’t enough

Slide 157

Slide 157 text

@holly_cummins #RedHat our code their code why mocks aren’t enough

Slide 158

Slide 158 text

@holly_cummins #RedHat our code their code contract test why mocks aren’t enough

Slide 159

Slide 159 text

@holly_cummins #RedHat our code their code contract test mock why mocks aren’t enough

Slide 160

Slide 160 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 161

Slide 161 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✔ their tests ✔ reality ✔ why mocks aren’t enough

Slide 162

Slide 162 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 163

Slide 163 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✔ their tests ✘ reality ✘ why mocks aren’t enough

Slide 164

Slide 164 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 165

Slide 165 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✘ their tests ✔ reality ✘ why mocks aren’t enough

Slide 166

Slide 166 text

@holly_cummins #RedHat

Slide 167

Slide 167 text

@holly_cummins #RedHat contract-first is a great way to TDD

Slide 168

Slide 168 text

@holly_cummins #RedHat contract-first is a great way to TDD (and a great design tool)

Slide 169

Slide 169 text

to the code! @holly_cummins #RedHat

Slide 170

Slide 170 text

#RedHat @holly_cummins demo recap: https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample

Slide 171

Slide 171 text

#RedHat @holly_cummins demo recap: https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample

Slide 172

Slide 172 text

#RedHat @holly_cummins demo recap: • consumer-driven contract testing can save your bacon • pact is a mock for the consumer • pact is a functional test for the producer • shared json contracts aligns expectations across services

Slide 173

Slide 173 text

@holly_cummins #RedHat pitfalls of contract testing: • we have to talk to the other team • our consumers can break our CI?! • but I’ve already written mocks and functional tests

Slide 174

Slide 174 text

@holly_cummins #RedHat unexpected benefits of contract testing: preserving backwards compatibility YAGNI YAWIBNCSDI*

Slide 175

Slide 175 text

@holly_cummins #RedHat unexpected benefits of contract testing: preserving backwards compatibility YAGNI YAWIBNCSDI* * (you already wrote it but now can safely delete it)

Slide 176

Slide 176 text

@holly_cummins #RedHat contract testing options: • pact.io • Microcks • spring contract • OpenAPI schema validation + mock generation • prism • schemathesis

Slide 177

Slide 177 text

@holly_cummins #RedHat ok but what should I do?

Slide 178

Slide 178 text

@holly_cummins #RedHat confidence effort integration testing landscape mock all the services testcontainers remote sandbox remocal sandbox contract tests local orchestration

Slide 179

Slide 179 text

@holly_cummins #RedHat confidence effort integration testing landscape mock all the services testcontainers remote sandbox remocal sandbox contract tests local orchestration if you don’t have a big platform team

Slide 180

Slide 180 text

Slide 181

Slide 181 text

@holly_cummins #RedHat so, does the thing work? test early, test often testcontainers rocks contract testing can prevent pain think both locally and system-level

Slide 182

Slide 182 text

® thank you @holly_cummins slides + code