Uh, Does This Thing Actually Work? Modern Microservices Testing

by Holly Cummins

Slide 1

Slide 1 text

Holly Cummins Red Hat #SkillsMatter uh, does this thing actually work? modern microservices testing

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

disclaimer: I am not a tester

Slide 4

Slide 4 text

disclaimer: I am not a tester … or even a test specialist.

Slide 5

Slide 5 text

disclaimer: I am not a tester … or even a test specialist. but that’s ok! testing is every developer’s job.

Slide 6

Slide 6 text

@holly_cummins #RedHat fallacies of microservices

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

#Quarkus @holly_cummins “microservices makes testing easy! the thing I have to test is so small!”

Slide 15

Slide 15 text

#Quarkus @holly_cummins Netflix microservice architecture “microservices makes testing easy! the thing I have to test is so small!”

Slide 16

Slide 16 text

#Quarkus @holly_cummins Netflix microservice architecture “microservices makes testing easy! the thing I have to test is so small!” if you only test the points, and not the lines … does the thing work?

Slide 17

Slide 17 text

#Quarkus @holly_cummins “That’s no moon. It’s our architecture.” Netflix microservice architecture

Slide 18

Slide 18 text

@holly_cummins #RedHat “microservices makes testing easy. we’re decoupled!”

Slide 19

Slide 19 text

@holly_cummins #RedHat “microservices makes testing easy. we’re decoupled!”

Slide 20

Slide 20 text

@holly_cummins #RedHat distributed != decoupled

Slide 21

Slide 21 text

@holly_cummins #RedHat “ohhhh, we weren’t expecting your service to do that … “

Slide 22

Slide 22 text

@holly_cummins #RedHat “uh, what do you mean you corrected the typo in your json?”

Slide 23

Slide 23 text

@holly_cummins #RedHat how do you know if you’re decoupled?

Slide 24

Slide 24 text

can you deploy independently? v5.1.4 v5.1.4 v5.1.4 v5.1.4

Slide 25

Slide 25 text

can you deploy independently? v6.2.8 v65.7 v5.1.4 v1.0.5

Slide 26

Slide 26 text

@holly_cummins #RedHat if you’re going to do microservices, you need to get good at automation. … and testing.

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

@holly_cummins #RedHat TDD

Slide 29

Slide 29 text

@holly_cummins #RedHat TDD: improves quality

Slide 30

Slide 30 text

@holly_cummins #RedHat TDD: an excellent design tool

Slide 31

Slide 31 text

@holly_cummins #RedHat

Slide 32

Slide 32 text

@holly_cummins #RedHat the real reason to do TDD:

Slide 33

Slide 33 text

@holly_cummins #RedHat the real reason to do TDD:

Slide 34

Slide 34 text

@holly_cummins #RedHat the real reason to do TDD: gamified testing

Slide 35

Slide 35 text

@holly_cummins #RedHat the test pyramid

Slide 36

Slide 36 text

@holly_cummins #RedHat the test pyramid (you can TDD at every level)

Slide 37

Slide 37 text

@holly_cummins #RedHat the test pyramid

Slide 38

Slide 38 text

@holly_cummins #RedHat the test pyramid end-to-end tests

Slide 39

Slide 39 text

@holly_cummins #RedHat the test pyramid end-to-end tests integration tests

Slide 40

Slide 40 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests

Slide 41

Slide 41 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort

Slide 42

Slide 42 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort high realism

Slide 43

Slide 43 text

@holly_cummins #RedHat the test pyramid end-to-end tests unit tests integration tests low effort high realism high value

Slide 44

Slide 44 text

@holly_cummins #RedHat all models are wrong some are useful

Slide 45

Slide 45 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism

Slide 46

Slide 46 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server

Slide 47

Slide 47 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints

Slide 48

Slide 48 text

@holly_cummins #RedHat tools evolve end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints

Slide 49

Slide 49 text

@holly_cummins #RedHat unit tests low effort

Slide 50

Slide 50 text

@holly_cummins #RedHat “why have I spent half a day writing mocks just so I can write a unit test?” the limitations of unit tests

Slide 51

Slide 51 text

@holly_cummins #RedHat the limitations of unit tests “why do my colleagues tell me my code is broken, even though my unit tests all pass?”

Slide 52

Slide 52 text

@holly_cummins #RedHat their code our code why unit tests aren’t enough

Slide 53

Slide 53 text

@holly_cummins #RedHat their code our code why mocks aren’t enough

Slide 54

Slide 54 text

@holly_cummins #RedHat their code our code why stubs aren’t enough

Slide 55

Slide 55 text

@holly_cummins #RedHat our code our mock why mocks aren’t enough

Slide 56

Slide 56 text

@holly_cummins #RedHat our code our mock tests ✔ why mocks aren’t enough

Slide 57

Slide 57 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ why mocks aren’t enough

Slide 58

Slide 58 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ reality ✘ why mocks aren’t enough

Slide 59

Slide 59 text

@holly_cummins #RedHat what needs integration tests? our service

Slide 60

Slide 60 text

@holly_cummins #RedHat what needs integration tests? our service all code that serialises or deserialises data

Slide 61

Slide 61 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers all code that serialises or deserialises data

Slide 62

Slide 62 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls all code that serialises or deserialises data

Slide 63

Slide 63 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls queues ETL all code that serialises or deserialises data

Slide 64

Slide 64 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 65

Slide 65 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls external applications colleague’s services other services filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 66

Slide 66 text

@holly_cummins #RedHat what needs integration tests? our service rest api usage by consumers db calls external applications colleague’s services filesystem config, etc queues ETL all code that serialises or deserialises data

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Slide 75

Slide 75 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 76

Slide 76 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 77

Slide 77 text

@holly_cummins #RedHat end-to-end tests unit tests integration tests low effort high realism

Slide 78

Slide 78 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism

Slide 79

Slide 79 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism

Slide 80

Slide 80 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism in staging

Slide 81

Slide 81 text

@holly_cummins #RedHat the test trophy end-to-end tests high realism in staging ?

Slide 82

Slide 82 text

@holly_cummins #RedHat “why do I keep getting failures in production even though everything looked fine in staging?” Photo by Eggs&Beer, Flickr

Slide 83

Slide 83 text

@holly_cummins #RedHat staging staging ✔ why staging is not enough our service other services other services other services other services

Slide 84

Slide 84 text

@holly_cummins #RedHat staging staging ✔ production ✘ why staging is not enough our service other services other services other services other services our service production other services other services other services other services

Slide 85

Slide 85 text

@holly_cummins #RedHat “test in prod or live a lie” - Charity Majors

Slide 86

Slide 86 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism

Slide 87

Slide 87 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism testing in production

Slide 88

Slide 88 text

@holly_cummins #RedHat the test trophy end-to-end tests unit tests integration tests low effort high realism static tests testing in production

Slide 89

Slide 89 text

@holly_cummins #RedHat confidence effort

Slide 90

Slide 90 text

@holly_cummins #RedHat confidence effort quality

Slide 91

Slide 91 text

@holly_cummins #RedHat confidence effort quality expense

Slide 92

Slide 92 text

@holly_cummins #RedHat confidence effort quality expense cheap and cheerful

Slide 93

Slide 93 text

@holly_cummins #RedHat confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 94

Slide 94 text

@holly_cummins #RedHat winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 95

Slide 95 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 96

Slide 96 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful

Slide 97

Slide 97 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests

Slide 98

Slide 98 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests

Slide 99

Slide 99 text

@holly_cummins #RedHat a waste winning confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests

Slide 100

Slide 100 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests static tests

Slide 101

Slide 101 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests

Slide 102

Slide 102 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 103

Slide 103 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 104

Slide 104 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 105

Slide 105 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests YMMV

Slide 106

Slide 106 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests

Slide 107

Slide 107 text

@holly_cummins #RedHat a waste confidence effort quality expense hard work, but valuable cheap and cheerful unit tests integration tests e2e tests testing in production static tests constantly question the value of your tests

Slide 108

Slide 108 text

@holly_cummins #RedHat integration tests

Slide 109

Slide 109 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests

Slide 110

Slide 110 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests

Slide 111

Slide 111 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal

Slide 112

Slide 112 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal

Slide 113

Slide 113 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal integration tests

Slide 114

Slide 114 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests integration tests all integration tests are not created equal integration tests integration tests

Slide 115

Slide 115 text

Slide 116

Slide 116 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal integration tests contract tests integration tests integration tests

Slide 117

Slide 117 text

@holly_cummins #RedHat confidence effort unit tests integration tests e2e tests testing in production static tests all integration tests are not created equal contract tests sandbox tests integration tests integration tests

Slide 118

Slide 118 text

Slide 119

Slide 119 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications

Slide 120

Slide 120 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications mock all the things

Slide 121

Slide 121 text

@holly_cummins #RedHat “why have I spent half a day writing mocks just so I can write a cleanly-scoped integration test?”

Slide 122

Slide 122 text

@holly_cummins #RedHat approaches to integration tests our service rest api db calls external applications filesystem config, etc queues ETL colleague’s services colleague’s services colleague’s services external applications

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Slide 125

Slide 125 text

Slide 126

Slide 126 text

@holly_cummins #RedHat

Slide 127

Slide 127 text

@holly_cummins #RedHat “I can’t bring up my dev stack

Slide 128

Slide 128 text

@holly_cummins #RedHat “I can’t bring up my dev stack … on my brand new Apple laptop with a M1 chip and 64GB of RAM.” - fintech developer

Slide 129

Slide 129 text

@holly_cummins #RedHat “I can’t bring up my dev stack … on my brand new Apple laptop with a M1 chip and 64GB of RAM.” digression: if this is you • maybe testing is only part of your problem • … and Quarkus could help :) - fintech developer

Slide 130

Slide 130 text

@holly_cummins #RedHat “I have to do all this shell scripting and manual process and orchestration before I can write a test or do development.”

Slide 131

Slide 131 text

Slide 132

Slide 132 text

@holly_cummins #RedHat “I miss developing on my actual laptop.”

Slide 133

Slide 133 text

Slide 134

Slide 134 text

Slide 135

Slide 135 text

Slide 136

Slide 136 text

Slide 137

Slide 137 text

@holly_cummins #RedHat pitfalls: • it all works great – until a test changes some state, eg • writing to a db • adding a user to a security provider • getting IP whitelists when running locally is tricky

Slide 138

Slide 138 text

@holly_cummins #RedHat

Slide 139

Slide 139 text

@holly_cummins #RedHat “could I be … a bit less realistic? and a bit more offline?”

Slide 140

Slide 140 text

@holly_cummins #RedHat networks: just say no

Slide 141

Slide 141 text

Slide 142

Slide 142 text

Slide 143

Slide 143 text

Slide 144

Slide 144 text

Slide 145

Slide 145 text

@holly_cummins #RedHat

Slide 146

Slide 146 text

@holly_cummins #RedHat drawback:

Slide 147

Slide 147 text

@holly_cummins #RedHat drawback: • the service still uses CPU

Slide 148

Slide 148 text

@holly_cummins #RedHat drawback: • the service still uses CPU • only works for well-known libraries

Slide 149

Slide 149 text

@holly_cummins #RedHat approaches to integration tests our service rest api external applications filesystem colleague’s services colleague’s services colleague’s services external applications

Slide 150

Slide 150 text

Slide 151

Slide 151 text

@holly_cummins #RedHat their code our code why mocks aren’t enough

Slide 152

Slide 152 text

@holly_cummins #RedHat our code our mock why mocks aren’t enough

Slide 153

Slide 153 text

@holly_cummins #RedHat our code our mock tests ✔ why mocks aren’t enough

Slide 154

Slide 154 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ why mocks aren’t enough

Slide 155

Slide 155 text

@holly_cummins #RedHat our code our mock our code their actual code tests ✔ reality ✘ why mocks aren’t enough

Slide 156

Slide 156 text

@holly_cummins #RedHat our code their code why mocks aren’t enough

Slide 157

Slide 157 text

@holly_cummins #RedHat our code their code contract test why mocks aren’t enough

Slide 158

Slide 158 text

@holly_cummins #RedHat our code their code contract test mock why mocks aren’t enough

Slide 159

Slide 159 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 160

Slide 160 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✔ their tests ✔ reality ✔ why mocks aren’t enough

Slide 161

Slide 161 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 162

Slide 162 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✔ their tests ✘ reality ✘ why mocks aren’t enough

Slide 163

Slide 163 text

@holly_cummins #RedHat our code their code contract test mock functional test why mocks aren’t enough

Slide 164

Slide 164 text

@holly_cummins #RedHat our code their code contract test mock functional test our tests ✘ their tests ✔ reality ✘ why mocks aren’t enough

Slide 165

Slide 165 text

@holly_cummins #RedHat

Slide 166

Slide 166 text

@holly_cummins #RedHat contract-first is a great way to TDD

Slide 167

Slide 167 text

@holly_cummins #RedHat contract-first is a great way to TDD (and a great design tool)

Slide 168

Slide 168 text

to the code! @holly_cummins #RedHat

Slide 169

Slide 169 text

#RedHat @holly_cummins demo recap: https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample

Slide 170

Slide 170 text

#RedHat @holly_cummins demo recap: https://github.com/holly-cummins/house-of-microservices-quarkus-contract-testing-sample

Slide 171

Slide 171 text

#RedHat @holly_cummins demo recap: • consumer-driven contract testing can save your bacon • pact is a mock for the consumer • pact is a functional test for the producer • shared json contracts aligns expectations across services

Slide 172

Slide 172 text

@holly_cummins #RedHat pitfalls of contract testing: • we have to talk to the other team • our consumers can break our CI?! • versions • pact is hard to grok • testing the mock • but I’ve invested in wiremock • duplicated tests is overhead

Slide 173

Slide 173 text

@holly_cummins #RedHat contract testing options: • pact.io • spring contract • OpenAPI schema validation + mock generation • prism • schemathesis

Slide 174

Slide 174 text

@holly_cummins #RedHat confidence effort integration testing landscape mock all the services testcontainers remote sandbox remocal sandbox contract tests local orchestration

Slide 175

Slide 175 text

® thank you @holly_cummins https://github.com/holly-cummins/house-of- microservices-quarkus-contract-testing-sample