Types, tests, and why flat-earthers are bad at QA (HolyJS)

Slide 1

Slide 1 text

T ypes, T ests, and why flat-earthers are bad at QA 2 0 1 9 H O L Y J S   M O S C O W Twitter: @thewizardlucas GitHub: @lucasfcosta WWW: lucasfcosta.com

Slide 2

Slide 2 text

2 0 1 9 H O L Y J S   M O S C O W boredpanda.com

Slide 3

Slide 3 text

2 0 1 9 H O L Y J S   M O S C O W boredpanda.com

Slide 4

Slide 4 text

What do we know about our programs?

Slide 5

Slide 5 text

What do we know?

Slide 6

Slide 6 text

What do we know? How can we know it?

Slide 7

Slide 7 text

We must know our program to make sure it does what it is supposed to do.

Slide 8

Slide 8 text

How do we get to know the world?

Slide 9

Slide 9 text

How do we get to know the world? E M P I R I C I S M

Slide 10

Slide 10 text

How do we get to know the world? E M P I R I C I S M R A T I O N A L I S M

Slide 11

Slide 11 text

How do we get to know our programs? T E S T S T Y P E S E M P I R I C I S M R A T I O N A L I S M

Slide 12

Slide 12 text

P r o g r a m m i n g E p i s t e m o l o g y

Slide 13

Slide 13 text

Physics and Mathematics The scientific method, mathematical proofs, tests, and types.

Slide 14

Slide 14 text

Tests and Types The scientific method, mathematical proofs, tests, and types.

Slide 15

Slide 15 text

Physics and tests

Slide 16

Slide 16 text

The Scientific Method Being proven wrong and the unattainable truth

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

2 0 1 9 H O L Y J S   M O S C O W Francis Bacon

Slide 19

Slide 19 text

2 0 1 9 H O L Y J S   M O S C O W Francis Bacon Inductivism A method of reasoning

Slide 20

Slide 20 text

2 0 1 9 H O L Y J S   M O S C O W Francis Bacon Inductivism A method of reasoning Observe similar effects and similar causes and generalise.

Slide 21

Slide 21 text

2 0 1 9 H O L Y J S   M O S C O W Francis Bacon Theory Hypothesis Look for patterns Observe Inductivism A method of reasoning

Slide 22

Slide 22 text

2 0 1 9 H O L Y J S   M O S C O W Francis Bacon Inductivism A method of reasoning Observe similar effects and similar causes and generalise. The premises are viewed as supplying some evidence for the truth of the conclusion.

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

2 0 1 9 H O L Y J S   M O S C O W No confirmations of an explanation make the explanation necessarily be true.

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

"A rare bird in the lands and very much like a black swan"

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Systems of thought are way more fragile than we think.

Slide 31

Slide 31 text

Systems of thought are way more fragile than we think. A million successful experiments cannot prove a theory correct, but one failed experiment can prove a theory wrong. – POPPER, Karl

Slide 32

Slide 32 text

Truth is unattainable

Slide 33

Slide 33 text

We are constantly replacing theories by others which have greater explanatory power. Science is a successive rejection of falsified theories

Slide 34

Slide 34 text

More evidence gets us closer to the truth

Slide 35

Slide 35 text

More tests gets us closer to correct software

Slide 36

Slide 36 text

We believe what's most probable, not what's necessarily true. More tests gets us closer to correct software

Slide 37

Slide 37 text

We believe what's most probable, not what's necessarily true. Blindly trusting science is dangerous. But so is not trusting in it at all. More tests gets us closer to correct software

Slide 38

Slide 38 text

We believe what's most probable, not what's necessarily true. Blindly trusting tests is dangerous. But so is not trusting in it at all. More tests gets us closer to correct software

Slide 39

Slide 39 text

We believe what's most probable, not what's necessarily true. Science is not dogmatic. Blindly trusting tests is dangerous. But so is not trusting in it at all. More tests gets us closer to correct software

Slide 40

Slide 40 text

We believe what's most probable, not what's necessarily true. Tests are not permanent. Blindly trusting tests is dangerous. But so is not trusting in it at all. More tests gets us closer to correct software

Slide 41

Slide 41 text

How we do Science How do we know what we know? T H E S C I E N T I F I C M E T H O D

Slide 42

Slide 42 text

Form a conjecture, state an explanation How we do Science How do we know what we know? T H E S C I E N T I F I C M E T H O D

Slide 43

Slide 43 text

Form a conjecture, state an explanation How we do Science How do we know what we know? T H E S C I E N T I F I C M E T H O D Deduce predictions from the hypothesis

Slide 44

Slide 44 text

Form a conjecture, state an explanation How we do Science How do we know what we know? T H E S C I E N T I F I C M E T H O D Deduce predictions from the hypothesis Test, make experiments

Slide 45

Slide 45 text

Form a conjecture, state an explanation How do we know what we know? T H E S C I E N T I F I C M E T H O D Deduce predictions from the hypothesis Test, make experiments Observe whether your theory matches reality How we do Science

Slide 46

Slide 46 text

Form a conjecture, state an explanation How we write tests How do we know what we know? T H E S C I E N T I F I C M E T H O D Deduce predictions from the hypothesis Test, make experiments Observe whether your theory matches reality

Slide 47

Slide 47 text

Be willing to be wrong It's easy to find confirmation for a theory if you are looking for it.

Slide 48

Slide 48 text

Be willing to be wrong It's easy to find confirmation for a theory if you are looking for it. Your assertions can have a true or false value, which you don't know in advance

Slide 49

Slide 49 text

Your hypothesis must be falsifiable Be willing to be wrong It's easy to find confirmation for a theory if you are looking for it. Your assertions can have a true or false value, which you don't know in advance

Slide 50

Slide 50 text

Your hypothesis must be falsifiable Be willing to be wrong It's easy to find confirmation for a theory if you are looking for it. Your observations should not strive for confirmation, but for disconfirmation Your assertions can have a true or false value, which you don't know in advance

Slide 51

Slide 51 text

Your hypothesis must be falsifiable Be willing to be wrong It's easy to find confirmation for a theory if you are looking for it. Your tests must be risky, they need to be able to falsify your theory. Your observations should not strive for confirmation, but for disconfirmation Your assertions can have a true or false value, which you don't know in advance

Slide 52

Slide 52 text

If it never fails It's useless

Slide 53

Slide 53 text

If it can't be refuted It's useless

Slide 54

Slide 54 text

SC IE NC E DISPR OV E S It's not possible to prove a theory correct as we can't test all possible scenarios taking into account all possible variables. PSEUDOSCIENCE PROVES Irrefutable theories are not scientific. Tracing appearances, not unveiling reality. Pseudoscience does not look for arguments contrary to its affirmations.

Slide 55

Slide 55 text

In the same way that physicists cannot prove they are right with experiments, tests can't prove that assumptions about our code are right x^2 f(x) = sin(x) + 1

Slide 56

Slide 56 text

A function with two tests x^2

Slide 57

Slide 57 text

Tests do not guarantee correctness. x^2 x^4 Both functions match our predictions and pass our tests.

Slide 58

Slide 58 text

More tests, more evidence x^2 x^4 With more tests we can rule out other intersecting implementations

Slide 59

Slide 59 text

Tests can't prove that our code is correct.

Slide 60

Slide 60 text

Tests can't prove that our code is correct. Tests can only prove that it isn't

Slide 61

Slide 61 text

Tests can't prove that our code is correct. Tests can only prove that it isn't Tests provide evidence

Slide 62

Slide 62 text

Experiments can't prove that our code is correct. Experiments can only prove that it isn't Experiments provide evidence

Slide 63

Slide 63 text

Confirmation B i a s

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

Experiments depend on observation We observe reality and try to find evidence which contradicts our findings

Slide 66

Slide 66 text

Tests depend on assertions A test which does not contain assertions simply verifies whether the code can be executed.

Slide 67

Slide 67 text

Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. [...] It presupposes similarity and classification, which in their turn presuppose interests, points of view, and problems. Karl R. Popper Conjectures and Refutations: The Growth of Scientific Knowledge

Slide 68

Slide 68 text

Code  Coverage

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

I define 100% coverage as having examined all possible combinations of all possible paths through all methods of a class, having reproduced every possible configuration of data bits accessible to those methods, at every machine language instruction along the paths of execution. James O'Coplien Why most unit testing is waste

Slide 71

Slide 71 text

Slide 72

Slide 72 text

You can't prove your code works.  You can only prove it doesn't

Slide 73

Slide 73 text

Go through every single point...

Slide 74

Slide 74 text

Types & Mathematics

Slide 75

Slide 75 text

C O N J E C T U R E 1 Mathematical truth P R O V I N G C O R R E C T N E S S A statement which does not have a proof, but is believed to be true

Slide 76

Slide 76 text

P R O O F A series of steps in reasoning for demonstrating a mathematical statement is true. 1 2 Mathematical truth C O N J E C T U R E A statement which does not have a proof, but is believed to be true P R O V I N G C O R R E C T N E S S

Slide 77

Slide 77 text

P R O O F T H E O R E M A mathematical statement that is proved using rigorous mathematical reasoning C O N J E C T U R E 1 2 3 Mathematical truth Put my name on it plz A series of steps in reasoning for demonstrating a mathematical statement is true. P R O V I N G C O R R E C T N E S S A statement which does not have a proof, but is believed to be true

Slide 78

Slide 78 text

Mathematics is not a science

Slide 79

Slide 79 text

Mathematics is not a science from our point of view, in the sense that it is not a natural science. The test of its validity is not experiment. Richard P. Feyman The Feynman Lectures on Physics Vol. 1

Slide 80

Slide 80 text

Types and Mathematics If it follows the rules, it's correct. T h e Q u e s t f o r T r u t h

Slide 81

Slide 81 text

Relations of Ideas M A T H E M A T I C S P H Y S I C S Matter of Facts D A V I D H U M E ' S A P R I O R I A P O S T E R I O R I

Slide 82

Slide 82 text

Why does physics relies on mathematics?

Slide 83

Slide 83 text

Abstraction The ability of concentrating in the essential aspects of a certain context.    Remove all unnecessary detail.

Slide 84

Slide 84 text

Abstraction The ability of concentrating in the essential aspects of a certain context.    Remove all unnecessary detail. String Integer

Slide 85

Slide 85 text

Abstraction A B The ability of concentrating in the essential aspects of a certain context.    Remove all unnecessary detail.

Slide 86

Slide 86 text

Abstraction A B The ability of concentrating in the essential aspects of a certain context.    Remove all unnecessary detail.

Slide 87

Slide 87 text

Abstraction A B Bartosz Milewski The end of road for abstraction. The ability of concentrating in the essential aspects of a certain context.    Remove all unnecessary detail.

Slide 88

Slide 88 text

Making claims and proving properties A B C f g h = g . f When we have a specific set of rules we can use to manipulate symbols we can reach truth. This is the beauty of abstraction.

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

If controversies were to arise, there would be no more need of disputation between two philosophers than between two calculators. For it would suffice for them to take their pencils in their hands and to sit down at the abacus, and say to each other: "calculemus" Gottfried Wilhelm von Leibniz

Slide 91

Slide 91 text

Why don't we just use mathematics then?

Slide 92

Slide 92 text

Kurt Gödel

Slide 93

Slide 93 text

Kurt Gödel If a logical system is consistent, it cannot be complete. There will be true statements which can't be proven.

Slide 94

Slide 94 text

Kurt Gödel There is no way to show that any useful formal system is free of false statements If a logical system is consistent, it cannot be complete. There will be true statements which can't be proven.

Slide 95

Slide 95 text

Getting a bit less philosophical

Slide 96

Slide 96 text

How can we constrain the implementation of a function in such a way that that the only possible implementation is the correct one?

Slide 97

Slide 97 text

How can we rule- out all incompatible implementations?

Slide 98

Slide 98 text

Number of allowed implementations Total Possible Implementations minus Implementations Invalidated by tests https://julien-truffaut.github.io/types-vs-tests

Slide 99

Slide 99 text

Number of allowed implementations = 1 We can be sure that our function is correct when: https://julien-truffaut.github.io/types-vs-tests

Slide 100

Slide 100 text

Number of allowed implementations = 1 We can be sure that our function is correct when: The one we want https://julien-truffaut.github.io/types-vs-tests

Slide 101

Slide 101 text

https://julien-truffaut.github.io/types-vs-tests Valid Implementation Count Julien Truffaut's Smaller VICs mean more constrained functions.

Slide 102

Slide 102 text

https://julien-truffaut.github.io/types-vs-tests Valid Implementation Count = 1 Only one possible implementation: the correct one

Slide 103

Slide 103 text

Cover every possible input and output

Slide 104

Slide 104 text

It's only feasible to cover every I/O pair by constraining them

Slide 105

Slide 105 text

Through a combination of experiments and logical constraints, we can achieve more certainty.

Slide 106

Slide 106 text

How many possible values can we have for each of these?

Slide 107

Slide 107 text

19 195 7 There are now 7195 valid implementations.  Just because we added types.

Slide 108

Slide 108 text

19 195 7 For every input tested, we reduce the number of possible implementations seven-fold.

Slide 109

Slide 109 text

19 195 7 For every country tested, the possibilities of its result collapse into being only one. 7195 / 7= 7194

Slide 110

Slide 110 text

19 195 7 It's now feasible to test all countries and collapse the possible implementations into one. 7195 / 7 / 7 / 7 / 7 ... = 70 = 1

Slide 111

Slide 111 text

Type systems allow us to constrain reality in such a way that testing all possible inputs become possible.

Slide 112

Slide 112 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S E L M C O N F R i c h a r d F e l d m a n R E A C T F I N L A N D P a t r i c k S t a p f e r

Slide 113

Slide 113 text

Leveraging correctness with types

Slide 114

Slide 114 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Blackjack

Slide 115

Slide 115 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Blackjack

Slide 116

Slide 116 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S

Slide 117

Slide 117 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Only has a name! Can have a countryCode without a phone and vice-versa

Slide 118

Slide 118 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Still don't need a contact

Slide 119

Slide 119 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S

Slide 120

Slide 120 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S We can have invalid cards!  Zero, negative numbers, huge numbers...

Slide 121

Slide 121 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Can only have valid cards!

Slide 122

Slide 122 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S

Slide 123

Slide 123 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S We can still have only one card!

Slide 124

Slide 124 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S We must have two or more cards

Slide 125

Slide 125 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S lol wrong Can have zero players Can have a 'win' state with a next player Can have a tie with no tiePlayers Can have a running state with a winner

Slide 126

Slide 126 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S States are now constrained Can have a next player which has busted or surrendered!

Slide 127

Slide 127 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S

Slide 128

Slide 128 text

Making impossible states impossible F O R M A L L Y E N F O R C I N G C O R R E C T N E S S Can only have valid players now :)

Slide 129

Slide 129 text

V I E W M O R E For a next time

Slide 130

Slide 130 text

From the most complex to the most simple problems

Slide 131

Slide 131 text

All languages are typed But some will only tell you at runtime. Uncaught TypeError: undefined is not a function Uncaught TypeError: Cannot read property 'foo' of undefined

Slide 132

Slide 132 text

Prohibiting invalid properties Collapsing the number of possible props Combinatorial Explosion

Slide 133

Slide 133 text

Exhaustive Checks No unhandled actions. Ensures unhandledAction is never called! All actions are handled so this can't happen!

Slide 134

Slide 134 text

Exhaustive Checks No unhandled actions. We're not handling unknown!

Slide 135

Slide 135 text

Exhaustive Checks No unhandled actions. We're not handling unknown!

Slide 136

Slide 136 text

Modeling Uncertainty https://fsharpforfunandproﬁt.com/rop/ - Scott Wlaschin

Slide 137

Slide 137 text

Producing reliable software depends on choosing good abstractions and executing the correct experiments.

Slide 138

Slide 138 text

Leveraging correctness with tests

Slide 139

Slide 139 text

A U T O M A T E D C R A P I S S T I L L C R A P W R I T I N G T E S T S I S N O T E N O U G H

Slide 140

Slide 140 text

Coupling and cost management What is the cost of having tests?  What value does having tests produce? How brittle should my tests be?

Slide 141

Slide 141 text

Capitalism 101

Slide 142

Slide 142 text

Capitalism 101 What matters: less costs, more revenue

Slide 143

Slide 143 text

Capitalism 101 What matters: less costs, more revenue What doesn't matter: code coverage, correctness, tests

Slide 144

Slide 144 text

Tests add upfront cost But reduce long-term costs. You get back in instalments.  Tests are subject to diminishing returns. T H E P R I C E

Slide 145

Slide 145 text

You pay for tests in maintenance T E S T S A R E C O D E T O O

Slide 146

Slide 146 text

Avoid coupling. The more tests you have to change when you do a change, the bigger your cost is. T H E P R I C E

Slide 147

Slide 147 text

Tests shouldn't be too fragile nor too loose. Think about when you would like them to break. T H E P R I C E

Slide 148

Slide 148 text

Tests shouldn't be too fragile nor too loose. T H E P R I C E Tests that never fail are useless. They don't produce any information.

Slide 149

Slide 149 text

Tests shouldn't be too fragile nor too loose. T H E P R I C E Tests that never fail are not scientific.

Slide 150

Slide 150 text

Tests shouldn't be too fragile nor too loose. T H E P R I C E Tests that never fail are pseudo-science.

Slide 151

Slide 151 text

Different kinds of tests What kinds of tests produce more value? Where do tests fit in the software development process?

Slide 152

Slide 152 text

We need to talk about Test Driven Development

Slide 153

Slide 153 text

Test Driven Development is not about tests TDD is about taking small steps.  TDD is a fear reduction tool.  TDD exists to help orienting developers when they change code.

Slide 154

Slide 154 text

Test Driven Development is not about tests TDD is about taking small steps.  TDD is a fear reduction tool.  TDD exists to help orienting developers when they change code.

Slide 155

Slide 155 text

Different types of tests generate different types of value. https://martinfowler.com/articles/practical-test-pyramid.html From MartinFowler.com The Practical Test Pyramid — Written by Ham Vocke

Slide 156

Slide 156 text

Different types of tests generate different types of value. https://martinfowler.com/articles/practical-test-pyramid.html Cheap, and fast but produce relatively low value From MartinFowler.com The Practical Test Pyramid — Written by Ham Vocke

Slide 157

Slide 157 text

Different types of tests generate different types of value. https://martinfowler.com/articles/practical-test-pyramid.html Cheap, and fast but produce relatively low value Short answer: From MartinFowler.com The Practical Test Pyramid — Written by Ham Vocke

Slide 158

Slide 158 text

Different types of tests generate different types of value. https://martinfowler.com/articles/practical-test-pyramid.html Cheap, and fast but produce relatively low value Short answer: no From MartinFowler.com The Practical Test Pyramid — Written by Ham Vocke

Slide 159

Slide 159 text

Different types of tests generate different types of value. https://martinfowler.com/articles/practical-test-pyramid.html Cheap, and fast but produce relatively low value Long answer: well... From MartinFowler.com The Practical Test Pyramid — Written by Ham Vocke

Slide 160

Slide 160 text

When to write what E2E Unit Integration Snapshot

Slide 161

Slide 161 text

When should I write unit tests? E2E Unit Integration Snapshot In parallel with writing code. Unit tests guide development and help refactoring code safely. Not mainly for correctness.  As documentation for your future self. As a contract, specification of the unit under test.

Slide 162

Slide 162 text

When should I write snapshot tests? E2E Unit Integration Snapshot When asserting on output is repetitive. When output is too big and detailed to manage. Not as guidance for iterating, but as an extra safeguard against failure.

Slide 163

Slide 163 text

When should I write snapshot tests? Jest extensively uses snapshots to test itself.

Slide 164

Slide 164 text

A few tips for snapshot tests E2E Unit Integration Snapshot

Slide 165

Slide 165 text

A few tips for snapshot tests • Find relevant serialisers for your problem domain. Readable snapshots matter.

Slide 166

Slide 166 text

A few tips for snapshot tests • Find relevant serialisers for your problem domain. Readable snapshots matter.

Slide 167

Slide 167 text

A few tips for snapshot tests • Find relevant serialisers for your problem domain. Readable snapshots matter. • Use custom asymmetric snapshot matchers to balance maintainability and rigorousness

Slide 168

Slide 168 text

A few tips for snapshot tests • Find relevant serialisers for your problem domain. Readable snapshots matter. • Use custom asymmetric snapshot matchers to balance maintainability and rigorousness • Don't be afraid to have tests with partial snapshots.

Slide 169

Slide 169 text

When should I write integration tests? E2E Unit Integration Snapshot To ensure correctness and prevent regressions. To ensure you are using third party dependencies correctly. When a certain behaviour is critical to your application. To test functional requirements.

Slide 170

Slide 170 text

Practices that I consider integration tests: E2E Unit Integration Snapshot • Interacting with actual components (Enzyme/ react-testing-library) • Sending actual HTTP requests • Hitting a database and fetching data from it • Asserting on I/O (i.e. interacting with the filesystem) • Spinning separate processes

Slide 171

Slide 171 text

When should I write end-to-end tests? E2E Unit Integration Snapshot The most valuable kind of testing from a correctness perspective. When interaction with a real UI matters. To avoid visual regressions. To ensure multiple services work together from a user's perspective.

Slide 172

Slide 172 text

Can't emphasise how good this is: E2E Unit Integration Snapshot • Amazing docs • Easy access to your application's runtime environment • Not flaky (but be careful with the global chain of events!) • Extremely quick to run • Extremely easy to setup

Slide 173

Slide 173 text

Avoiding false positives How can I setup tests in such a way as to catch the most bugs? How can I avoid getting false positives?  How do assertion libraries work?

Slide 174

Slide 174 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Assertions that are loose by design • .includes • .isDefined • .increases • .decreases } The set of passing results is too broad

Slide 175

Slide 175 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Assertions that are loose by design expect(result).to.be.a.number

Slide 176

Slide 176 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Assertions that are loose by design Can go from 5e-324 to 1.7976931348623157e+308 expect(result).to.be.a.number

Slide 177

Slide 177 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Assertions that are loose by design NaN expect(result).to.be.a.number

Slide 178

Slide 178 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Negated assertions. expect(result).to.not.be(1) Negated assertions are the loosest assertions one can make. + ∞ - ∞ 0 1

Slide 179

Slide 179 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. Loose assertions are essentially assertions with a semantic "or" expect(result).to.be(1).or.to.be(2)

Slide 180

Slide 180 text

Avoid loose assertions A V O I D I N G F A L S E P O S I T I V E S Assertions which allow multiple different outputs. This is why .or has never been included in Chai.

Slide 181

Slide 181 text

Assert on one subject at a time A V O I D I N G F A L S E N E G A T I V E S expect(result).to.not.throw(TypeError, "example msg") Is it an error if both don't match?  What if one matches and the other doesn't?

Slide 182

Slide 182 text

Avoid tautological tests A V O I D I N G F A L S E P O S I T I V E S Don't test your code against itself. Build inputs and expected outputs within your testing code. Using application code to do tests means the correctness of the test depends on the correctness of the application itself. circular assertion

Slide 183

Slide 183 text

Meaningful Feedback What is the right size of a test? How to debug in a scientific manner? How do test runners work?

Slide 184

Slide 184 text

Choose the right assertions M E A N I N G F U L F E E D B A C K Assertion libraries generate information for test runners to show you meaningful output.

Slide 185

Slide 185 text

How test runners provide output M E A N I N G F U L F E E D B A C K Assertions produce AsertionError instances.

Slide 186

Slide 186 text

Diffs are the runner's responsibility M E A N I N G F U L F E E D B A C K Runners generate diffs based on the AssertionErrors thrown

Slide 187

Slide 187 text

Slide 188

Slide 188 text

Slide 189

Slide 189 text

Slide 190

Slide 190 text

Slide 191

Slide 191 text

Assertion libraries can help by generating meaningful errors M E A N I N G F U L F E E D B A C K They can omit certain parts of the stack trace and provide meaningful information about the operators used. For each part of this assertion we keep resetting what is the start of the stack frame we are going to provide. We only display the bottom stack frames, hiding our internal frames.

Slide 192

Slide 192 text

Assertions behind the scenes M E A N I N G F U L F E E D B A C K How Chai handles assertions. new Assertion({ name: "HolyJS"})

Slide 193

Slide 193 text

Accessed properties trigger getter functions which always return the assertion object (this) Each time a property is accessed, we reset the starting point of the stack. Assertions behind the scenes M E A N I N G F U L F E E D B A C K How Chai handles assertions.

Slide 194

Slide 194 text

Accessing the property deep sets a flag called "deep" in the assertion object, which indicates to the equal assertion that it should perform a deep comparison Assertions behind the scenes M E A N I N G F U L F E E D B A C K How Chai handles assertions.

Slide 195

Slide 195 text

The deep flag cannot be unset. Do one assertion at a time! Assertions behind the scenes M E A N I N G F U L F E E D B A C K How Chai handles assertions.

Slide 196

Slide 196 text

More readable tests are easier to understand and debug M E A N I N G F U L F E E D B A C K Use plugins or build your own.

Slide 197

Slide 197 text

More readable tests are easier to understand and debug M E A N I N G F U L F E E D B A C K Use plugins or build your own.

Slide 198

Slide 198 text

More readable tests are easier to understand and debug M E A N I N G F U L F E E D B A C K Use plugins or build your own.

Slide 199

Slide 199 text

Isolating external dependencies What parts of my application should I test and when? How can I eliminate dependency on external libraries? 

Slide 200

Slide 200 text

When should I mock? T E S T I S O L A T I O N Easy Answer: Mock what is not yours. Hard Answer: It depends. Unit Under Test External Library 1 External Library 2 JSDOM (DOM APIs) ./module_b.js

Slide 201

Slide 201 text

When should I mock? T E S T I S O L A T I O N Easy Answer: Mock what is not yours. Unit Under Test External Library 1 External Library 2 JSDOM (DOM APIs) ./module_b.js External libraries should have already been tested by their creators.    At most, you want to check whether the correct method was called. Test your code, not someone else's. Especially valid for DOM APIs!

Slide 202

Slide 202 text

When should I mock? T E S T I S O L A T I O N Easy Answer: Mock what is not yours. Hard Answer: It depends. Unit Under Test External Library 1 External Library 2 JSDOM (DOM APIs) ./module_b.js The more mocks you have, the more detached from reality your test becomes.   The more mocks you have, the more decoupled your test becomes. Maintenance Cost vs. Isolation How coupled to the dependency is the mock? How critical is the code under test?

Slide 203

Slide 203 text

When should I mock? T E S T I S O L A T I O N Easy Answer: Mock what is not yours. Hard Answer: It depends. Unit Under Test External Library 1 External Library 2 JSDOM (DOM APIs) ./module_b.js Maintenance Cost vs. Isolation How coupled to the dependency is the mock? How critical is the code under test? React You'll definitely want to mock this so that you can do fake pushes through the web sockets socket.io You don't want to mock this because you will be interacting with components Hard to mock. Test loses its value. You can trust the DOM APIs work. Ah, you also have no choice.

Slide 204

Slide 204 text

What do you mean by trusting browser APIs? T E S T I S O L A T I O N Trust that they work, but check that you are using them Unit Under Test External Library 1 External Library 2 JSDOM (DOM APIs) ./module_b.js

Slide 205

Slide 205 text

What do you mean by trusting browser APIs? T E S T I S O L A T I O N Trust that they work, but check that you are using them JSDOM (DOM APIs)

Slide 206

Slide 206 text

When should I mock? T E S T I S O L A T I O N Create transitive guarantees. Unit Under Test ./module_b.js ./module_c.js module_b is covered module_c is covered

Slide 207

Slide 207 text

When should I mock? T E S T I S O L A T I O N Create transitive guarantees. Unit Under Test ./module_b.js ./module_c.js module_b is covered module_c is covered mocking module_b helps you avoid redundant checks If module_c is covered I don't need to check it in module_b's tests. If UUT uses module_b and, transitively, module_c, I can mock module_b. Avoid brittleness. Avoid redundant checks. Avoid tests becoming too big.

Slide 208

Slide 208 text

How can I mock? T E S T I S O L A T I O N mocking imports proxyquire rewire rewiremock Jest's Mocks Sinon's Mocks, Stubs, and Spies If you're not using jest More custom behaviour Plugin integration Sandboxes If you're already using jest Easily mocking entire modules Can automatically clear mocks Simple and well documented API to assert on If you're not using jest Mocking on import level Depends on being paired with Sinon or another mocking code Mocking code in general.

Slide 209

Slide 209 text

How can I mock? T E S T I S O L A T I O N Mess with the requests yourself. Mocking HTTP responses. Use a specific library. nock You have full control over what's happening. Can get repetitive. Reasonably annoying to set matchers for requests. Default behaviour is not always what's best or consistent. Well defined API. No need for wrappers. Can get a bit verbose if you need to mock uncommon features. chaijs/chai-http visionmedia/supertest For your assertions:

Slide 210

Slide 210 text

Eliminating non-determinism Why does determinism matters?  How to make non-deterministic tests deterministic?

Slide 211

Slide 211 text

Why determinism matters? D E T E R M I N I S M Semantically speaking, flaky tests are the same as failing tests. Flaky tests decrease the confidence in each build. Is it a flaky test or is it flaky application code?

Slide 212

Slide 212 text

How to solve non-determinism D E T E R M I N I S M Approach 1: Mock-out non-deterministic pieces

Slide 213

Slide 213 text

How to solve non-determinism D E T E R M I N I S M Approach 2: Take variability into account This means you solve the problem on the testing side. Use loose assertions willingly! Allow broader sets of results. Use matchers

Slide 214

Slide 214 text

How to solve non-determinism D E T E R M I N I S M Approach 3: Make the code deterministic Not always possible. Ordering results within your tests. Eliminating the usage of randomness. Providing a deterministic state or seed.

Slide 215

Slide 215 text

Speeding-up test runs Why does quick feedback matters?  How can I speed up test runs? How are my tests scheduled?

Slide 216

Slide 216 text

Why does quick feedback matters? Q U I C K F E E D B A C K Quick feedback encourages you to take more gradual steps (proper TDD). If tests are slow, they won't get run frequently enough. Quick feedback decreases frustration, creating a positive feedback loop.

Slide 217

Slide 217 text

No content

Slide 218

Slide 218 text

2 0 1 9 H O L Y J S   M O S C O W Good tests kill flawed theories; we remain alive to guess again. — POPPER, Karl

Slide 219

Slide 219 text

Write code, read books 2 0 1 9 H O L Y J S   M O S C O W Twitter: @thewizardlucas GitHub: @lucasfcosta WWW: lucasfcosta.com

Slide 220

Slide 220 text

Thank you. 2 0 1 9 H O L Y J S   M O S C O W Twitter: @thewizardlucas GitHub: @lucasfcosta WWW: lucasfcosta.com