Automated Tests Aren't Enough

Slide 1

Slide 1 text

Automated Tests Aren’t Enough @danlew42

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Can we get by with fewer testers?

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Markdown Rewrite! Experience Specs Testability https://www.ﬂickr.com/photos/albertogp123/5843577306

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Automated Tests Weren’t Enough Based on a true story

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

There were plenty of tests… …But they were incorrect!

Slide 15

Slide 15 text

Automation cannot prove correctness

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Correctness Humans Subjective

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

There were plenty of tests… …But the UI was ugly!

Slide 21

Slide 21 text

Automation cannot prove goodness

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

https://www.ﬂickr.com/photos/eugene/186551247/

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

Goodness Humans Subjective

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

There were plenty of tests… …But I had a blindspot!

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

There were plenty of tests… …Only worked on my device!

Slide 30

Slide 30 text

Automation cannot prove coverage

Slide 31

Slide 31 text

https://www.ﬂickr.com/photos/49580580@N02/41906545634/

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Coverage Humans Exploration

Slide 34

Slide 34 text

Automation Can Be Expensive • 100% code coverage is nearly impossible • UI testing sucks • Tests require maintenance

Slide 35

Slide 35 text

Nomenclature • Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc. • Checking is the process of making evaluations by applying algorithmic decision rules to speciﬁc observations of a product. • James Bach & Michael Bolton - satisﬁce.com/blog/archives/856

Slide 36

Slide 36 text

❤ Automation

Slide 37

Slide 37 text

❤ Automation Manual +

Slide 38

Slide 38 text

Manual Testing Strengths Weaknesses • Exploratory • Tests correctness • Tests goodness • Easy to “write” and “update” • Black-box testing only • Unreliable • Time intensive • Diﬃcult reproductions

Slide 39

Slide 39 text

Automated Checks Strengths Weaknesses • Test any layer • Repeatable • Fast • Easy reproduction • Cannot explore • Cannot prove correctness • Cannot prove goodness • Expensive to maintain

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text