Slide 1
Slide 1 text
@MaritvanDijk77
github.com/mlvandijk
medium.com/@mlvandijk
Great test automation does not
absolve you from manual /
exploratory testing
TestBash Utrecht – 24 May 2019
Slide 2
Slide 2 text
About me
• Technical Test Engineer at bol.com (NL)
• Core contributor to Cucumber
• Organizer European Testing Conference
@MaritvanDijk77
Slide 3
Slide 3 text
Recent years
•More test automation
@MaritvanDijk77
Slide 4
Slide 4 text
Recent years
•More test automation
•No more dedicated testers
@MaritvanDijk77
Slide 5
Slide 5 text
Recent years
•More test automation
•No more dedicated testers
•No more manual testing - at all
@MaritvanDijk77
Slide 6
Slide 6 text
Confession time
@MaritvanDijk77
Slide 7
Slide 7 text
Confession time
•I was not a tester
@MaritvanDijk77
Slide 8
Slide 8 text
Confession time
•I was not a tester
•I got into test automation by “accident”
@MaritvanDijk77
Slide 9
Slide 9 text
Automate all the things
(meme)
@MaritvanDijk77
Slide 10
Slide 10 text
But…
•We cannot automate everything
@MaritvanDijk77
Slide 11
Slide 11 text
But…
•We cannot automate everything
•Automation cannot tell you
what it’s like to actually use it
@MaritvanDijk77
Slide 12
Slide 12 text
“If you don’t test it,
your users will.”
Lanette Creamer
@lanettecream
@MaritvanDijk77
Slide 13
Slide 13 text
But…
•We cannot automate everything
•Automation cannot tell you
what it’s like to actually use it
•Unknown unknowns
@MaritvanDijk77
Slide 14
Slide 14 text
But…
•We cannot automate everything
•Automation cannot tell you
what it’s like to actually use it
•Unknown unknowns
•Costs vs benefit
@MaritvanDijk77
Slide 15
Slide 15 text
E2E Tests are hard
@MaritvanDijk77
Slide 16
Slide 16 text
E2E Tests are hard
•Consistent state in several systems
@MaritvanDijk77
Slide 17
Slide 17 text
E2E Tests are hard
•Consistent state in several systems
•Collaboration between teams / depts
@MaritvanDijk77
Slide 18
Slide 18 text
Context
•50 – 60 teams
•Autonomous
•Microservices
@MaritvanDijk77
Slide 19
Slide 19 text
Our test strategy
•Unit test – fast, cheap
(close to the logic)
@MaritvanDijk77
E2E
Integration
tests
Unit tests
Slide 20
Slide 20 text
Our test strategy
•Integration – slower
(preferably in isolation)
•Unit test – fast, cheap
(close to the logic)
@MaritvanDijk77
E2E
Integration
tests
Unit tests
Slide 21
Slide 21 text
Our test strategy
•E2E – very slow & brittle
(esp. on shared environment)
•Integration – slower
(preferably in isolation)
•Unit test – fast, cheap
(close to the logic)
@MaritvanDijk77
E2E
Integration
tests
Unit tests
Slide 22
Slide 22 text
Tested components: unit tests
•Functions / (small) classes
•Errors / Exceptions
@MaritvanDijk77
Unit Unit Unit
Integration
Integration
Slide 23
Slide 23 text
Tested components: integration
•Integration between components (e.g. database)
@MaritvanDijk77
Unit Unit Unit
Integration
Integration
Slide 24
Slide 24 text
Tested components: integration
•Integration between components (e.g. database)
•Communication between systems (e.g. other services)
@MaritvanDijk77
Unit Unit Unit
Integration
Integration
Slide 25
Slide 25 text
Tested whole?
If we have tested all components,
the whole system should just work… right?
@MaritvanDijk77
Unit Unit Unit
Integration
Integration
Slide 26
Slide 26 text
Here be bugs
Well…
@MaritvanDijk77
Unit Unit Unit
Integration
Integration
Slide 27
Slide 27 text
Example 1: Rebuilding a workflow system
+ Familiar with process
@MaritvanDijk77
Slide 28
Slide 28 text
Example 1: Rebuilding a workflow system
+ Familiar with process
+ Close cooperation with BA
@MaritvanDijk77
Slide 29
Slide 29 text
Example 1: Rebuilding a workflow system
+ Familiar with process
+ Close cooperation with BA
+ Unit & integration tests
@MaritvanDijk77
Slide 30
Slide 30 text
Overview
Workflow
System
Other
System
Other
System
Other
System
Legend
“ours”
not “ours”
Other
System
Other
System
Slide 31
Slide 31 text
Pair testing
•Technical knowledge (dev)
•Functional knowledge (test)
@MaritvanDijk77
Slide 32
Slide 32 text
Benefits of pair testing
+ Less error prone
@MaritvanDijk77
Slide 33
Slide 33 text
Benefits of pair testing
+ Less error prone
+ No time wasted on reproduction
@MaritvanDijk77
Slide 34
Slide 34 text
Benefits of pair testing
+ Less error prone
+ No time wasted on reproduction
+ Bugs were fixed quickly
@MaritvanDijk77
Slide 35
Slide 35 text
Bug collection
Workflow
System
Other
System
Other
System
Other
System
Legend
“ours”
not “ours”
Other
System
Other
System
Slide 36
Slide 36 text
Example 2: Replacing an exchange system
+ Replace as is
@MaritvanDijk77
Slide 37
Slide 37 text
Example 2: Replacing an exchange system
+ Replace as is
+ Close cooperation with developer
@MaritvanDijk77
Slide 38
Slide 38 text
Example 2: Replacing an exchange system
+ Replace as is
+ Close cooperation with developer
+ Unit & integration tests
@MaritvanDijk77
Slide 39
Slide 39 text
Example 2: Replacing an exchange system
+ Replace as is
+ Close cooperation with developer
+ Unit & integration tests
+ Shadow run on production (@ngurov)
@MaritvanDijk77
Slide 40
Slide 40 text
Example 2: Replacing an exchange system
+ Replace as is
+ Close cooperation with developer
+ Unit & integration tests
+ Shadow run on production
+ Phased go-live
@MaritvanDijk77
Slide 41
Slide 41 text
Overview
Exchange
System
Other
System
Other
System
Other
System
Other
System
Other
System
Legend
“ours”
not “ours”
Other
System
Other
System
Slide 42
Slide 42 text
Other
System
Other
System
Bug collection
Exchange
System
Other
System
Other
System
Other
System
Other
System
Other
System
Legend
“ours”
not “ours”
Other
System
Other
System
Slide 43
Slide 43 text
We cannot automate everything
•What it’s like to actually use it
•Unknown unknowns
@MaritvanDijk77
Slide 44
Slide 44 text
TL; DR: Test your code!
https://www.redbubble.com/people/hogfish (Trish Khoo)
@MaritvanDijk77
Slide 45
Slide 45 text
@MaritvanDijk77
github.com/mlvandijk
medium.com/@mlvandijk
Great test automation does not
absolve you from manual /
exploratory testing
TestBash Utrecht – 24 May 2019