This slide saves me time in
the long run even though it
took time to create
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 2
Slide 2 text
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 3
Slide 3 text
When I get a
feature request
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 4
Slide 4 text
Inventory management software
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 5
Slide 5 text
I need to decide
how to test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 6
Slide 6 text
I could
Outside-In
TDD
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 7
Slide 7 text
I could just write
integration
tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 8
Slide 8 text
I could
not test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 9
Slide 9 text
I'm making a
decision
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 10
Slide 10 text
Testing will be
worth it
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 11
Slide 11 text
What does worth it
mean?
!
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 12
Slide 12 text
How do I
know?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 13
Slide 13 text
High Cost Tests &
High Value Tests
Noel Rappin (@noelrap)
https://www.tablexi.com
http://techdoneright.io
http://www.noelrappin.com
http://pragprog.com/book/nrtest3
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 14
Slide 14 text
How can you
measure
cost and value?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 15
Slide 15 text
Tests are at
the
intersection
of:
Code
Process
Design
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 16
Slide 16 text
Time
is our metric
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 17
Slide 17 text
How do tests
cost time?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 18
Slide 18 text
You have to
write
the test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 19
Slide 19 text
The test
runs
A lot
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 20
Slide 20 text
The test needs to
be
understood
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 21
Slide 21 text
The test needs to
be
fixed
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 22
Slide 22 text
How do tests
save time?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 23
Slide 23 text
Writing the test
improves
code design
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 24
Slide 24 text
Running the test
is
faster
than manual
testing
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 25
Slide 25 text
The test
validates
the code
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 26
Slide 26 text
Catches
Bugs
Faster
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 27
Slide 27 text
Cost
!
vs
Value
!
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 28
Slide 28 text
Dev:
! "
Forever:
! " #
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 29
Slide 29 text
Spoiler alert:
There is no right answer
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 30
Slide 30 text
Strategy
not
Tactics
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 31
Slide 31 text
Some data
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 32
Slide 32 text
End-to-End Integration
Capybara
Starts With Ends With
User Input HTML Output
Write Time Run Time
30 mins 0.5 - 3 seconds
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 33
Slide 33 text
Workflow
Intermediate Object
Starts Ends
Params and workflow.run Database changes
Write Time Run Time
15 mins 0.05 - 0.3 seconds
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 34
Slide 34 text
Unit
One method
Starts Ends
Call a method Output of that method
Write Time Run Time
1-5 mins 0.001 - 0.04 seconds
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 35
Slide 35 text
Type Specs Total run
time
Avg run
time
Write
time
System 22 12.72 0.570 ~11 hrs
Workflow 40 2.36 0.059 ~10 hrs
Unit 119 1.86 0.015 ~10 hrs
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 36
Slide 36 text
System tests are
12% of the tests
and 75% of the run
time
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 37
Slide 37 text
The slowest 4
tests are 40% of
the run time
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 38
Slide 38 text
The run times have a wider
range
Kind Min Max Variance
Write 1 min 30 min 30x
Run 0.001 sec 3 sec 3000x
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 39
Slide 39 text
Another project
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 40
Slide 40 text
Type Specs Total time Avg time
System 409 579 1.42
Workflow 534 206 0.38
Unit 773 93 0.12
Total 1716 878 0.51
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 41
Slide 41 text
End to end tests are 23% of
tests and 66% of run time
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 42
Slide 42 text
What does that
suggest?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 43
Slide 43 text
Balance
time spent
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 44
Slide 44 text
As you write
similar tests,
costs go
down
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 45
Slide 45 text
Short term cost
! "
is not related to
test type
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 46
Slide 46 text
Long term cost
! " #
is
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 47
Slide 47 text
Long term cost:
runtime
failure
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 48
Slide 48 text
In other words:
complexity
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 49
Slide 49 text
Long term savings
come from
focus
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 50
Slide 50 text
A small fraction
of your tests can
be the
bulk
of your cost
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 51
Slide 51 text
Big payoff in
avoiding
the slowest tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 52
Slide 52 text
That's a long way
to get to
"don't write slow
tests"
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 53
Slide 53 text
No individual test
causes a slow
suite
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 54
Slide 54 text
It's an aggregate
set of decisions
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 55
Slide 55 text
Is a slow test
suite
inevitable?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 56
Slide 56 text
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 57
Slide 57 text
!
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 58
Slide 58 text
Test suites get
longer as the code
gets more
complicated.
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 59
Slide 59 text
Only CI runs all
the tests...
we can throw
hardware at that
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 60
Slide 60 text
As long as I can
run the tests I'm
working on
quickly, I'm fine
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 61
Slide 61 text
A short history of
the Rails
community's
thoughts about
testing...
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 62
Slide 62 text
1. Testing is
great
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 63
Slide 63 text
2. Testing is slow
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 64
Slide 64 text
3. Let's try and
make testing
faster
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 65
Slide 65 text
4. That's hard.
Let's throw CI at
it.
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 66
Slide 66 text
There's still a
cost to a long
suite
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 67
Slide 67 text
Giving up causes
you to lose the
value of tests in
improving code
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 68
Slide 68 text
Inventory management software
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 69
Slide 69 text
First test: Capybara
integration
Fails on:
• View failure
• Controller failure
• Handoff from controller to
logic failure
• Logic failure
• Database access failure
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 70
Slide 70 text
Next test: workflow
object
Fails on:
• Logic failure
• Database access failure
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 71
Slide 71 text
Maybe: Unit test
Fails on
• specific bit of logic
• database access failure
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 72
Slide 72 text
Failure paths:
Bad Input
"A", -3, ""
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 73
Slide 73 text
System Test
Workflow Test
Unit test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 74
Slide 74 text
Unit test
A partial test of
the workflow
Or a model test
Or a logic object
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 75
Slide 75 text
New feature
blank row that can become a
new item
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 76
Slide 76 text
Workflow test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 77
Slide 77 text
New bug
a new row that
duplicates a name
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 78
Slide 78 text
Not a system test
workflow or
partial workflow
test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 79
Slide 79 text
What if you don't
like unit tests?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 80
Slide 80 text
"TDD is
Dead"
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 81
Slide 81 text
Argument against
unit tests:
Unit tests cost
too much
to write
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 82
Slide 82 text
Often true in a
legacy context
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 83
Slide 83 text
True if framework
doesn't handle
tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 84
Slide 84 text
You might hear:
Unit tests cause
hard to understand
designs
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 85
Slide 85 text
True if you don't
like small units
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 86
Slide 86 text
You might just be
writing
bad
unit tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 87
Slide 87 text
A lot of
Copy/Paste
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 88
Slide 88 text
A lot of unrelated
assertions
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 89
Slide 89 text
Logic change is
far away
from method under
test
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 90
Slide 90 text
Capybara is not a
unit test
framework
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 91
Slide 91 text
Lack of
unit tests:
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 92
Slide 92 text
Good
legacy
code strategy
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 93
Slide 93 text
At the cost of
harder to
diagnose
tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 94
Slide 94 text
Strategies
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 95
Slide 95 text
What will make a
test
fail?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 96
Slide 96 text
If it can't fail
uniquely
Do you need it?
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 97
Slide 97 text
Create the minimum
amount of objects
needed to trigger
the failure
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 98
Slide 98 text
Use multiple test
failures as an
opportunity
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 99
Slide 99 text
Sometimes you can
delete
tests
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 100
Slide 100 text
Use integration
tests to save
development time
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 101
Slide 101 text
Tests have
costs
! "
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 102
Slide 102 text
Tests have
value
! "
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
Slide 103
Slide 103 text
Noel Rappin
(@noelrap)
http://pragprog.com/book/nrtest3
workshops@tablexi.com
http://techdoneright.io/
Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap