This slide saves me time in
the long run even though it
took time to create
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 2
Slide 2 text
When I get a
feature request
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 3
Slide 3 text
Inventory management software
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 4
Slide 4 text
I need to decide
how to test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 5
Slide 5 text
I could
Outside-In
TDD
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 6
Slide 6 text
I could just write
integration
tests
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 7
Slide 7 text
I could
not test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 8
Slide 8 text
I'm making a
decision
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 9
Slide 9 text
Testing will be
worth it
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 10
Slide 10 text
What does worth it
mean?
!
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 11
Slide 11 text
How do I
know?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 12
Slide 12 text
High Cost Tests &
High Value Tests
Noel Rappin (@noelrap)
https://www.tablexi.com
http://techdoneright.io
http://www.noelrappin.com
http://pragprog.com/book/nrtest3
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 13
Slide 13 text
How can you
measure
cost and value?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 14
Slide 14 text
Tests are at
the
intersection
of:
Code
Process
Design
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 15
Slide 15 text
Time
is our metric
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 16
Slide 16 text
How do tests
cost time?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 17
Slide 17 text
You have to
write
the test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 18
Slide 18 text
The test
runs
A lot
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 19
Slide 19 text
The test needs to
be
understood
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 20
Slide 20 text
The test needs to
be
fixed
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 21
Slide 21 text
How do tests
save time?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 22
Slide 22 text
Writing the test
improves
code design
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 23
Slide 23 text
Running the test
is
faster
than manual
testing
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 24
Slide 24 text
The test
validates
the code
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 25
Slide 25 text
Catches
Bugs
Faster
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 26
Slide 26 text
Cost
!
vs
Value
!
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 27
Slide 27 text
Dev:
! "
Forever:
! " #
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 28
Slide 28 text
Spoiler alert:
There is no right answer
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 29
Slide 29 text
Strategy
not
Tactics
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 30
Slide 30 text
Some data
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 31
Slide 31 text
End-to-End Integration
Capybara
Starts With Ends With
User Input HTML Output
Write Time Run Time
30 mins 0.5 - 3 seconds
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 32
Slide 32 text
Workflow
Intermediate Object
Starts Ends
Params and workflow.run Database changes
Write Time Run Time
15 mins 0.05 - 0.3 seconds
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 33
Slide 33 text
Unit
One method
Starts Ends
Call a method Output of that method
Write Time Run Time
1-5 mins 0.001 - 0.04 seconds
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 34
Slide 34 text
Type Specs Total run
time
Avg run
time
Write
time
System 22 12.72 0.570 ~11 hrs
Workflow 40 2.36 0.059 ~10 hrs
Unit 119 1.86 0.015 ~10 hrs
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 35
Slide 35 text
System tests are
12% of the code
and 75% of the
output
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 36
Slide 36 text
The slowest 4
tests are 40% of
the run time
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 37
Slide 37 text
The run times have a wider
range
Kind Min Max Variance
Write 1 min 30 min 30x
Run 0.001 sec 3 sec 3000x
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 38
Slide 38 text
Another project
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 39
Slide 39 text
Type Specs Total time Avg time
Cucumber 116 520 4.48
System 175 110 0.62
Workflow 493 209 0.42
Unit 772 92 0.19
Total 1571 987 6.28
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 40
Slide 40 text
End to end tests are 18% of
tests and 67% of run time
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 41
Slide 41 text
What does that
suggest?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 42
Slide 42 text
Balance
time spent
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 43
Slide 43 text
As you write
similar tests,
costs go
down
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 44
Slide 44 text
Short term cost
! "
is not related to
test type
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 45
Slide 45 text
Long term cost
! " #
is
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 46
Slide 46 text
Long term cost:
runtime
failure
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 47
Slide 47 text
In other words:
complexity
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 48
Slide 48 text
Long term savings
come from
focus
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 49
Slide 49 text
A small fraction
of your tests can
be the
bulk
of your cost
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 50
Slide 50 text
Big payoff in
avoiding
the slowest tests
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 51
Slide 51 text
That's a long way
to get to
"don't write slow
tests"
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 52
Slide 52 text
No individual test
causes a slow
suite
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 53
Slide 53 text
It's an aggregate
set of decisions
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 54
Slide 54 text
Inventory management software
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 55
Slide 55 text
First test: Capybara
integration
Fails on:
• View failure
• Controller failure
• Handoff from controller to
logic failure
• Logic failure
• Database access failure
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 56
Slide 56 text
Next test: workflow
object
Fails on:
• Logic failure
• Database access failure
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 57
Slide 57 text
Maybe: Unit test
Fails on
• specific bit of logic
• database access failure
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 58
Slide 58 text
Failure paths:
Bad Input
"A", -3, ""
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 59
Slide 59 text
System Test
Workflow Test
Unit test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 60
Slide 60 text
Unit test
A partial test of
the workflow
Or a model test
Or a logic object
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 61
Slide 61 text
New feature
blank row that can become a
new item
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 62
Slide 62 text
Workflow test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 63
Slide 63 text
New bug
a new row that
duplicates a name
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 64
Slide 64 text
Not a system test
workflow or
partial workflow
test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 65
Slide 65 text
Signs your test is
too
big?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 66
Slide 66 text
A lot of
Copy/Paste
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 67
Slide 67 text
A lot of unrelated
assertions
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 68
Slide 68 text
Logic change is
far away
from method under
test
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 69
Slide 69 text
Capybara is not a
unit test
framework
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 70
Slide 70 text
What if you don't
like unit tests?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 71
Slide 71 text
"TDD is
Dead"
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 72
Slide 72 text
Saw this as JS
testing advice:
Write tests
Not Too Many
Mostly integration
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 73
Slide 73 text
Unit tests cost
too much
to write
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 74
Slide 74 text
Often true in a
legacy context
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 75
Slide 75 text
True if framework
doesn't handle
tests
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 76
Slide 76 text
Unit tests force
hard to
understand
designs
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 77
Slide 77 text
True if you don't
like small units
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 78
Slide 78 text
!
matters less if
you are not
running tests
while developing
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 79
Slide 79 text
Really good
legacy
code strategy
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 80
Slide 80 text
Cost of harder to
diagnose
tests
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 81
Slide 81 text
Strategies
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 82
Slide 82 text
What will make a
test
fail?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 83
Slide 83 text
It it can't fail
uniquely
Do you need it?
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 84
Slide 84 text
Create the minimum
amount of objects
needed to trigger
the failure
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 85
Slide 85 text
Use multiple test
failures as an
opportunity
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 86
Slide 86 text
Sometimes you can
delete
tests
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 87
Slide 87 text
Use integration
tests to save
development time
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 88
Slide 88 text
Tests have
costs
! "
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 89
Slide 89 text
Tests have
value
! "
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
Slide 90
Slide 90 text
Noel Rappin
(@noelrap)
http://pragprog.com/book/nrtest3
(15% off with code "RubyConf NOLA")
&
http://techdoneright.io/
Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap