Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Cost Tests and High Value Tests: GOTO Chicago 2018

High Cost Tests and High Value Tests: GOTO Chicago 2018

There is a value in writing tests and there is also a cost. The currency is time. The trade-offs are difficult to evaluate because the cost and value are often seen by different people. The writer of the test bears much of the short term cost while long term benefits and cost are borne by the rest of the team. By planning around both the the cost and value of your tests, you’ll improve your tests and your code. How much do slow tests cost? When is it worth it to test an edge case? How can you tell if testing is helping? Here are some strategies to improve your tests and code.

6b767d8a4c9910e007c122d81eb4de73?s=128

Noel Rappin

April 26, 2018
Tweet

More Decks by Noel Rappin

Other Decks in Technology

Transcript

  1. This slide saves me time in the long run even

    though it took time to create Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  2. Noel Rappin | High Cost Tests and High Value Tests

    | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  3. When I get a feature request Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  4. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  5. I need to decide how to test Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  6. I could Outside-In TDD Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  7. I could just write integration tests Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  8. I could not test Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  9. I'm making a decision Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  10. Testing will be worth it Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  11. What does worth it mean? ! Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  12. How do I know? Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  13. High Cost Tests & High Value Tests Noel Rappin (@noelrap)

    https://www.tablexi.com http://techdoneright.io http://www.noelrappin.com http://pragprog.com/book/nrtest3 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  14. How can you measure cost and value? Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  15. Tests are at the intersection of: Code Process Design Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  16. Time is our metric Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  17. How do tests cost time? Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  18. You have to write the test Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  19. The test runs A lot Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  20. The test needs to be understood Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  21. The test needs to be fixed Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  22. How do tests save time? Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  23. Writing the test improves code design Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  24. Running the test is faster than manual testing Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  25. The test validates the code Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  26. Catches Bugs Faster Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  27. Cost ! vs Value ! Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  28. Dev: ! " Forever: ! " # Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  29. Spoiler alert: There is no right answer Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  30. Strategy not Tactics Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  31. Some data Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  32. End-to-End Integration Capybara Starts With Ends With User Input HTML

    Output Write Time Run Time 30 mins 0.5 - 3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  33. Workflow Intermediate Object Starts Ends Params and workflow.run Database changes

    Write Time Run Time 15 mins 0.05 - 0.3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  34. Unit One method Starts Ends Call a method Output of

    that method Write Time Run Time 1-5 mins 0.001 - 0.04 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  35. Type Specs Total run time Avg run time Write time

    System 22 12.72 0.570 ~11 hrs Workflow 40 2.36 0.059 ~10 hrs Unit 119 1.86 0.015 ~10 hrs Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  36. System tests are 12% of the tests and 75% of

    the run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  37. The slowest 4 tests are 40% of the run time

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  38. The run times have a wider range Kind Min Max

    Variance Write 1 min 30 min 30x Run 0.001 sec 3 sec 3000x Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  39. Another project Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  40. Type Specs Total time Avg time System 409 579 1.42

    Workflow 534 206 0.38 Unit 773 93 0.12 Total 1716 878 0.51 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  41. End to end tests are 23% of tests and 66%

    of run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  42. What does that suggest? Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  43. Balance time spent Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  44. As you write similar tests, costs go down Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  45. Short term cost ! " is not related to test

    type Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  46. Long term cost ! " # is Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  47. Long term cost: runtime failure Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  48. In other words: complexity Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  49. Long term savings come from focus Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  50. A small fraction of your tests can be the bulk

    of your cost Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  51. Big payoff in avoiding the slowest tests Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  52. That's a long way to get to "don't write slow

    tests" Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  53. No individual test causes a slow suite Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  54. It's an aggregate set of decisions Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  55. Is a slow test suite inevitable? Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  56. Noel Rappin | High Cost Tests and High Value Tests

    | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  57. ! Noel Rappin | High Cost Tests and High Value

    Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  58. Test suites get longer as the code gets more complicated.

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  59. Only CI runs all the tests... we can throw hardware

    at that Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  60. As long as I can run the tests I'm working

    on quickly, I'm fine Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  61. A short history of the Rails community's thoughts about testing...

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  62. 1. Testing is great Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  63. 2. Testing is slow Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  64. 3. Let's try and make testing faster Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  65. 4. That's hard. Let's throw CI at it. Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  66. There's still a cost to a long suite Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  67. Giving up causes you to lose the value of tests

    in improving code Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  68. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  69. First test: Capybara integration Fails on: • View failure •

    Controller failure • Handoff from controller to logic failure • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  70. Next test: workflow object Fails on: • Logic failure •

    Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  71. Maybe: Unit test Fails on • specific bit of logic

    • database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  72. Failure paths: Bad Input "A", -3, "" Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  73. System Test Workflow Test Unit test Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  74. Unit test A partial test of the workflow Or a

    model test Or a logic object Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  75. New feature blank row that can become a new item

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  76. Workflow test Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  77. New bug a new row that duplicates a name Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  78. Not a system test workflow or partial workflow test Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  79. What if you don't like unit tests? Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  80. "TDD is Dead" Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  81. Argument against unit tests: Unit tests cost too much to

    write Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  82. Often true in a legacy context Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  83. True if framework doesn't handle tests Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  84. You might hear: Unit tests cause hard to understand designs

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  85. True if you don't like small units Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  86. You might just be writing bad unit tests Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  87. A lot of Copy/Paste Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  88. A lot of unrelated assertions Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  89. Logic change is far away from method under test Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  90. Capybara is not a unit test framework Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  91. Lack of unit tests: Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  92. Good legacy code strategy Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  93. At the cost of harder to diagnose tests Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  94. Strategies Noel Rappin | High Cost Tests and High Value

    Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  95. What will make a test fail? Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  96. If it can't fail uniquely Do you need it? Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  97. Create the minimum amount of objects needed to trigger the

    failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  98. Use multiple test failures as an opportunity Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  99. Sometimes you can delete tests Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  100. Use integration tests to save development time Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  101. Tests have costs ! " Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  102. Tests have value ! " Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
  103. Noel Rappin (@noelrap) http://pragprog.com/book/nrtest3 workshops@tablexi.com http://techdoneright.io/ Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap