Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Cost Tests and High Value Tests

Noel Rappin
November 16, 2017

High Cost Tests and High Value Tests

There is a value in writing tests and there is also a cost. The currency is time. The trade-offs are difficult to evaluate because the cost and value are often seen by different people. The writer of the test bears much of the short term cost while long term benefits and cost are borne by the rest of the team. By planning around both the the cost and value of your tests, you’ll improve your tests and your code. How much do slow tests cost? When is it worth it to test an edge case? How can you tell if testing is helping? Here are some strategies to improve your tests and code.

Noel Rappin

November 16, 2017
Tweet

More Decks by Noel Rappin

Other Decks in Technology

Transcript

  1. This slide saves me time in the long run even

    though it took time to create Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  2. When I get a feature request Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  3. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  4. I need to decide how to test Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  5. I could Outside-In TDD Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  6. I could just write integration tests Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  7. I could not test Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  8. I'm making a decision Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  9. Testing will be worth it Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  10. What does worth it mean? ! Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  11. How do I know? Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  12. High Cost Tests & High Value Tests Noel Rappin (@noelrap)

    https://www.tablexi.com http://techdoneright.io http://www.noelrappin.com http://pragprog.com/book/nrtest3 Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  13. How can you measure cost and value? Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  14. Tests are at the intersection of: Code Process Design Noel

    Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  15. Time is our metric Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  16. How do tests cost time? Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  17. You have to write the test Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  18. The test runs A lot Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  19. The test needs to be understood Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  20. The test needs to be fixed Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  21. How do tests save time? Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  22. Writing the test improves code design Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  23. Running the test is faster than manual testing Noel Rappin

    | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  24. The test validates the code Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  25. Catches Bugs Faster Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  26. Cost ! vs Value ! Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  27. Dev: ! " Forever: ! " # Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  28. Spoiler alert: There is no right answer Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  29. Strategy not Tactics Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  30. Some data Noel Rappin | High Cost Tests and High

    Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  31. End-to-End Integration Capybara Starts With Ends With User Input HTML

    Output Write Time Run Time 30 mins 0.5 - 3 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  32. Workflow Intermediate Object Starts Ends Params and workflow.run Database changes

    Write Time Run Time 15 mins 0.05 - 0.3 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  33. Unit One method Starts Ends Call a method Output of

    that method Write Time Run Time 1-5 mins 0.001 - 0.04 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  34. Type Specs Total run time Avg run time Write time

    System 22 12.72 0.570 ~11 hrs Workflow 40 2.36 0.059 ~10 hrs Unit 119 1.86 0.015 ~10 hrs Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  35. System tests are 12% of the code and 75% of

    the output Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  36. The slowest 4 tests are 40% of the run time

    Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  37. The run times have a wider range Kind Min Max

    Variance Write 1 min 30 min 30x Run 0.001 sec 3 sec 3000x Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  38. Another project Noel Rappin | High Cost Tests and High

    Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  39. Type Specs Total time Avg time Cucumber 116 520 4.48

    System 175 110 0.62 Workflow 493 209 0.42 Unit 772 92 0.19 Total 1571 987 6.28 Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  40. End to end tests are 18% of tests and 67%

    of run time Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  41. What does that suggest? Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  42. Balance time spent Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  43. As you write similar tests, costs go down Noel Rappin

    | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  44. Short term cost ! " is not related to test

    type Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  45. Long term cost ! " # is Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  46. Long term cost: runtime failure Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  47. In other words: complexity Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  48. Long term savings come from focus Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  49. A small fraction of your tests can be the bulk

    of your cost Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  50. Big payoff in avoiding the slowest tests Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  51. That's a long way to get to "don't write slow

    tests" Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  52. No individual test causes a slow suite Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  53. It's an aggregate set of decisions Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  54. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  55. First test: Capybara integration Fails on: • View failure •

    Controller failure • Handoff from controller to logic failure • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  56. Next test: workflow object Fails on: • Logic failure •

    Database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  57. Maybe: Unit test Fails on • specific bit of logic

    • database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  58. Failure paths: Bad Input "A", -3, "" Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  59. System Test Workflow Test Unit test Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  60. Unit test A partial test of the workflow Or a

    model test Or a logic object Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  61. New feature blank row that can become a new item

    Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  62. Workflow test Noel Rappin | High Cost Tests and High

    Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  63. New bug a new row that duplicates a name Noel

    Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  64. Not a system test workflow or partial workflow test Noel

    Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  65. Signs your test is too big? Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  66. A lot of Copy/Paste Noel Rappin | High Cost Tests

    and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  67. A lot of unrelated assertions Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  68. Logic change is far away from method under test Noel

    Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  69. Capybara is not a unit test framework Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  70. What if you don't like unit tests? Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  71. "TDD is Dead" Noel Rappin | High Cost Tests and

    High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  72. Saw this as JS testing advice: Write tests Not Too

    Many Mostly integration Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  73. Unit tests cost too much to write Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  74. Often true in a legacy context Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  75. True if framework doesn't handle tests Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  76. Unit tests force hard to understand designs Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  77. True if you don't like small units Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  78. ! matters less if you are not running tests while

    developing Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  79. Really good legacy code strategy Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  80. Cost of harder to diagnose tests Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  81. Strategies Noel Rappin | High Cost Tests and High Value

    Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  82. What will make a test fail? Noel Rappin | High

    Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  83. It it can't fail uniquely Do you need it? Noel

    Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  84. Create the minimum amount of objects needed to trigger the

    failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  85. Use multiple test failures as an opportunity Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  86. Sometimes you can delete tests Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  87. Use integration tests to save development time Noel Rappin |

    High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  88. Tests have costs ! " Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  89. Tests have value ! " Noel Rappin | High Cost

    Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
  90. Noel Rappin (@noelrap) http://pragprog.com/book/nrtest3 (15% off with code "RubyConf NOLA")

    & http://techdoneright.io/ Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap