Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Strategy: New Model, Better Outcomes

Testing Strategy: New Model, Better Outcomes

Originally delivered @ Music City Agile in Nashville, TN, USA on 8/17/2016

Pyramids? Quadrants? Cupcakes?! There are a number models that describe approaches to test automation strategy and their possible outcomes.

In this talk, we’ll look at classical models of testing patterns and antipatterns - technical, cultural, productive and anti-productive. Adding layers of professional experience, real world example, and case study, I’ll add detail and nuance to these models.

With a new lens, focused on tests as value amplifiers, I'll share a new approach to evolving a testing strategy appropriate for your product. Attendees will learn effective measurement technique, putting code coverage fixation to rest.

7483b82f17f81773e336c5429fd6a807?s=128

David Laribee

August 17, 2016
Tweet

Transcript

  1.  @#$&!? NEW MODEL, BETTER OUTCOMES! TESTING STRATEGY:

  2. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Hi. I’m Dave. Coaching

    & Workshops Continuous Delivery, Test Driven Development, Technical Debt Management 20 Years, Former C#/Arch MVP, Former Coach @ VersionOne, ALT.NET Organizer, Playing, Full Stack Coach @laribee dave@nerdnoir.com
  3. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Model |ˈmädl| noun 1.

    a system or thing used as an example to follow or imitate.
  4. @laribee #testingmodels © 2016 Nerd/Noir, LLC. @#$&!? Debugger Driven Development

  5. None
  6. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Every time you end

    a debugging session… Monsters devour a kitten.
  7. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  8. @#$&!?

  9. PRETTY SURE GUSTA! I love my debugger. Unconsciously incompetent and

    life is good. The maintenance team will handle it. @#$&!?
  10. PRETTY SURE GUSTA! I love my debugger. Unconsciously incompetent and

    life is good. The maintenance team will handle it. FML - This is getting complicated. This app was supposed to be small. It was a one off! Did you know that 5 “Marlboro Men” died of smoking-related diseases? #notsustainable EEEEEEEEEEEEE @#$&!?
  11. @laribee #testingmodels © 2016 Nerd/Noir, LLC. The Testing Pyramid

  12. None
  13. Alistair Scott

  14. None
  15. PRETTY SURE GUSTA! Explains past mistakes. End-to-end tests are costly

    and brittle. Isolation yields more pointed feedback. Developers test too!
  16. PRETTY SURE GUSTA! Explains past mistakes. End-to-end tests are costly

    and brittle. Isolation yields more pointed feedback. Developers test too! What’s 10% mean? 90%? Count? Runtime? How big a pyramid are we talking? Uh… I’m a test automation person… what does this mean for me? Where do all of these unit tests come from? EEEEEEEEEEEEE
  17. @laribee #testingmodels © 2016 Nerd/Noir, LLC. The Testing Cupcake Anti-pattern

  18. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Alistair Scott

  19. Fabio Pereira

  20. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  21. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Org / Team Test

    Inventory
  22. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  23. None
  24. PRETTY SURE GUSTA! That explains some of the angst I

    feel. Our organization/culture has a huge influence on our testing strategy. Helps explain the “testing pyramid” model.
  25. PRETTY SURE GUSTA! That explains some of the angst I

    feel. Our organization/culture has a huge influence on our testing strategy. Helps explain the “testing pyramid” model. This describes us. THIS DESCRIBES US. Changing this threatens my role. “We have a major investment in Test Center.” Why do we keep ignoring those 500 cucumber test failures? Cupcakes are delicious. I prefer mine kitten-free. EEEEEEEEEEEEE
  26. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Agile Testing Quadrants

  27. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Marick, Gregory, Crispin

  28. Alistair Scott

  29. Alistair Scott

  30. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Gojko Adzic Safety Innovating

    w/ Data
  31. None
  32. PRETTY SURE GUSTA! Tests have an audience. Different tests serve

    different concerns. Maybe we shouldn’t invent and/or automate everything ourselves (performance testing, etc.)? Manual testing ain’t dead.
  33. PRETTY SURE GUSTA! Tests have an audience. Different tests serve

    different concerns. Maybe we shouldn’t invent and/or automate everything ourselves (performance testing, etc.)? Manual testing ain’t dead. Business folks testing?! I don’t get the “support the team” vs. “critique the product” divide. Business-technology is kind of a limiting divide, don’t you think? How do tests tell us if we’re building something people want / will use? EEEEEEEEEEEEE
  34. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Test First & TDD

  35. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  36. @laribee #testingmodels © 2016 Nerd/Noir, LLC. “DETROIT SCHOOL” TDD

  37. @laribee #testingmodels © 2016 Nerd/Noir, LLC. •Extreme Programming - Chrysler

    C3 Project •Red-Green-Refactor •Black Box: drive out implementation from public API. •Heavy emphasis on refactoring. •AKA - “Classicist” (Fowler) “DETROIT SCHOOL” TDD
  38. @laribee #testingmodels © 2016 Nerd/Noir, LLC. “LONDON SCHOOL” TDD

  39. @laribee #testingmodels © 2016 Nerd/Noir, LLC. •“Growing Object Oriented Software

    Guided By Tests” •Heavy use of Mock Objects (a stub with an assert). •White Box: discover interfaces and interactions. •Tests are a industrial by-product of design. •AKA - “Mockist” (Fowler) “LONDON SCHOOL” TDD
  40. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller

  41. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller Adapter Mock /

    Stub
  42. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller Application Service Mock

    / Stub Adapter Mock / Stub
  43. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller Application Service Mock

    / Stub Adapter Mock / Stub Business Object / Entity Actual
  44. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller Application Service Mock

    / Stub Adapter Mock / Stub Business Object / Entity Actual
  45. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller Application Service Mock

    / Stub Adapter Mock / Stub Business Object / Entity Actual Application Service Controller
  46. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller External API /

    Library Write an integrated test after Application Service Mock / Stub Adapter Mock / Stub Business Object / Entity Actual Application Service Controller
  47. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Controller External API /

    Library Write an integrated test after Application Service Mock / Stub Adapter Mock / Stub Business Object / Entity Actual Application Service Controller
  48. None
  49. PRETTY SURE GUSTA! Test-first and, to a lesser extent, test-closely

    ensure unit tests get written and not forgotten / put off. Design your code by treating everything as an API. Your tests are simply the first client! Scales with level of detail: product, iteration, code artifact. A nice tool, but you know what they say about silver bullets.
  50. PRETTY SURE GUSTA! Test-first and, to a lesser extent, test-closely

    ensure unit tests get written and not forgotten / put off. Design your code by treating everything as an API. Your tests are simply the first client! Scales with level of detail: product, iteration, code artifact. A nice tool, but you know what they say about silver bullets. Discipline is hard. I forget to refactor sometimes. Rugged and steep learning curve. Mock objects are weird. London school tests seem redundant (and weird), especially so with code first. Practicing both Detroit and London (state- vs. interaction- based verification) blurs the line of unit. EEEEEEEEEEEEE
  51. @laribee #testingmodels © 2016 Nerd/Noir, LLC. T.A.T.F.T.

  52. @laribee #testingmodels © 2016 Nerd/Noir, LLC. When do you test?

    ALL THE F-ING TIME!
  53. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  54. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  55. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  56. None
  57. PRETTY SURE GUSTA! A whole team testing mindset makes such

    a huge difference in product and code quality. “The secret of genius is to carry the spirit of the child into old age, which means never losing your enthusiasm.” - Huxley
  58. PRETTY SURE GUSTA! A whole team testing mindset makes such

    a huge difference in product and code quality. “The secret of genius is to carry the spirit of the child into old age, which means never losing your enthusiasm.” - Huxley Is execution bias creating a team fixation? What other things could we do to improve our product and/or code quality? “Enthusiasm just creates bubbles; it doesn't keep them from popping.” - Svitak EEEEEEEEEEEEE
  59. @laribee #testingmodels © 2016 Nerd/Noir, LLC.  Code Coverage (Through

    Tests)
  60. Yeah… I’m gonna go ahead and need you to get

    us to 50% coverage… That’d be great….
  61. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Tyler Vigen - Spurious

    Coorelations
  62. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  63. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  64. 

  65. PRETTY SURE GUSTA! Code coverage can be handy while unit

    testing (for the people creating the tests). A coverage tool can help fortify refactoring resolve and be a useful, if not temporary, tool on the team- wide test-driven path. 
  66. PRETTY SURE GUSTA! Code coverage can be handy while unit

    testing (for the people creating the tests). A coverage tool can help fortify refactoring resolve and be a useful, if not temporary, tool on the team- wide test-driven path. Manager: “How else do you expect me to measure quality! If I can’t measure it, I can’t manage it.” Developer to Manager: “Do we get to measure you by useless minutes spent in meetings?” Coverage tells me nothing about test suite quality. Maybe start with an engaged senior engineer? EEEEEEEEEEEEE 
  67. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Exploratory Testing

  68. @laribee #testingmodels © 2016 Nerd/Noir, LLC. “Exploratory testing is simultaneous

    learning, test design, and test execution.” James Bach
  69. @laribee #testingmodels © 2016 Nerd/Noir, LLC. James A. Whittaker

  70. @laribee #testingmodels © 2016 Nerd/Noir, LLC.

  71. None
  72. PRETTY SURE GUSTA! “People support a world they help to

    create” - Carnegie It’s an all hands on deck approach. A sound alternative to maintaining extensive DB of manual test scripts (which often go stale).
  73. PRETTY SURE GUSTA! “People support a world they help to

    create” - Carnegie It’s an all hands on deck approach. A sound alternative to maintaining extensive DB of manual test scripts (which often go stale). Developers develop. Testers test. Stay off our lawns. What about my test scripts?! As a manager of testers, engineers, or product(s) - how do I know we haven’t regressed? EEEEEEEEEEEEE
  74. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Strategy |ˈstradƧjē| noun 1.

    a high level plan to achieve one or more goals under conditions of uncertainty.
  75. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Gestalt

  76. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Gestalt |ɡƧˈSHtält,-ˈSHtôlt| noun 1.

    an organized whole that is perceived as more than the sum of its parts. https://en.wikipedia.org/wiki/Gestalt_psychology
  77. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  78. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design

  79. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working.
  80. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers.
  81. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests.
  82. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests.
  83. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests. Feature tests.
  84. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests. Feature tests. Exploratory testing.
  85. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests. Feature tests. Exploratory testing. Curation over collection.
  86. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests. Feature tests. Exploratory testing. Curation over collection. Written & understood by team!
  87. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Design Confidence

    the software is working. Find regressions before customers. Smoke tests. Automated acceptance tests. Feature tests. Exploratory testing. Curation over collection. Written & understood by team! Time boxed (< 30 minutes? < 1 Hour?) S.A.F.E. - Justin Searls
  88. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  89. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  90. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests.
  91. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST!
  92. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback!
  93. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback! State-based verification.
  94. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback! State-based verification. Black box.
  95. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback! State-based verification. Black box. Refactoring Dependent.
  96. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback! State-based verification. Black box. Refactoring Dependent. “Detroit School.” Old school.
  97. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Developer tests. Millisecond feedback… FAST! High value feedback! State-based verification. Black box. Refactoring Dependent. “Detroit School.” Old school. Small graphs of independent objects.
  98. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  99. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  100. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK.
  101. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK. Heavy use of test doubles (mocks, stubs) to design object interactions.
  102. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK. Heavy use of test doubles (mocks, stubs) to design object interactions. Helps us discover boundaries.
  103. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK. Heavy use of test doubles (mocks, stubs) to design object interactions. Helps us discover boundaries. White-box. That’s OK!
  104. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK. Heavy use of test doubles (mocks, stubs) to design object interactions. Helps us discover boundaries. White-box. That’s OK! “London School.” New-old school.
  105. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery Also unit tests. Also: FAST FEEDBACK. Heavy use of test doubles (mocks, stubs) to design object interactions. Helps us discover boundaries. White-box. That’s OK! “London School.” New-old school. Deletion is OK! No sacred cows here.
  106. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  107. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety

  108. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Product-level testing! Whole team

    testing! Testing culture. Exploratory prompts automation. Mindful of its own size. Advanced: Passionate users. Advanced: Session Testing Advanced: The Testing Shard +/- Advanced: Contract Testing Safety
  109. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  110. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Safety Specs Examples Design

    Discovery
  111. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Observing Coding Designing Testing

  112. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Gestalt Gestalt

  113. @laribee #testingmodels © 2016 Nerd/Noir, LLC. Thank you, Nashville! @laribee

    dave@nerdnoir.com Download slides - nerdnoir.com/testing-models