Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test automation fundamentals for developers

Test automation fundamentals for developers

Deck for our in-house training course on test-automation fundamentals. Covers performance testing, end-to-end testing but mainly focuses on unit testing principles & practices.

Exercises can be found on github: https://github.com/jovaneyck/TestingExercises.

Jo Van Eyck

December 01, 2016
Tweet

More Decks by Jo Van Eyck

Other Decks in Programming

Transcript

  1. ae nv/sa Interleuvenlaan 27b, B-3001 Heverlee T +32 16 39

    30 60 - F +32 16 39 30 70 www.ae.be TA Fundamentals Testing
  2. Company confidential – Do not distribute without notice ©AE 2014

     2  Why test?  Kinds of tests  Manual testing  UI/End-To-End testing  Performance testing  Unit testing  Integration testing  Test-Driven Design  Testing existing code Agenda
  3. Company confidential – Do not distribute without notice ©AE 2014

     7 Why? If you look at how most programmers spend their time, you'll find that writing code is actually a small fraction. Some time is spent figuring out what ought to be going on, some time is spent designing, but most time is spent debugging. I'm sure every reader can remember long hours of debugging, often long into the night. Every programmer can tell a story of a bug that took a whole day (or more) to find. Fixing the bug is usually pretty quick, but finding it is a nightmare. And then when you do fix a bug, there's always a chance that another one will appear and that you might not even notice it till much later. Then you spend ages finding that bug. Martin Fowler, Refactoring
  4. Company confidential – Do not distribute without notice ©AE 2014

     11 Why do we test?  Correctness now and later  Faster feedback on correctness  Design feedback  Facilitate incremental change  Stress-free development  Confidence in code  Documents the behavior of your system  Encourages clean code
  5. Company confidential – Do not distribute without notice ©AE 2014

     13 Kinds of tests  Unit test – tests a small piece of code in isolation  Integration test – tests component or project as a whole  End-to-end test – test entire application from UI to DB and back  UI test – test UI logic only  Exploratory testing – tests performed by tester  Performance testing – verify non-functional requirements are met  Security testing – pen-testing, OWASP vulnerabilities,…  …  Whatever works for your project/team
  6. Company confidential – Do not distribute without notice ©AE 2014

     14 Kinds of tests  Unit test – tests a small piece of code in isolation  Integration test – tests component or project as a whole  End-to-end test – test entire application from UI to DB and back  UI test – test UI logic only  Exploratory testing – tests performed by tester  Performance testing – verify non-functional requirements are met  Security testing – pen-testing, OWASP vulnerabilities,…  …  Whatever works for your project
  7. Company confidential – Do not distribute without notice ©AE 2014

     16 What tests to write?  Only test it once  Maximize ROI http://martinfowler.com/bliki/TestPyramid.html http://everzet.com/post/107204911916/economy-of-tests 90% 10% 10
  8. Company confidential – Do not distribute without notice ©AE 2014

     17 What tests to write?  The invisible test pyramid©
  9. Company confidential – Do not distribute without notice ©AE 2014

     18 What tests to write?  Maximize ROI depends on  Business impact when things go wrong  Reversibility of breaking changes  Lifetime of the tests  Team • Size, separate QA people,… • Experience with testing
  10. Company confidential – Do not distribute without notice ©AE 2014

     22  In an ideal world  Exploratory testing • Inventing and executing interesting test cases  Usability testing  In the real world Manual testing
  11. Company confidential – Do not distribute without notice ©AE 2014

     23  Exercise  Go to www.tradzster.nl  Execute test cases • Place multileg order • Send signal • Strategy shop • Subscribe to strategy • Fat finger protection • AdvizorConnect • Strategyshop • Dealticket Manual testing
  12. Company confidential – Do not distribute without notice ©AE 2014

     25  Tools: Selenium webdriver (web), protractor (angularJS), teststack White (.NET WPF), WindowLicker (Java swing) UI testing
  13. Company confidential – Do not distribute without notice ©AE 2014

     27  Scripted tests >>> record-replay  Isolate and minimize knowledge about UI structure  Page Object pattern UI testing
  14. Company confidential – Do not distribute without notice ©AE 2014

     28  Window Driver/Page Object pattern UI testing
  15. Company confidential – Do not distribute without notice ©AE 2014

     30  What kind of performance?  Response times • A user must see results in 0.1s • One message must be processed within 1sec  High concurrency • Requests of 100 concurrent users should have an acceptable response time < 10 ms.  High throughput • Messages up to 100mb/gb must be processed within x seconds. Performance testing
  16. Company confidential – Do not distribute without notice ©AE 2014

     31  Think about what you want to test?  Behavior under avg / max / extreme load • CPU consumption • Memory consumption • Disk space consumption (database) • Network consumption • Throughput speed • Throughput bandwidth • Database access Performance testing
  17. Company confidential – Do not distribute without notice ©AE 2014

     32  Performance starts at the design phase  Monitor, monitor, monitor  Scalable by default  But don’t over-engineer • Too many users for your system is a good problem to have • CRUD app <> high-speed stock trading Performance testing
  18. Company confidential – Do not distribute without notice ©AE 2014

     33  Predictability  Whatever the situation, the system remains responsive  Ideally every request should take more or less the same amount of resources  GC cycles (?) Performance testing
  19. Company confidential – Do not distribute without notice ©AE 2014

     34  Traceability  Create means to measure the performance  Performance measuring does not stop after go-live Performance testing
  20. Company confidential – Do not distribute without notice ©AE 2014

     35  Watch out with your environment  Make sure you can relate your tests to production  Look for shared resources • DB • Server  Load • Both on the system under test as other systems using shared resources  Test system • System running the tests can also be a limiting factor Performance testing
  21. Company confidential – Do not distribute without notice ©AE 2014

     38  There’s only a performance problem when the profiler says so  Note: profiler != user Performance testing
  22. Company confidential – Do not distribute without notice ©AE 2014

     39 Performance testingb  Exercise  Download Jmeter  Load-test the following website • http://dosme.azurewebsites.net  What’s the average response time • For a single user? • For 10 concurrent users?
  23. Company confidential – Do not distribute without notice ©AE 2014

     43 What is a unit?  A public API call  A method  An object  A set of objects closely working together  A single unit of work  End result  Method output  State changed  Callout to “external” code “A unit of work is the sum of actions that take place between the invocation of a public method in the system and a single noticeable end result by a test of that system.“ Osherove
  24. Company confidential – Do not distribute without notice ©AE 2014

     44 What is a unit test?  A test is not a unit test if  It talks to the database  It communicates across the network  It touches the file system  It can’t run at the same time as other unit tests  You have to do something special to your environment (such as editing configuration files) to run it http://www.artima.com/weblogs/viewpost.jsp?thread=126923
  25. Company confidential – Do not distribute without notice ©AE 2014

     46 Why unit test? “ To create a tiny universe where the software exists to do one thing and do it well. “ Joshua Graham
  26. Company confidential – Do not distribute without notice ©AE 2014

     47 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck
  27. Company confidential – Do not distribute without notice ©AE 2014

     48 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck
  28. Company confidential – Do not distribute without notice ©AE 2014

     49 Unit testing - how  Structure of a unit test:  Arrange the System Under Test (SUT)  Act on the SUT  Assert the result or behaviour  Clean up (if necessary)  Tests often follow the Given - When - Then pattern.  Given a certain set of preconditions  When the method under test is executed  Then a certain result is expected
  29. Company confidential – Do not distribute without notice ©AE 2014

     50 Unit testing - how  Test Setup and test fixture:  @BeforeClass  @Before  Test Execution:  @Test  Test Validation:  junit.framework.Assert  Cleanup  @After  @AfterClass
  30. Company confidential – Do not distribute without notice ©AE 2014

     51 Unit testing - Labs  LAB 1: Simple unit test  Test the Calculator class Focus on Structure of a Unit test Test Fixture Asserting
  31. Company confidential – Do not distribute without notice ©AE 2014

     52 Unit testing – Best practices  Parametrized unit test
  32. Company confidential – Do not distribute without notice ©AE 2014

     53 Unit testing - Labs  LAB 2: Parametrized unit test  Cleanup tests for the Calculator class Focus on Remove duplication
  33. Company confidential – Do not distribute without notice ©AE 2014

     54 Why trust unit tests?  Seen them fail for the right reasons  Test-First Development  Defect injection
  34. Company confidential – Do not distribute without notice ©AE 2014

     55 Why trust unit tests?  Very simple code  No conditionals, loops, branching constructs,…  Cyclomatic complexity == 1
  35. Company confidential – Do not distribute without notice ©AE 2014

     56 Styles of verification  state-based  Interaction-based
  36. Company confidential – Do not distribute without notice ©AE 2014

     58 Unit testing - Test Doubles  Goal: test one unit of code without dependencies  What to do with dependencies?  Use the real object • Downside: testing more than the SUT • Use a Test Double
  37. Company confidential – Do not distribute without notice ©AE 2014

     60 Unit testing - Test Doubles  Dummy: objects passed around but not used  Makes the compiler happy  Fake: working implementation not suitable for production (eg in memory database)  Stub: provides canned answers  Cannot fail a unit test themselves  Mock: pre-programmed with expectations  Act as assertions, can fail a test  Stub queries, expect commands
  38. Company confidential – Do not distribute without notice ©AE 2014

     61 Unit testing – State-based test https://www.industriallogic.com/blog/mock-objects-pictures/
  39. Company confidential – Do not distribute without notice ©AE 2014

     64 Unit testing - Labs  LAB 3: Unit test with stubs & mocks  Test the LogAnalyzer class in isolation • Needed a stub? • Needed a mock?
  40. Company confidential – Do not distribute without notice ©AE 2014

     65 Unit testing best practices – inversion of contol  Inversion of control / dependency injection: Instead of: creating dependencies yourself
  41. Company confidential – Do not distribute without notice ©AE 2014

     66 Unit testing – inversion of contol  Inversion of control / dependency injection: declare what you need and let the consumer/container provide you an implementation
  42. Company confidential – Do not distribute without notice ©AE 2014

     67 Unit testing - best practices  Testing time-related code
  43. Company confidential – Do not distribute without notice ©AE 2014

     68 Unit testing - best practices  Testing multi-threaded code
  44. Company confidential – Do not distribute without notice ©AE 2014

     69 Unit testing - best practices  Isolate the concept that’s difficult/slow to test  Threading  Time  Asynchronicity  Expensive resources (DB, network, IO)  Provide controllable fakes for these dependencies in your tests
  45. Company confidential – Do not distribute without notice ©AE 2014

     70 Unit testing - best practices  Exercise: test the BeerOClockChecker class
  46. Company confidential – Do not distribute without notice ©AE 2014

     71 Level of granularity  Solitary vs. Sociable unit tests  Isolated vs. Integrated tests  Mockist vs. Classicist style TDD  Formal proof vs. Scientific method  Isolated tests need seams: interfaces, dependency inversion, … • Easy with new code • Painful with existing code • Static = global = painful in test (and production) code
  47. Company confidential – Do not distribute without notice ©AE 2014

     72 Level of granularity  Solitary vs. Sociable unit tests  Not OR, but AND • Some happy path sociable test to make sure everything is wired properly • Cover all edge/exception cases with solitary tests  + Prevents combinatorial explosion of (costly) sociable tests  - 100% solitary test coverage != 100% correct code • Stubs and expectations are assumptions about the outside world, need to test/keep these in sync as well  Economics: testing pyramid
  48. Company confidential – Do not distribute without notice ©AE 2014

     73 Level of granularity  Solitary tests and C#: not the ideal match  Virtual all the things  Interface all the things  Make your “unit” larger • Tests break when (irrelevant part of unit) breaks 
  49. Company confidential – Do not distribute without notice ©AE 2014

     75 Primary goal of integration testing
  50. Company confidential – Do not distribute without notice ©AE 2014

     76 Integrated tests are a scam (?)  10 layers in your system  3 branching points in each layer  Number of possible code paths:  310 = 59,049 possible code paths  4 branching points? • 410 = 1,048,576 possible code paths  How many tests can you write in a day? http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/
  51. Company confidential – Do not distribute without notice ©AE 2014

     77 Integrated tests are a scam (?)  Alternatives?  Only test each thing once.  Prefer isolated testing over integrated testing.  Linear instead of exponential curve  Principle of mathematical induction  Given that layer 2 and down all work correctly  Prove layer 1 works = 3 branches = 3 tests  O(3 x 10 + C) = (focused) tests  Assumptions in layer 1 become tests in layer 2
  52. Company confidential – Do not distribute without notice ©AE 2014

     78 Integrated tests  Sometimes you need integrated tests  3rd party dependencies  Database  UI/browser tests  Keep them low in number and laser-focused  Is the system glued together correctly?  Use fake dependencies in rest of the tests
  53. Company confidential – Do not distribute without notice ©AE 2014

     79 Hexagonal architecture http://www.natpryce.com/articles/000772.html
  54. Company confidential – Do not distribute without notice ©AE 2014

     80 Hexagonal architecture http://www.natpryce.com/articles/000772.html
  55. Company confidential – Do not distribute without notice ©AE 2014

     81 Hexagonal architecture http://www.natpryce.com/articles/000772.html
  56. Company confidential – Do not distribute without notice ©AE 2014

     86 Some guidelines  Prefer solitary tests  Very few “happy path” sociable tests if feeling anxious  The “integrated test whisperer”  No more than 12 End-to-end tests for entire system http://blog.jayfields.com/2011/07/high-level-test-whisperer.html
  57. Company confidential – Do not distribute without notice ©AE 2014

     87 Unit testing - best practices  What’s the problem with tests?
  58. Company confidential – Do not distribute without notice ©AE 2014

     88 Unit testing - best practices  What’s the problem with tests?
  59. Company confidential – Do not distribute without notice ©AE 2014

     89 Unit testing - best practices  Treat test code like production code:  Should always work!  Should last a long time  Should be maintainable  Should be readable  Should be maintained by the entire team  Should be in source control  Should run every commit Follow same code guidelines as you would follow for production code.
  60. Company confidential – Do not distribute without notice ©AE 2014

     90 Unit testing - best practices  FIRST  Fast  Isolated  Repeatable  Timely
  61. Company confidential – Do not distribute without notice ©AE 2014

     91 Good unit tests (GUTs)  How fast? “If you have to ask how fast your test suite should be, it should be faster.” Corey Haines
  62. Company confidential – Do not distribute without notice ©AE 2014

     92 Unit testing - best practices  Should be easy to read and understand  Should require minimum navigation/screen estate  Avoid complex @Setup  Avoid deep test class hierarchies  Keep test code predictable: Assert Last, Expect Literals
  63. Company confidential – Do not distribute without notice ©AE 2014

     93 Unit testing - best practices  The happy path is not enough  Test boundary conditions  Null, empty String, …
  64. Company confidential – Do not distribute without notice ©AE 2014

     94 Unit testing - best practices  If your class is difficult to test  it is probably too complex  it is a candidate for refactoring  a class is easier to test if it honors the SOLID principles: • Single responsibility • Open for extension / closed for modification • Liskov substitution • Interface segregation • Dependency inversion • Consider writing your tests first
  65. Company confidential – Do not distribute without notice ©AE 2014

     95 Unit testing - best practices  Design for testability  No statics  Contract-based development  Role-based interfaces  Single Responsibility  Dependency Inversion Principle  Dumb constructors  …
  66. Company confidential – Do not distribute without notice ©AE 2014

     96 Unit testing - best practices  Always a trade-off  Test-induced design damage  But testability should be VERY high on your priority list http://david.heinemeierhansson.com/2014/test-induced-design-damage.html http://martinfowler.com/articles/is-tdd-dead/
  67. Company confidential – Do not distribute without notice ©AE 2014

     98 Good unit tests  Test BEHAVIOUR, not IMPLEMENTATION  Don’t test privates  Test through “public API”  Test a “meaningful behaviour” instead of a “method on a class”  Act as executable specification of your system
  68. Company confidential – Do not distribute without notice ©AE 2014

     100 Good unit tests http://www.slideshare.net/Kevlin/programming-with-guts-41938796
  69. Company confidential – Do not distribute without notice ©AE 2014

     101 Good unit tests  Single “assertion” per test  One reason to fail  Maximize feedback when a test fails • Which cases work, which cases don’t? • Narrow problem down  OK to assert multiple facets of a same concept (?)  Assert Last
  70. Company confidential – Do not distribute without notice ©AE 2014

     103 Good unit tests  DRY  Don’t Repeat Yourself  Mostly knowledge, not just mechanical code duplication  SPOT: Single Point Of Truth  DAMP  Descriptive And Meaningful Phrases • Create a testing “DSL” that describes your SUT  Descriptive And Maintainable Procedures • Avoid brittle tests http://verraes.net/2014/08/dry-is-about-knowledge/ http://blog.jayfields.com/2006/05/dry-code-damp-dsls.html
  71. Company confidential – Do not distribute without notice ©AE 2014

     104 Good unit tests  Prefer DAMP > DRY with a blowtorch  Not mutually exclusive  A constant trade-off
  72. Company confidential – Do not distribute without notice ©AE 2014

     105 DAMP unit tests  Test Data Builders & SUT Factory  Decouple test from implementation  Decrease churn • “new” in test code might be a smell • “new” in production code as well  http://www.natpryce.com/articles/000714.html
  73. Company confidential – Do not distribute without notice ©AE 2014

     106 DAMP unit tests  Handrolled Test Data Builders
  74. Company confidential – Do not distribute without notice ©AE 2014

     107 DAMP unit tests  Test Data Builders with AutoFixture
  75. Company confidential – Do not distribute without notice ©AE 2014

     108 DAMP unit tests  Don’t overspecify your assertions/expectations  Argument matchers
  76. Company confidential – Do not distribute without notice ©AE 2014

     111 Unit testing – test-first-desgin
  77. Company confidential – Do not distribute without notice ©AE 2014

     112 Unit testing – test-driven-design It’s a design technique, not a testing technique
  78. Company confidential – Do not distribute without notice ©AE 2014

     113 TDD – demo  Roman numerals kata  http://codingdojo.org/cgi-bin/index.pl?KataRomanNumerals
  79. Company confidential – Do not distribute without notice ©AE 2014

     114 TDD – advanced topics  Triangulation  As tests get more specific, code gets more generic
  80. Company confidential – Do not distribute without notice ©AE 2014

     115 Outside-in vs Inside-out development  Outside-in  Dream up “ideal API”  Fake implementation of collaborators  Finished implementation? • Test-drive implementation of collaborator s  Stub vs. Mock  Inside-out  Create “building blocks”  Compose application by combining real building blocks  Cf. Solitary vs. Sociable unit tests
  81. Company confidential – Do not distribute without notice ©AE 2014

     116 TDD – advanced topics  Outside-in TDD/double loop TDD  Start with failing acceptance test  Unit test top-level object  When you discover a new collaborator: interface + mock  Rinse and repeat one level down http://coding-is-like-cooking.info/2013/04/outside-in-development-with-double-loop-tdd/
  82. Company confidential – Do not distribute without notice ©AE 2014

     117 Unit testing - labs  Exercise: Test Driven Design  Test-drive an implementation of FizzBuzz • Focus on red-green-refactor cycle • Triangulation  http://wiki.c2.com/?FizzBuzzTest
  83. Company confidential – Do not distribute without notice ©AE 2014

     118 Unit testing - BDD  Behaviour-Driven Design
  84. Company confidential – Do not distribute without notice ©AE 2014

     122 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java
  85. Company confidential – Do not distribute without notice ©AE 2014

     123 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java
  86. Company confidential – Do not distribute without notice ©AE 2014

     125  How much of the code is under test?  Code coverage  If a line is executed by a unit test, it is covered.  <> asserted on!  Branch coverage  Counts the number of branches/codepaths that were covered in a unit test. Metrics
  87. Company confidential – Do not distribute without notice ©AE 2014

     126 A project with good coverage is not necessarily “well tested”  Should never be a goal on itself  But can be a serious smell if low Metrics on unit testing
  88. Company confidential – Do not distribute without notice ©AE 2014

     127 Code coverage demo Metrics on unit testing
  89. Company confidential – Do not distribute without notice ©AE 2014

     130 Legacy code  Techniques for getting existing code under test
  90. Company confidential – Do not distribute without notice ©AE 2014

     131 Legacy code  The main problem  Hidden and difficult to test dependencies make code impossible to test
  91. Company confidential – Do not distribute without notice ©AE 2014

     134 Legacy code  Characterization tests  Write tests for existing code before changing it to lock in existing functionality  Golden master  ApprovalTests library
  92. Company confidential – Do not distribute without notice ©AE 2014

     135 Legacy code  Extract and override  Introduce seams with minimal impact on production code
  93. Company confidential – Do not distribute without notice ©AE 2014

     136 Legacy code  Clean up that sh*t  Replace inheritance with delegation  Separate Lifecycle Management from behaviour  Dependency Inversion Principle
  94. Company confidential – Do not distribute without notice ©AE 2014

     137 Legacy code – where to start? http://www.slideshare.net/MozaicWorks/sandro-mancuso-testing-and-refactoring-legacy-code
  95. Company confidential – Do not distribute without notice ©AE 2014

     138 Legacy code  Use code coverage as a compass
  96. Company confidential – Do not distribute without notice ©AE 2014

     139 Legacy code  Exercise:  Write isolated tests for the TripService class • Code coverage as a compass • Extract and override • Dependency Inversion • Refactoring  Exercise  Get the trivia codebase under test • Golden master • Hint: ApprovalTests library! • https://github.com/jbrains/trivia
  97. Company confidential – Do not distribute without notice ©AE 2014

     140 Further reading  Test-Driven Development, by example, Beck  The Art Of Unit Testing, Osherove  Working effectively with Legacy Code, Feathers  Growing Object-Oriented Software, guided by tests, Pryce & Freeman  Xunit test patterns, refactoring test code, Meszaros  Working effectively with unit tests, Fields