Test automation fundamentals for developers

ae nv/sa Interleuvenlaan 27b, B-3001 Heverlee T +32 16 39
30 60 - F +32 16 39 30 70 www.ae.be TA Fundamentals Testing

Company confidential – Do not distribute without notice ©AE 2014
 2  Why test?  Kinds of tests  Manual testing  UI/End-To-End testing  Performance testing  Unit testing  Integration testing  Test-Driven Design  Testing existing code Agenda

 3 Why?

 4 Why?

 5 Why?

 6 Why?

 7 Why? If you look at how most programmers spend their time, you'll find that writing code is actually a small fraction. Some time is spent figuring out what ought to be going on, some time is spent designing, but most time is spent debugging. I'm sure every reader can remember long hours of debugging, often long into the night. Every programmer can tell a story of a bug that took a whole day (or more) to find. Fixing the bug is usually pretty quick, but finding it is a nightmare. And then when you do fix a bug, there's always a chance that another one will appear and that you might not even notice it till much later. Then you spend ages finding that bug. Martin Fowler, Refactoring

 8 Why?

 9 Why?

 10 Why?

 11 Why do we test?  Correctness now and later  Faster feedback on correctness  Design feedback  Facilitate incremental change  Stress-free development  Confidence in code  Documents the behavior of your system  Encourages clean code

 12 Kinds of tests

 13 Kinds of tests  Unit test – tests a small piece of code in isolation  Integration test – tests component or project as a whole  End-to-end test – test entire application from UI to DB and back  UI test – test UI logic only  Exploratory testing – tests performed by tester  Performance testing – verify non-functional requirements are met  Security testing – pen-testing, OWASP vulnerabilities,…  …  Whatever works for your project/team

 14 Kinds of tests  Unit test – tests a small piece of code in isolation  Integration test – tests component or project as a whole  End-to-end test – test entire application from UI to DB and back  UI test – test UI logic only  Exploratory testing – tests performed by tester  Performance testing – verify non-functional requirements are met  Security testing – pen-testing, OWASP vulnerabilities,…  …  Whatever works for your project

 15 What tests to write?

 16 What tests to write?  Only test it once  Maximize ROI http://martinfowler.com/bliki/TestPyramid.html http://everzet.com/post/107204911916/economy-of-tests 90% 10% 10

 17 What tests to write?  The invisible test pyramid©

 18 What tests to write?  Maximize ROI depends on  Business impact when things go wrong  Reversibility of breaking changes  Lifetime of the tests  Team • Size, separate QA people,… • Experience with testing

 19 Testing in the ALM lifecycle

 20 Testing in the ALM lifecycle

 21 MANUAL TESTING

 22  In an ideal world  Exploratory testing • Inventing and executing interesting test cases  Usability testing  In the real world Manual testing

 23  Exercise  Go to www.tradzster.nl  Execute test cases • Place multileg order • Send signal • Strategy shop • Subscribe to strategy • Fat finger protection • AdvizorConnect • Strategyshop • Dealticket Manual testing

 24 UI/END-TO-END TESTING

 25  Tools: Selenium webdriver (web), protractor (angularJS), teststack White (.NET WPF), WindowLicker (Java swing) UI testing

 26 UI testing - Selenium

 27  Scripted tests >>> record-replay  Isolate and minimize knowledge about UI structure  Page Object pattern UI testing

 28  Window Driver/Page Object pattern UI testing

 29 PERFORMANCE TESTING

 30  What kind of performance?  Response times • A user must see results in 0.1s • One message must be processed within 1sec  High concurrency • Requests of 100 concurrent users should have an acceptable response time < 10 ms.  High throughput • Messages up to 100mb/gb must be processed within x seconds. Performance testing

 31  Think about what you want to test?  Behavior under avg / max / extreme load • CPU consumption • Memory consumption • Disk space consumption (database) • Network consumption • Throughput speed • Throughput bandwidth • Database access Performance testing

 32  Performance starts at the design phase  Monitor, monitor, monitor  Scalable by default  But don’t over-engineer • Too many users for your system is a good problem to have • CRUD app <> high-speed stock trading Performance testing

 33  Predictability  Whatever the situation, the system remains responsive  Ideally every request should take more or less the same amount of resources  GC cycles (?) Performance testing

 34  Traceability  Create means to measure the performance  Performance measuring does not stop after go-live Performance testing

 35  Watch out with your environment  Make sure you can relate your tests to production  Look for shared resources • DB • Server  Load • Both on the system under test as other systems using shared resources  Test system • System running the tests can also be a limiting factor Performance testing

 36 Performance testing

 37 Performance testing

 38  There’s only a performance problem when the profiler says so  Note: profiler != user Performance testing

 39 Performance testingb  Exercise  Download Jmeter  Load-test the following website • http://dosme.azurewebsites.net  What’s the average response time • For a single user? • For 10 concurrent users?

 40 Performance testing  Exercise

 41 UNIT TESTING

 42 Unit testing

 43 What is a unit?  A public API call  A method  An object  A set of objects closely working together  A single unit of work  End result  Method output  State changed  Callout to “external” code “A unit of work is the sum of actions that take place between the invocation of a public method in the system and a single noticeable end result by a test of that system.“ Osherove

 44 What is a unit test?  A test is not a unit test if  It talks to the database  It communicates across the network  It touches the file system  It can’t run at the same time as other unit tests  You have to do something special to your environment (such as editing configuration files) to run it http://www.artima.com/weblogs/viewpost.jsp?thread=126923

 45 Who writes unit tests?

 46 Why unit test? “ To create a tiny universe where the software exists to do one thing and do it well. “ Joshua Graham

 47 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck

 48 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck

 49 Unit testing - how  Structure of a unit test:  Arrange the System Under Test (SUT)  Act on the SUT  Assert the result or behaviour  Clean up (if necessary)  Tests often follow the Given - When - Then pattern.  Given a certain set of preconditions  When the method under test is executed  Then a certain result is expected

 50 Unit testing - how  Test Setup and test fixture:  @BeforeClass  @Before  Test Execution:  @Test  Test Validation:  junit.framework.Assert  Cleanup  @After  @AfterClass

 51 Unit testing - Labs  LAB 1: Simple unit test  Test the Calculator class Focus on Structure of a Unit test Test Fixture Asserting

 52 Unit testing – Best practices  Parametrized unit test

 53 Unit testing - Labs  LAB 2: Parametrized unit test  Cleanup tests for the Calculator class Focus on Remove duplication

 54 Why trust unit tests?  Seen them fail for the right reasons  Test-First Development  Defect injection

 55 Why trust unit tests?  Very simple code  No conditionals, loops, branching constructs,…  Cyclomatic complexity == 1

 56 Styles of verification  state-based  Interaction-based

 57 Mocking and stubbing UNIT TESTING

 58 Unit testing - Test Doubles  Goal: test one unit of code without dependencies  What to do with dependencies?  Use the real object • Downside: testing more than the SUT • Use a Test Double

 59 Unit testing - Test Doubles

 60 Unit testing - Test Doubles  Dummy: objects passed around but not used  Makes the compiler happy  Fake: working implementation not suitable for production (eg in memory database)  Stub: provides canned answers  Cannot fail a unit test themselves  Mock: pre-programmed with expectations  Act as assertions, can fail a test  Stub queries, expect commands

 61 Unit testing – State-based test https://www.industriallogic.com/blog/mock-objects-pictures/

 62 Unit testing – State-based test

 63 Unit testing – State-based test

 64 Unit testing - Labs  LAB 3: Unit test with stubs & mocks  Test the LogAnalyzer class in isolation • Needed a stub? • Needed a mock?

 65 Unit testing best practices – inversion of contol  Inversion of control / dependency injection: Instead of: creating dependencies yourself

 66 Unit testing – inversion of contol  Inversion of control / dependency injection: declare what you need and let the consumer/container provide you an implementation

 67 Unit testing - best practices  Testing time-related code

 68 Unit testing - best practices  Testing multi-threaded code

 69 Unit testing - best practices  Isolate the concept that’s difficult/slow to test  Threading  Time  Asynchronicity  Expensive resources (DB, network, IO)  Provide controllable fakes for these dependencies in your tests

 70 Unit testing - best practices  Exercise: test the BeerOClockChecker class

 71 Level of granularity  Solitary vs. Sociable unit tests  Isolated vs. Integrated tests  Mockist vs. Classicist style TDD  Formal proof vs. Scientific method  Isolated tests need seams: interfaces, dependency inversion, … • Easy with new code • Painful with existing code • Static = global = painful in test (and production) code

 72 Level of granularity  Solitary vs. Sociable unit tests  Not OR, but AND • Some happy path sociable test to make sure everything is wired properly • Cover all edge/exception cases with solitary tests  + Prevents combinatorial explosion of (costly) sociable tests  - 100% solitary test coverage != 100% correct code • Stubs and expectations are assumptions about the outside world, need to test/keep these in sync as well  Economics: testing pyramid

 73 Level of granularity  Solitary tests and C#: not the ideal match  Virtual all the things  Interface all the things  Make your “unit” larger • Tests break when (irrelevant part of unit) breaks 

 74 INTEGRATION TESTING

 75 Primary goal of integration testing

 76 Integrated tests are a scam (?)  10 layers in your system  3 branching points in each layer  Number of possible code paths:  310 = 59,049 possible code paths  4 branching points? • 410 = 1,048,576 possible code paths  How many tests can you write in a day? http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/

 77 Integrated tests are a scam (?)  Alternatives?  Only test each thing once.  Prefer isolated testing over integrated testing.  Linear instead of exponential curve  Principle of mathematical induction  Given that layer 2 and down all work correctly  Prove layer 1 works = 3 branches = 3 tests  O(3 x 10 + C) = (focused) tests  Assumptions in layer 1 become tests in layer 2

 78 Integrated tests  Sometimes you need integrated tests  3rd party dependencies  Database  UI/browser tests  Keep them low in number and laser-focused  Is the system glued together correctly?  Use fake dependencies in rest of the tests

 79 Hexagonal architecture http://www.natpryce.com/articles/000772.html

 82 Sidenote: DB tests

 84 Sidenote: DB tests – also FIRST!

 86 Some guidelines  Prefer solitary tests  Very few “happy path” sociable tests if feeling anxious  The “integrated test whisperer”  No more than 12 End-to-end tests for entire system http://blog.jayfields.com/2011/07/high-level-test-whisperer.html

 87 Unit testing - best practices  What’s the problem with tests?

 88 Unit testing - best practices  What’s the problem with tests?

 89 Unit testing - best practices  Treat test code like production code:  Should always work!  Should last a long time  Should be maintainable  Should be readable  Should be maintained by the entire team  Should be in source control  Should run every commit Follow same code guidelines as you would follow for production code.

 90 Unit testing - best practices  FIRST  Fast  Isolated  Repeatable  Timely

 91 Good unit tests (GUTs)  How fast? “If you have to ask how fast your test suite should be, it should be faster.” Corey Haines

 92 Unit testing - best practices  Should be easy to read and understand  Should require minimum navigation/screen estate  Avoid complex @Setup  Avoid deep test class hierarchies  Keep test code predictable: Assert Last, Expect Literals

 93 Unit testing - best practices  The happy path is not enough  Test boundary conditions  Null, empty String, …

 94 Unit testing - best practices  If your class is difficult to test  it is probably too complex  it is a candidate for refactoring  a class is easier to test if it honors the SOLID principles: • Single responsibility • Open for extension / closed for modification • Liskov substitution • Interface segregation • Dependency inversion • Consider writing your tests first

 95 Unit testing - best practices  Design for testability  No statics  Contract-based development  Role-based interfaces  Single Responsibility  Dependency Inversion Principle  Dumb constructors  …

 96 Unit testing - best practices  Always a trade-off  Test-induced design damage  But testability should be VERY high on your priority list http://david.heinemeierhansson.com/2014/test-induced-design-damage.html http://martinfowler.com/articles/is-tdd-dead/

 97 Testing internals

 98 Good unit tests  Test BEHAVIOUR, not IMPLEMENTATION  Don’t test privates  Test through “public API”  Test a “meaningful behaviour” instead of a “method on a class”  Act as executable specification of your system

 99 Good unit tests

 100 Good unit tests http://www.slideshare.net/Kevlin/programming-with-guts-41938796

 101 Good unit tests  Single “assertion” per test  One reason to fail  Maximize feedback when a test fails • Which cases work, which cases don’t? • Narrow problem down  OK to assert multiple facets of a same concept (?)  Assert Last

 102 Good unit tests  Assert Last

 103 Good unit tests  DRY  Don’t Repeat Yourself  Mostly knowledge, not just mechanical code duplication  SPOT: Single Point Of Truth  DAMP  Descriptive And Meaningful Phrases • Create a testing “DSL” that describes your SUT  Descriptive And Maintainable Procedures • Avoid brittle tests http://verraes.net/2014/08/dry-is-about-knowledge/ http://blog.jayfields.com/2006/05/dry-code-damp-dsls.html

 104 Good unit tests  Prefer DAMP > DRY with a blowtorch  Not mutually exclusive  A constant trade-off

 105 DAMP unit tests  Test Data Builders & SUT Factory  Decouple test from implementation  Decrease churn • “new” in test code might be a smell • “new” in production code as well  http://www.natpryce.com/articles/000714.html

 106 DAMP unit tests  Handrolled Test Data Builders

 107 DAMP unit tests  Test Data Builders with AutoFixture

 108 DAMP unit tests  Don’t overspecify your assertions/expectations  Argument matchers

 109 Test driven design UNIT TESTING

 110 Why test-first?

 111 Unit testing – test-first-desgin

 112 Unit testing – test-driven-design It’s a design technique, not a testing technique

 113 TDD – demo  Roman numerals kata  http://codingdojo.org/cgi-bin/index.pl?KataRomanNumerals

 114 TDD – advanced topics  Triangulation  As tests get more specific, code gets more generic

 115 Outside-in vs Inside-out development  Outside-in  Dream up “ideal API”  Fake implementation of collaborators  Finished implementation? • Test-drive implementation of collaborator s  Stub vs. Mock  Inside-out  Create “building blocks”  Compose application by combining real building blocks  Cf. Solitary vs. Sociable unit tests

 116 TDD – advanced topics  Outside-in TDD/double loop TDD  Start with failing acceptance test  Unit test top-level object  When you discover a new collaborator: interface + mock  Rinse and repeat one level down http://coding-is-like-cooking.info/2013/04/outside-in-development-with-double-loop-tdd/

 117 Unit testing - labs  Exercise: Test Driven Design  Test-drive an implementation of FizzBuzz • Focus on red-green-refactor cycle • Triangulation  http://wiki.c2.com/?FizzBuzzTest

 118 Unit testing - BDD  Behaviour-Driven Design

 119 Unit testing - tools

 122 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java

 123 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java

 124 METRICS

 125  How much of the code is under test?  Code coverage  If a line is executed by a unit test, it is covered.  <> asserted on!  Branch coverage  Counts the number of branches/codepaths that were covered in a unit test. Metrics

 126 A project with good coverage is not necessarily “well tested”  Should never be a goal on itself  But can be a serious smell if low Metrics on unit testing

 127 Code coverage demo Metrics on unit testing

 128 When to invest in tests?

 129 WORKING WITH UNTESTED CODE

 130 Legacy code  Techniques for getting existing code under test

 131 Legacy code  The main problem  Hidden and difficult to test dependencies make code impossible to test

 132 Legacy code

 133 Legacy code  Solution: seams

 134 Legacy code  Characterization tests  Write tests for existing code before changing it to lock in existing functionality  Golden master  ApprovalTests library

 135 Legacy code  Extract and override  Introduce seams with minimal impact on production code

 136 Legacy code  Clean up that sh*t  Replace inheritance with delegation  Separate Lifecycle Management from behaviour  Dependency Inversion Principle

 137 Legacy code – where to start? http://www.slideshare.net/MozaicWorks/sandro-mancuso-testing-and-refactoring-legacy-code

 138 Legacy code  Use code coverage as a compass

 139 Legacy code  Exercise:  Write isolated tests for the TripService class • Code coverage as a compass • Extract and override • Dependency Inversion • Refactoring  Exercise  Get the trivia codebase under test • Golden master • Hint: ApprovalTests library! • https://github.com/jbrains/trivia

 140 Further reading  Test-Driven Development, by example, Beck  The Art Of Unit Testing, Osherove  Working effectively with Legacy Code, Feathers  Growing Object-Oriented Software, guided by tests, Pryce & Freeman  Xunit test patterns, refactoring test code, Meszaros  Working effectively with unit tests, Fields

@ae_nv linkedin.com/company/ae-nv-sa

Test automation fundamentals for developers

Test automation fundamentals for developers

More Decks by Jo Van Eyck

Other Decks in Programming

Featured

Transcript