Share
Embed
Test automation fundamentals for developers
Deck for our in-house training course on test-automation fundamentals. Covers performance testing, end-to-end testing but mainly focuses on unit testing principles & practices.
Exercises can be found on github: https://github.com/jovaneyck/TestingExercises.
Jo Van Eyck
December 01, 2016
Transcript
141 slides
-
ae nv/sa Interleuvenlaan 27b, B-3001 Heverlee T +32 16 39
30 60 - F +32 16 39 30 70 www.ae.be TA Fundamentals Testing -
Company confidential – Do not distribute without notice ©AE 2014
2 Why test? Kinds of tests Manual testing UI/End-To-End testing Performance testing Unit testing Integration testing Test-Driven Design Testing existing code Agenda -
Company confidential – Do not distribute without notice ©AE 2014
7 Why? If you look at how most programmers spend their time, you'll find that writing code is actually a small fraction. Some time is spent figuring out what ought to be going on, some time is spent designing, but most time is spent debugging. I'm sure every reader can remember long hours of debugging, often long into the night. Every programmer can tell a story of a bug that took a whole day (or more) to find. Fixing the bug is usually pretty quick, but finding it is a nightmare. And then when you do fix a bug, there's always a chance that another one will appear and that you might not even notice it till much later. Then you spend ages finding that bug. Martin Fowler, Refactoring -
Company confidential – Do not distribute without notice ©AE 2014
11 Why do we test? Correctness now and later Faster feedback on correctness Design feedback Facilitate incremental change Stress-free development Confidence in code Documents the behavior of your system Encourages clean code -
Company confidential – Do not distribute without notice ©AE 2014
12 Kinds of tests -
Company confidential – Do not distribute without notice ©AE 2014
13 Kinds of tests Unit test – tests a small piece of code in isolation Integration test – tests component or project as a whole End-to-end test – test entire application from UI to DB and back UI test – test UI logic only Exploratory testing – tests performed by tester Performance testing – verify non-functional requirements are met Security testing – pen-testing, OWASP vulnerabilities,… … Whatever works for your project/team -
Company confidential – Do not distribute without notice ©AE 2014
14 Kinds of tests Unit test – tests a small piece of code in isolation Integration test – tests component or project as a whole End-to-end test – test entire application from UI to DB and back UI test – test UI logic only Exploratory testing – tests performed by tester Performance testing – verify non-functional requirements are met Security testing – pen-testing, OWASP vulnerabilities,… … Whatever works for your project -
Company confidential – Do not distribute without notice ©AE 2014
15 What tests to write? -
Company confidential – Do not distribute without notice ©AE 2014
16 What tests to write? Only test it once Maximize ROI http://martinfowler.com/bliki/TestPyramid.html http://everzet.com/post/107204911916/economy-of-tests 90% 10% 10 -
Company confidential – Do not distribute without notice ©AE 2014
17 What tests to write? The invisible test pyramid© -
Company confidential – Do not distribute without notice ©AE 2014
18 What tests to write? Maximize ROI depends on Business impact when things go wrong Reversibility of breaking changes Lifetime of the tests Team • Size, separate QA people,… • Experience with testing -
Company confidential – Do not distribute without notice ©AE 2014
19 Testing in the ALM lifecycle -
Company confidential – Do not distribute without notice ©AE 2014
20 Testing in the ALM lifecycle -
Company confidential – Do not distribute without notice ©AE 2014
21 MANUAL TESTING -
Company confidential – Do not distribute without notice ©AE 2014
22 In an ideal world Exploratory testing • Inventing and executing interesting test cases Usability testing In the real world Manual testing -
Company confidential – Do not distribute without notice ©AE 2014
23 Exercise Go to www.tradzster.nl Execute test cases • Place multileg order • Send signal • Strategy shop • Subscribe to strategy • Fat finger protection • AdvizorConnect • Strategyshop • Dealticket Manual testing -
Company confidential – Do not distribute without notice ©AE 2014
24 UI/END-TO-END TESTING -
Company confidential – Do not distribute without notice ©AE 2014
25 Tools: Selenium webdriver (web), protractor (angularJS), teststack White (.NET WPF), WindowLicker (Java swing) UI testing -
Company confidential – Do not distribute without notice ©AE 2014
26 UI testing - Selenium -
Company confidential – Do not distribute without notice ©AE 2014
27 Scripted tests >>> record-replay Isolate and minimize knowledge about UI structure Page Object pattern UI testing -
Company confidential – Do not distribute without notice ©AE 2014
28 Window Driver/Page Object pattern UI testing -
Company confidential – Do not distribute without notice ©AE 2014
29 PERFORMANCE TESTING -
Company confidential – Do not distribute without notice ©AE 2014
30 What kind of performance? Response times • A user must see results in 0.1s • One message must be processed within 1sec High concurrency • Requests of 100 concurrent users should have an acceptable response time < 10 ms. High throughput • Messages up to 100mb/gb must be processed within x seconds. Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
31 Think about what you want to test? Behavior under avg / max / extreme load • CPU consumption • Memory consumption • Disk space consumption (database) • Network consumption • Throughput speed • Throughput bandwidth • Database access Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
32 Performance starts at the design phase Monitor, monitor, monitor Scalable by default But don’t over-engineer • Too many users for your system is a good problem to have • CRUD app <> high-speed stock trading Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
33 Predictability Whatever the situation, the system remains responsive Ideally every request should take more or less the same amount of resources GC cycles (?) Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
34 Traceability Create means to measure the performance Performance measuring does not stop after go-live Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
35 Watch out with your environment Make sure you can relate your tests to production Look for shared resources • DB • Server Load • Both on the system under test as other systems using shared resources Test system • System running the tests can also be a limiting factor Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
36 Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
37 Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
38 There’s only a performance problem when the profiler says so Note: profiler != user Performance testing -
Company confidential – Do not distribute without notice ©AE 2014
39 Performance testingb Exercise Download Jmeter Load-test the following website • http://dosme.azurewebsites.net What’s the average response time • For a single user? • For 10 concurrent users? -
Company confidential – Do not distribute without notice ©AE 2014
40 Performance testing Exercise -
Company confidential – Do not distribute without notice ©AE 2014
41 UNIT TESTING -
Company confidential – Do not distribute without notice ©AE 2014
42 Unit testing -
Company confidential – Do not distribute without notice ©AE 2014
43 What is a unit? A public API call A method An object A set of objects closely working together A single unit of work End result Method output State changed Callout to “external” code “A unit of work is the sum of actions that take place between the invocation of a public method in the system and a single noticeable end result by a test of that system.“ Osherove -
Company confidential – Do not distribute without notice ©AE 2014
44 What is a unit test? A test is not a unit test if It talks to the database It communicates across the network It touches the file system It can’t run at the same time as other unit tests You have to do something special to your environment (such as editing configuration files) to run it http://www.artima.com/weblogs/viewpost.jsp?thread=126923 -
Company confidential – Do not distribute without notice ©AE 2014
45 Who writes unit tests? -
Company confidential – Do not distribute without notice ©AE 2014
46 Why unit test? “ To create a tiny universe where the software exists to do one thing and do it well. “ Joshua Graham -
Company confidential – Do not distribute without notice ©AE 2014
47 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck -
Company confidential – Do not distribute without notice ©AE 2014
48 Why unit test? “Test only if you would want it to work and keep on working in the future.” Kent Beck & Jo Van Eyck -
Company confidential – Do not distribute without notice ©AE 2014
49 Unit testing - how Structure of a unit test: Arrange the System Under Test (SUT) Act on the SUT Assert the result or behaviour Clean up (if necessary) Tests often follow the Given - When - Then pattern. Given a certain set of preconditions When the method under test is executed Then a certain result is expected -
Company confidential – Do not distribute without notice ©AE 2014
50 Unit testing - how Test Setup and test fixture: @BeforeClass @Before Test Execution: @Test Test Validation: junit.framework.Assert Cleanup @After @AfterClass -
Company confidential – Do not distribute without notice ©AE 2014
51 Unit testing - Labs LAB 1: Simple unit test Test the Calculator class Focus on Structure of a Unit test Test Fixture Asserting -
Company confidential – Do not distribute without notice ©AE 2014
52 Unit testing – Best practices Parametrized unit test -
Company confidential – Do not distribute without notice ©AE 2014
53 Unit testing - Labs LAB 2: Parametrized unit test Cleanup tests for the Calculator class Focus on Remove duplication -
Company confidential – Do not distribute without notice ©AE 2014
54 Why trust unit tests? Seen them fail for the right reasons Test-First Development Defect injection -
Company confidential – Do not distribute without notice ©AE 2014
55 Why trust unit tests? Very simple code No conditionals, loops, branching constructs,… Cyclomatic complexity == 1 -
Company confidential – Do not distribute without notice ©AE 2014
56 Styles of verification state-based Interaction-based -
Company confidential – Do not distribute without notice ©AE 2014
57 Mocking and stubbing UNIT TESTING -
Company confidential – Do not distribute without notice ©AE 2014
58 Unit testing - Test Doubles Goal: test one unit of code without dependencies What to do with dependencies? Use the real object • Downside: testing more than the SUT • Use a Test Double -
Company confidential – Do not distribute without notice ©AE 2014
59 Unit testing - Test Doubles -
Company confidential – Do not distribute without notice ©AE 2014
60 Unit testing - Test Doubles Dummy: objects passed around but not used Makes the compiler happy Fake: working implementation not suitable for production (eg in memory database) Stub: provides canned answers Cannot fail a unit test themselves Mock: pre-programmed with expectations Act as assertions, can fail a test Stub queries, expect commands -
Company confidential – Do not distribute without notice ©AE 2014
61 Unit testing – State-based test https://www.industriallogic.com/blog/mock-objects-pictures/ -
Company confidential – Do not distribute without notice ©AE 2014
62 Unit testing – State-based test -
Company confidential – Do not distribute without notice ©AE 2014
63 Unit testing – State-based test -
Company confidential – Do not distribute without notice ©AE 2014
64 Unit testing - Labs LAB 3: Unit test with stubs & mocks Test the LogAnalyzer class in isolation • Needed a stub? • Needed a mock? -
Company confidential – Do not distribute without notice ©AE 2014
65 Unit testing best practices – inversion of contol Inversion of control / dependency injection: Instead of: creating dependencies yourself -
Company confidential – Do not distribute without notice ©AE 2014
66 Unit testing – inversion of contol Inversion of control / dependency injection: declare what you need and let the consumer/container provide you an implementation -
Company confidential – Do not distribute without notice ©AE 2014
67 Unit testing - best practices Testing time-related code -
Company confidential – Do not distribute without notice ©AE 2014
68 Unit testing - best practices Testing multi-threaded code -
Company confidential – Do not distribute without notice ©AE 2014
69 Unit testing - best practices Isolate the concept that’s difficult/slow to test Threading Time Asynchronicity Expensive resources (DB, network, IO) Provide controllable fakes for these dependencies in your tests -
Company confidential – Do not distribute without notice ©AE 2014
70 Unit testing - best practices Exercise: test the BeerOClockChecker class -
Company confidential – Do not distribute without notice ©AE 2014
71 Level of granularity Solitary vs. Sociable unit tests Isolated vs. Integrated tests Mockist vs. Classicist style TDD Formal proof vs. Scientific method Isolated tests need seams: interfaces, dependency inversion, … • Easy with new code • Painful with existing code • Static = global = painful in test (and production) code -
Company confidential – Do not distribute without notice ©AE 2014
72 Level of granularity Solitary vs. Sociable unit tests Not OR, but AND • Some happy path sociable test to make sure everything is wired properly • Cover all edge/exception cases with solitary tests + Prevents combinatorial explosion of (costly) sociable tests - 100% solitary test coverage != 100% correct code • Stubs and expectations are assumptions about the outside world, need to test/keep these in sync as well Economics: testing pyramid -
Company confidential – Do not distribute without notice ©AE 2014
73 Level of granularity Solitary tests and C#: not the ideal match Virtual all the things Interface all the things Make your “unit” larger • Tests break when (irrelevant part of unit) breaks -
Company confidential – Do not distribute without notice ©AE 2014
74 INTEGRATION TESTING -
Company confidential – Do not distribute without notice ©AE 2014
75 Primary goal of integration testing -
Company confidential – Do not distribute without notice ©AE 2014
76 Integrated tests are a scam (?) 10 layers in your system 3 branching points in each layer Number of possible code paths: 310 = 59,049 possible code paths 4 branching points? • 410 = 1,048,576 possible code paths How many tests can you write in a day? http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/ -
Company confidential – Do not distribute without notice ©AE 2014
77 Integrated tests are a scam (?) Alternatives? Only test each thing once. Prefer isolated testing over integrated testing. Linear instead of exponential curve Principle of mathematical induction Given that layer 2 and down all work correctly Prove layer 1 works = 3 branches = 3 tests O(3 x 10 + C) = (focused) tests Assumptions in layer 1 become tests in layer 2 -
Company confidential – Do not distribute without notice ©AE 2014
78 Integrated tests Sometimes you need integrated tests 3rd party dependencies Database UI/browser tests Keep them low in number and laser-focused Is the system glued together correctly? Use fake dependencies in rest of the tests -
Company confidential – Do not distribute without notice ©AE 2014
79 Hexagonal architecture http://www.natpryce.com/articles/000772.html -
Company confidential – Do not distribute without notice ©AE 2014
80 Hexagonal architecture http://www.natpryce.com/articles/000772.html -
Company confidential – Do not distribute without notice ©AE 2014
81 Hexagonal architecture http://www.natpryce.com/articles/000772.html -
Company confidential – Do not distribute without notice ©AE 2014
82 Sidenote: DB tests -
Company confidential – Do not distribute without notice ©AE 2014
83 Sidenote: DB tests -
Company confidential – Do not distribute without notice ©AE 2014
84 Sidenote: DB tests – also FIRST! -
Company confidential – Do not distribute without notice ©AE 2014
85 Sidenote: DB tests -
Company confidential – Do not distribute without notice ©AE 2014
86 Some guidelines Prefer solitary tests Very few “happy path” sociable tests if feeling anxious The “integrated test whisperer” No more than 12 End-to-end tests for entire system http://blog.jayfields.com/2011/07/high-level-test-whisperer.html -
Company confidential – Do not distribute without notice ©AE 2014
87 Unit testing - best practices What’s the problem with tests? -
Company confidential – Do not distribute without notice ©AE 2014
88 Unit testing - best practices What’s the problem with tests? -
Company confidential – Do not distribute without notice ©AE 2014
89 Unit testing - best practices Treat test code like production code: Should always work! Should last a long time Should be maintainable Should be readable Should be maintained by the entire team Should be in source control Should run every commit Follow same code guidelines as you would follow for production code. -
Company confidential – Do not distribute without notice ©AE 2014
90 Unit testing - best practices FIRST Fast Isolated Repeatable Timely -
Company confidential – Do not distribute without notice ©AE 2014
91 Good unit tests (GUTs) How fast? “If you have to ask how fast your test suite should be, it should be faster.” Corey Haines -
Company confidential – Do not distribute without notice ©AE 2014
92 Unit testing - best practices Should be easy to read and understand Should require minimum navigation/screen estate Avoid complex @Setup Avoid deep test class hierarchies Keep test code predictable: Assert Last, Expect Literals -
Company confidential – Do not distribute without notice ©AE 2014
93 Unit testing - best practices The happy path is not enough Test boundary conditions Null, empty String, … -
Company confidential – Do not distribute without notice ©AE 2014
94 Unit testing - best practices If your class is difficult to test it is probably too complex it is a candidate for refactoring a class is easier to test if it honors the SOLID principles: • Single responsibility • Open for extension / closed for modification • Liskov substitution • Interface segregation • Dependency inversion • Consider writing your tests first -
Company confidential – Do not distribute without notice ©AE 2014
95 Unit testing - best practices Design for testability No statics Contract-based development Role-based interfaces Single Responsibility Dependency Inversion Principle Dumb constructors … -
Company confidential – Do not distribute without notice ©AE 2014
96 Unit testing - best practices Always a trade-off Test-induced design damage But testability should be VERY high on your priority list http://david.heinemeierhansson.com/2014/test-induced-design-damage.html http://martinfowler.com/articles/is-tdd-dead/ -
Company confidential – Do not distribute without notice ©AE 2014
97 Testing internals -
Company confidential – Do not distribute without notice ©AE 2014
98 Good unit tests Test BEHAVIOUR, not IMPLEMENTATION Don’t test privates Test through “public API” Test a “meaningful behaviour” instead of a “method on a class” Act as executable specification of your system -
Company confidential – Do not distribute without notice ©AE 2014
99 Good unit tests -
Company confidential – Do not distribute without notice ©AE 2014
100 Good unit tests http://www.slideshare.net/Kevlin/programming-with-guts-41938796 -
Company confidential – Do not distribute without notice ©AE 2014
101 Good unit tests Single “assertion” per test One reason to fail Maximize feedback when a test fails • Which cases work, which cases don’t? • Narrow problem down OK to assert multiple facets of a same concept (?) Assert Last -
Company confidential – Do not distribute without notice ©AE 2014
102 Good unit tests Assert Last -
Company confidential – Do not distribute without notice ©AE 2014
103 Good unit tests DRY Don’t Repeat Yourself Mostly knowledge, not just mechanical code duplication SPOT: Single Point Of Truth DAMP Descriptive And Meaningful Phrases • Create a testing “DSL” that describes your SUT Descriptive And Maintainable Procedures • Avoid brittle tests http://verraes.net/2014/08/dry-is-about-knowledge/ http://blog.jayfields.com/2006/05/dry-code-damp-dsls.html -
Company confidential – Do not distribute without notice ©AE 2014
104 Good unit tests Prefer DAMP > DRY with a blowtorch Not mutually exclusive A constant trade-off -
Company confidential – Do not distribute without notice ©AE 2014
105 DAMP unit tests Test Data Builders & SUT Factory Decouple test from implementation Decrease churn • “new” in test code might be a smell • “new” in production code as well http://www.natpryce.com/articles/000714.html -
Company confidential – Do not distribute without notice ©AE 2014
106 DAMP unit tests Handrolled Test Data Builders -
Company confidential – Do not distribute without notice ©AE 2014
107 DAMP unit tests Test Data Builders with AutoFixture -
Company confidential – Do not distribute without notice ©AE 2014
108 DAMP unit tests Don’t overspecify your assertions/expectations Argument matchers -
Company confidential – Do not distribute without notice ©AE 2014
109 Test driven design UNIT TESTING -
Company confidential – Do not distribute without notice ©AE 2014
110 Why test-first? -
Company confidential – Do not distribute without notice ©AE 2014
111 Unit testing – test-first-desgin -
Company confidential – Do not distribute without notice ©AE 2014
112 Unit testing – test-driven-design It’s a design technique, not a testing technique -
Company confidential – Do not distribute without notice ©AE 2014
113 TDD – demo Roman numerals kata http://codingdojo.org/cgi-bin/index.pl?KataRomanNumerals -
Company confidential – Do not distribute without notice ©AE 2014
114 TDD – advanced topics Triangulation As tests get more specific, code gets more generic -
Company confidential – Do not distribute without notice ©AE 2014
115 Outside-in vs Inside-out development Outside-in Dream up “ideal API” Fake implementation of collaborators Finished implementation? • Test-drive implementation of collaborator s Stub vs. Mock Inside-out Create “building blocks” Compose application by combining real building blocks Cf. Solitary vs. Sociable unit tests -
Company confidential – Do not distribute without notice ©AE 2014
116 TDD – advanced topics Outside-in TDD/double loop TDD Start with failing acceptance test Unit test top-level object When you discover a new collaborator: interface + mock Rinse and repeat one level down http://coding-is-like-cooking.info/2013/04/outside-in-development-with-double-loop-tdd/ -
Company confidential – Do not distribute without notice ©AE 2014
117 Unit testing - labs Exercise: Test Driven Design Test-drive an implementation of FizzBuzz • Focus on red-green-refactor cycle • Triangulation http://wiki.c2.com/?FizzBuzzTest -
Company confidential – Do not distribute without notice ©AE 2014
118 Unit testing - BDD Behaviour-Driven Design -
Company confidential – Do not distribute without notice ©AE 2014
119 Unit testing - tools -
Company confidential – Do not distribute without notice ©AE 2014
120 Unit testing - tools -
Company confidential – Do not distribute without notice ©AE 2014
121 Unit testing - tools -
Company confidential – Do not distribute without notice ©AE 2014
122 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java -
Company confidential – Do not distribute without notice ©AE 2014
123 Powertools https://github.com/MajorT1990/TestingExercises/blob/master/java/src/test/java/TripServiceTest.java -
Company confidential – Do not distribute without notice ©AE 2014
125 How much of the code is under test? Code coverage If a line is executed by a unit test, it is covered. <> asserted on! Branch coverage Counts the number of branches/codepaths that were covered in a unit test. Metrics -
Company confidential – Do not distribute without notice ©AE 2014
126 A project with good coverage is not necessarily “well tested” Should never be a goal on itself But can be a serious smell if low Metrics on unit testing -
Company confidential – Do not distribute without notice ©AE 2014
127 Code coverage demo Metrics on unit testing -
Company confidential – Do not distribute without notice ©AE 2014
128 When to invest in tests? -
Company confidential – Do not distribute without notice ©AE 2014
129 WORKING WITH UNTESTED CODE -
Company confidential – Do not distribute without notice ©AE 2014
130 Legacy code Techniques for getting existing code under test -
Company confidential – Do not distribute without notice ©AE 2014
131 Legacy code The main problem Hidden and difficult to test dependencies make code impossible to test -
Company confidential – Do not distribute without notice ©AE 2014
132 Legacy code -
Company confidential – Do not distribute without notice ©AE 2014
133 Legacy code Solution: seams -
Company confidential – Do not distribute without notice ©AE 2014
134 Legacy code Characterization tests Write tests for existing code before changing it to lock in existing functionality Golden master ApprovalTests library -
Company confidential – Do not distribute without notice ©AE 2014
135 Legacy code Extract and override Introduce seams with minimal impact on production code -
Company confidential – Do not distribute without notice ©AE 2014
136 Legacy code Clean up that sh*t Replace inheritance with delegation Separate Lifecycle Management from behaviour Dependency Inversion Principle -
Company confidential – Do not distribute without notice ©AE 2014
137 Legacy code – where to start? http://www.slideshare.net/MozaicWorks/sandro-mancuso-testing-and-refactoring-legacy-code -
Company confidential – Do not distribute without notice ©AE 2014
138 Legacy code Use code coverage as a compass -
Company confidential – Do not distribute without notice ©AE 2014
139 Legacy code Exercise: Write isolated tests for the TripService class • Code coverage as a compass • Extract and override • Dependency Inversion • Refactoring Exercise Get the trivia codebase under test • Golden master • Hint: ApprovalTests library! • https://github.com/jbrains/trivia -
Company confidential – Do not distribute without notice ©AE 2014
140 Further reading Test-Driven Development, by example, Beck The Art Of Unit Testing, Osherove Working effectively with Legacy Code, Feathers Growing Object-Oriented Software, guided by tests, Pryce & Freeman Xunit test patterns, refactoring test code, Meszaros Working effectively with unit tests, Fields