Share
Embed
Developer Tests - Things to Know
There are many great talks that discuss challenges developers face when writing software tests. In this talk let's look at test design problems that may seem to be simple but yet fundamentally important and often misunderstood even by experienced programmers.
Vaidas Pilkauskas
October 15, 2014
Transcript
108 slides
-
“We couldn’t understand why people without technical knowledge had to
tell programmers “what” to do and, furthermore, they had to supervise “how” programmers did it.” Cristian Rennella http://qz.com/260846/why-our-startup-has-no-bosses-no-office-and-a-four-day-work-week/ -
What I’m going to talk about • Things we argue
about during code reviews • Things that took me time to understand and prove that they are actually good way to go • Small things we have no time to discuss during big talks -
So what is test? It is system’s exercise under predefined
conditions and then verification of an expected outcome. -
Test phases @Test public void serverShouldExecuteJobSuccessfully() { Server server =
new Server(); // set up Job job = new Job(); // set up Status status = server.execute(job); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // tear down } -
@Before public void before() { server = new Server(); }
@Test public void serverShouldExecuteJobSuccessfully() { Job job = new Job(); // set up Status status = server.execute(job); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // tear down } -
@Before public void before() { server = new Server(); Job
job = new Job(); } @Test public void serverShouldExecuteJobSuccessfully() { Status status = server.execute(job); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // tear down } -
@Test public void serverShouldQueueJobWithFutureDate() { // * set up which
is actual for the current method // * use scope specific name Job futureJob = new Job(futureDate()); // set up Status status = server.execute(futureJob); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // tear down } -
@Before public void before() { server = new Server(); Job
job = new Job(); } @Test public void serverShouldExecuteJobSuccessfully() { Status status = server.execute(job); // exercise assertEquals(SUCCESS, status); // verify } @After public void after() { server.shutdown(); // tear down } -
@Before .. @Test public void serverShouldExecuteJobSuccessfully() { // * no
need to name intermediate var, but // * may hide return meaning of server.execute() // execute & verify assertEquals(SUCCESS, server.execute(job)); } @After .. -
Refactoring Refactoring is about improving the design of existing code.
It is the process of changing a software system in such a way that it does not alter the external behavior of the code, yet improves its internal structure. Martin Fowler Refactoring: Improving the Design of Existing Code -
Test behaviour not methods • Think of a contract •
And responsibilities -
Test behaviour not methods • Think of a contract •
And responsibilities • Specify requirements as tests -
Test behaviour not methods • Think of a contract •
And responsibilities • Specify requirements as tests • Happens naturally when done in test-first approach -
Matchers • Enhanced readability • Assertions on the right level
of abstraction • Encapsulate testing logic • Reusable • Detailed match error messages (do not leave them out in your custom matchers!) -
Matcher libraries • Hamcrest - standard matcher lib for JUnit
• AssertJ - fluent assertions (IDE friendly) • Bring common matchers for you to use • Write your own custom matchers -
Custom matchers • Help communicate test intention • Abstract assertion
logic in case standard matchers are not enough • Are reusable and save time in large projects • You may have custom message to be more specific about test failure -
Custom matchers @Test public void testBookIsbn() { Book book =
new Book(1l, "5555", "A book"); assertThat(book, hasIsbn("1234")); } -
fail() In some cases like testing exceptions you may want
to force test to fail if some expected situation does not happen -
fail() try{ // do stuff... fail("Exception not thrown"); }catch(Exception e){
assertTrue(e.hasSomeFlag()); } -
fail() • Fundamentally not bad, but better use matchers for
expected failure • Matchers help to clarify test intention • Don’t forget - expected behaviour is an opposite of a failing test -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); String expected = "Person[" + person.getName() + " " + person.getSurname() + "]" assertEquals(expected, person.toString()); } -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); String expected = "Person[" + person.getName() + " " + person.getSurname() + "]" assertEquals(expected, person. toString()); } -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); assertEquals("Person[Vilkas Pilkas]", person.toString()); } -
Why would you want to turn off the test? •
Well, because it fails… :) -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) • Do not comment out or false assert your test -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) • Do not comment out or false assert your test • If you do not need a test - delete it -
Exceptions • If you can, use matchers instead of ◦
@Test(expected=?) -
JUnit expected exception @Test(expected=IndexOutOfBoundsException.class) public void shouldThrowIndexOutOfBoundsException() { ArrayList emptyList
= new ArrayList(); Object o = emptyList.get(0); } //matcher in Specs2 (Scala) server.process(None) must throwA[NothingToProccess] -
Exceptions • If you can, use matchers instead of ◦
@Test(expected=?) ◦ try-catch approach -
try and catch public void shouldThrowIndexOutOfBoundsException() { ArrayList emptyList =
new ArrayList(); try { Object o = emptyList.get(0); fail("Should throw IndexOutOfBoundsException"); } catch(IndexOutOfBoundsException e)){ //consider asserting message! } } -
Exceptions • If you can, use matchers instead of ◦
@Test(expected=?) ◦ try-catch approach • catch-exception lib -
catch-exception lib List myList = new ArrayList(); catchException(myList).get(1); assertThat(caughtException(), allOf(
is(IndexOutOfBoundsException.class), hasMessage("Index: 1, Size: 0"), hasNoCause() ) ); -
Exceptions • If you can, use matchers instead of ◦
@Test(expected=?) ◦ try-catch approach • catch-exception lib • What about ExpectedException Rule? ◦ My personal opinion - not that intuitive ◦ breaks arrange/act/assert flow -
ExpectedException rule @Rule public ExpectedException exception = ExpectedException.none(); @Test public
void testExpectedException() { exception.expect(IllegalArgumentException.class); exception.expectMessage(containsString('Invalid age')); new Person('Vilkas', -1); } //Person constructor public Person(String name, int age) { if (age <= 0) throw new IllegalArgumentException('Invalid age:' + age); // ... } -
Asynchronous code • Do not Thread.sleep - makes test slow
• Use Awaitility, or similar DSL for synchronizing asynchronous operations -
Awaitility (Java 8 example) @Test public void shouldPersistNewUser() { publish(new
CreateUserCommand("Vilkas Pilkas")); await().until(userRepo::size, is(1)); //how long to await? (Default is 10 seconds) await().until(userRepo::isNotEmpty); } -
Asynchronous code • Do not Thread.sleep - makes test slow
• Use Awaitility, or similar DSL for synchronizing asynchronous operations • Use reasonable await time to avoid flaky tests -
Problem public class MyService { ... public void process(LocalDate date)
{ if (date.isBefore(LocalDate.now()) { ... } } } -
Testing with Time • Design your system where time is
a collaborator • Inject test specific time provider in your test ◦ constant time ◦ slow time ◦ boundary cases time -
Control time with Clock public class MyService { private Clock
clock; // dependency inject ... public void process(LocalDate date) { if (date.isBefore(LocalDate.now(clock)) { ... } } } -
Collections - multiple properties to assert • Is null? •
Size • Order • Content -
Collections • Most of the time you want to assert
on collection content -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion • Unless you want to say something important in your test! -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion • Unless you want to say something important in your test! • Use matchers! -
Access modifiers • Rule is simple - never access anything
that is not public in your tests -
Access modifiers • Rule is simple - never access anything
that is not public in your tests • Private things are implementation details which are not part of the public contract -
Access modifiers • Rule is simple - never access anything
that is not public in your tests • Private things are implementation details which are not part of the public contract • Same applies for protected/package modifiers. They must be there for production code, but not available to your tests -
Random values in tests • Most of the time you
do not want it -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot (eg. password generation*) *Thanks to Aleksandar Tomovski for a good example -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot • Use property based testing (which is also hard) -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot • Use property based testing (which is also hard) • Do not make dummy values random -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests • Random is hard to repeat -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests • Random is hard to repeat • Flickering tests -
How many assertions per test? • Unit test - one
assertion per test. Must be clear and readable • Proper unit tests should fail for exactly one reason • End to end - best case one assertions per test, but more allowed • Consider custom matchers -
Test Doubles The name comes from the notion of a
Stunt Double in movies -
Why do we need test doubles? • To test in
an isolated environment by replacing real collaborators with doubles • To have fast tests • To test interactions • To change collaborators behaviour in test -
Types of Test Doubles • Dummy • Fake • Stub
• Spy • Mock -
Dummy Dummy objects are passed around but never actually used.
Usually they are just used to fill parameter lists -
Fake Fake objects actually have working implementations, but usually take
some shortcut which makes them not suitable for production (an in memory database is a good example) -
Stub Stubs provide canned answers to calls made during the
test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent' -
Mock Mocks are what we are talking about here: objects
pre-programmed with expectations which form a specification of the calls they are expected to receive -
Where to keep your tests • Recommendation: keep them in
separate package from production code -
Class Naming • End test classes with Test suffix (or
one required by your test framework) • For long test classes: ◦ split by feature ◦ by input type -
Method Naming • Everybody should follow the same pattern •
testGeneratePassword vs shouldGenerateValidPassword -
Comments in Test code • Fundamentally good option to explain
complicated parts, but: • better use good method naming • custom matcher • do less, so that intention is clear • comments are not so bad in isolated well named set up method (not the first thing to be seen in test method)