Share
Embed
Developer Tests - Things to Know (Vilnius JUG)
There are many great talks that discuss challenges developers face when writing software tests. In this talk let's look at test design problems that may seem to be simple but yet fundamentally important and often misunderstood even by experienced programmers.
Vaidas Pilkauskas
December 11, 2014
Transcript
103 slides
-
me My hobbies • developer at Wix.com • main language
- Scala • main professional interest - developer communities If there is time left after my hobbies • mountain bicycle rider, snowboarder • consumer of rock music, contemporary art, etc -
me - how to contact me connect with me on
LinkedIn http://lt.linkedin. com/pub/vaidas-pilkauskas/8/77/863/ add me on G+ https://www.google.com/+VaidasPilkauskas follow on Twitter @liucijus -
“We couldn’t understand why people without technical knowledge had to
tell programmers “what” to do and, furthermore, they had to supervise “how” programmers did it.” Cristian Rennella http://qz.com/260846/why-our-startup-has-no-bosses-no-office-and-a-four-day-work-week/ -
What this talk is about • Things we argue about
during code reviews • Things that took me time to understand and prove that they are actually good way to go • Small things we have no time to discuss during big talks -
So what is test? It is system’s exercise under predefined
conditions and then verification of an expected outcome. -
Test phases in code Server server = new NotesServer(); //
setup Note note = new Note("test note"); // setup Status status = server.add(note); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // teardown -
Start everything in one method @Test public void serverShouldAddNoteSuccessfully() {
Server server = new NotesServer(); // setup Note note = new Note("test note"); // setup Status status = server.add(note); // exercise assertEquals(SUCCESS, status); // verify server.shutdown(); // teardown } -
Refactor to lifecycle methods @Before public void before() { server
= new NotesServer(); // setup note = new Note("test note"); // setup } @Test public void serverShouldAddNoteSuccessfully() { Status status = server.add(note); // exercise assertEquals(SUCCESS, status); // verify } @After public void after() { server.shutdown(); // teardown } -
DON’Ts - #2 Do not mix setup code from different
tests - shared code must be relevant to the tests that use it -
DON’Ts - #3 Setup and teardown are there to solve
DRY problem and to help structure your test code -
How many colors are there in the picture? Source: Josef
Albers Interaction of Color -
Typical setup inside @Before @Before public void before() { server
= new NotesServer(); // setup note = new Note("test note"); // setup } @Test public void serverShouldAddNoteSuccessfully() { Status status = server.add(note); // exercise assertEquals(SUCCESS, status); // verify } @After public void after() { server.shutdown(); // teardown } -
Setup pollution - #1 • Every line in setup must
be relevant to all tests. -
Setup pollution - #2 • It is tempting to add
additional setup tuning just to fix/enhance one test. -
Setup pollution - #5 • Setup hides details! Do not
hide test preconditions. -
Setting up “job” inside test method @Before public void before()
{ server = new NotesServer(); // setup } @Test public void serverShouldAddNoteSuccessfully() { note = new Note("test note"); // setup Status status = server.add(note); // exercise assertEquals(SUCCESS, status); // verify } -
But then be more specific @Test public void serverShouldAddSingleLineNoteSuccesfully() {
// * set up which is actual for the current method // * use scope specific name Note singleLineNote = new Note("test note"); // setup Status status = server.add(singleLineNote); // exercise assertEquals(SUCCESS, status); // verify } -
Give good names to setup methods @Before public void createNotesServer()
{ server = new NotesServer(); } -
Summary of test code organization • DRY principle. • Readability.
BDD vs. DRY • Consistency. Maintain the same style across your codebase. • Complexity. It may dictate the way you go. -
Refactoring Refactoring is about improving the design of existing code.
It is the process of changing a software system in such a way that it does not alter the external behavior of the code, yet improves its internal structure. Martin Fowler Refactoring: Improving the Design of Existing Code -
Test behaviour not methods • Think of a contract •
And responsibilities -
Test behaviour not methods • Think of a contract •
And responsibilities • Specify requirements as tests -
Test behaviour not methods • Think of a contract •
And responsibilities • Specify requirements as tests • Happens naturally when done in test-first approach -
Matchers • Enhance readability • Asserts on the right level
of abstraction • Encapsulate testing logic • Reusable • Detailed match error messages (do not leave them out in your custom matchers!) -
Matchers • Enhance readability • Asserts on the right level
of abstraction • Encapsulate testing logic • Reusable • Detailed match error messages (do not leave them out in your custom matchers!) -
Matcher libraries • Hamcrest - standard matcher lib for JUnit
• AssertJ - fluent assertions (IDE friendly) • Provides common matchers • You can write your own custom matchers -
Custom matchers • Help communicate test intention • Abstract assertion
logic in case standard matchers are not enough • Are reusable and save time in large projects • You may have a custom message to be more specific about test failure -
Custom matchers • Help communicate test intention • Abstract assertion
logic in case standard matchers are not enough • Are reusable and save time in large projects • You may have a custom message to be more specific about test failure -
Custom matchers @Test public void shouldHaveIsbnGenerated() { Book book =
new Book(1l, "5555", "A book"); assertThat(book, hasIsbn("1234")); } -
fail() In some cases (e.g. testing exceptions) you may want
to force test to fail if some expected situation does not happen -
fail() try { // do stuff... fail("Exception not thrown"); }
catch(Exception e){ assertTrue(e.hasSomeFlag()); } -
fail() • Fundamentally not bad, but better use matchers for
expected failure • Matchers help to clarify test intention • Don’t forget - expected behaviour is an opposite of a failing test -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); String expected = "Person[" + person.getName() + " " + person.getSurname() + "]" assertEquals(expected, person.toString()); } -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); String expected = "Person[" + person.getName() + " " + person.getSurname() + "]" assertEquals(expected, person. toString()); } -
Anti-pattern: The Ugly Mirror @Test public void personToStringShouldIncludeNameAndSurname() { Person
person = new Person("Vilkas", "Pilkas"); assertEquals("Person[Vilkas Pilkas]", person.toString()); } -
Why would you want to turn off the test? Well,
because it fails… :) -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) • Do not comment out or false assert your test -
Ignoring tests • Always use ignore/pending API from your test
library (JUnit @Ignore) • Do not comment out or false assert your test • If you do not need a test - delete it -
Exceptions • If you can, use matchers instead of ◦
@Test(expected=?) ◦ try-catch approach -
JUnit expected exception @Test(expected=IndexOutOfBoundsException.class) public void shouldThrowIndexOutOfBoundsException() { ArrayList emptyList
= new ArrayList(); Object o = emptyList.get(0); } //matcher in Specs2 (Scala) server.process(None) must throwA[NothingToProccess] -
try and catch public void shouldThrowIndexOutOfBoundsException() { ArrayList emptyList =
new ArrayList(); try { Object o = emptyList.get(0); fail("Should throw IndexOutOfBoundsException"); } catch(IndexOutOfBoundsException e)){ //consider asserting message! } } -
catch-exception lib List myList = new ArrayList(); catchException(myList).get(1); assertThat(caughtException(), allOf(
is(IndexOutOfBoundsException.class), hasMessage("Index: 1, Size: 0"), hasNoCause() ) ); -
Exceptions • What about ExpectedException Rule? ◦ My personal opinion
- not that intuitive ◦ breaks arrange/act/assert flow -
ExpectedException rule @Rule public ExpectedException exception = ExpectedException.none(); @Test public
void testExpectedException() { exception.expect(IllegalArgumentException.class); exception.expectMessage(containsString('Invalid age')); new Person('Vilkas', -1); } //Person constructor public Person(String name, int age) { if (age <= 0) throw new IllegalArgumentException('Invalid age:' + age); // ... } -
Problem public class MyService { ... public void process(LocalDate date)
{ if (date.isBefore(LocalDate.now()) { ... } } } -
Testing with Time • Design your system where time is
a collaborator • Inject test specific time provider in your test ◦ constant time ◦ slow time ◦ boundary cases time -
Control time with Clock public class MyService { private Clock
clock; // dependency inject ... public void process(LocalDate date) { if (date.isBefore(LocalDate.now(clock)) { ... } } } -
Collections - multiple properties to assert • Is null? •
Size • Order • Content -
Collections • Most of the time you want to assert
on collection content -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion • Unless you want to say something important in your test! -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion • Unless you want to say something important in your test! • Use matchers! -
Collections • Most of the time you want to assert
on collection content • Prefer exact content matching • Avoid incomplete assertions • Do not sort just because it is easier to assert! • Multiple assertions are worse than single content assertion • Unless you want to say something important in your test! • Use matchers! -
Random values in tests • Most of the time you
do not want it -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot (eg. password generation*) *Thanks to Aleksandar Tomovski for a good example -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot • Use property based testing (which is also hard) -
Random values in tests • Most of the time you
do not want it • Unless you depend on randomness a lot • Use property based testing (which is also hard) • Do not make dummy values random -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests • Random is hard to repeat -
Generate Multiple Test Cases • Quality over quantity • Think
of boundary cases, that you may want to detect with random test • Use parameterized tests • Random is hard to repeat • Flickering tests -
How many assertions per test? • Unit test - one
assertion per test. Must be clear and readable • Proper test should fail for exactly one reason • End to end - best case one assertion per test, but more are allowed • Consider custom matchers -
How many assertions per test? • Unit test - one
assertion per test. Must be clear and readable • Proper test should fail for exactly one reason • End to end - best case one assertion per test, but more are allowed • Consider custom matchers -
What can be better in this test [pseudocode] @Test shouldRetrieveUserByLogin()
{ String userJson = "{\"username\": \"vaidas\"}"; HttpRequest post = new Post("https://localhost:8080/users", userJson); HttpResponse postResp = HttpClient().execute(post); assertThat(postResp.status, is(200)); HttpRequest get = new Get("https://localhost:8080/users/vaidas"); HttpResponse getResp = HttpClient().execute(get); User user = mapper.readValue(getResp, User.class); assertThat(username, is("vaidas")); } -
Decoupling from low level details [pseudocode] @Test shouldRetrieveUserByUsername() { CreateUserResponse
createResp = aCreateUserRequest().withUsername("vaidas").execute(); assertThat(createResp, hasStatus(OK)); GetUserResponse getResp = aGetUserRequest().withUsername("vaidas").execute(); assertThat(getResp, allOf(hasStatus(OK), hasUsername("vaidas"))); } -
Thoughts on end-to-end testing • What is real e2e? REST
vs. UI • Testing by using system itself • Vs. using low level DB driver to verify data persistence -
Comments in Test code • Fundamentally good option to explain
complicated parts, but: • better use good method naming • custom matcher • do less, so that intention is clear • comments are not so bad in isolated well named methods -
Summary • Context matters • Tests are controversial topic •
And very opinionated