Unit Testing: From Spek 2 to JUnit 5

The slides are from a talk I gave internally at Babylon Health. In this talk, I describe our strategy for migrating our unit tests on Android from Spek 2 to JUnit 5, detailing our reasons for migrating as well as the migration process itself and our thoughts along the way.

Transcript

1. Unit Testing: From Spek 2 to JUnit 5 Darren Atherton
2. Solving a Problem
3. Why Not Spek 2? • Performance ◦ IDE hanging completely for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
4. Why Not Spek 2? • Performance ◦ IDE hanging completely for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
5. Why Not Spek 2? • Performance ◦ IDE hanging completely for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
6. Why Not Spek 2? • Performance ◦ IDE hanging completely for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
7. Why Not Spek 2? // assertion failed! • Problems running single tests • Problems running any tests
8. Why Not Spek 2? // assertion failed! • Problems running single tests • Problems running any tests
9. Why Not Spek 2? // assertion failed! • Problems running single tests • Problems running any tests
10. Why Not Spek 2? • Timeout and flakiness issues // assertions failed!
11. Why Not Spek 2? • Memoization issues ◦ Hard to understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
12. Why Not Spek 2? • Memoization issues ◦ Hard to understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
13. Why Not Spek 2? • Memoization issues ◦ Hard to understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
14. Why Not Spek 2? • Memoization issues ◦ Hard to understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
15. Why Not Spek 2? • Memoization issues ◦ Hard to understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
16. Why Not Spek 2? • lateinit var fields • Nullable fields Introduces subtle initialisation bugs because fields are not immutable Adds noise to the test as we must deal with the nullable type throughout the test
17. Why Not Plain JUnit? • Lack of structure ◦ No consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
18. Why Not Plain JUnit? • Lack of structure ◦ No consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
19. Why Not Plain JUnit? • Lack of structure ◦ No consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
20. Why Not Plain JUnit? • Lack of structure ◦ No consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
21. Why Not JUnit 4? • JUnit 5: ◦ Has better support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
22. Why Not JUnit 4? • JUnit 5: ◦ Has better support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
23. Why Not JUnit 4? • JUnit 5: ◦ Has better support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
24. Why Not JUnit 4? • JUnit 5: ◦ Has better support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
25. Why Not JUnit 4? • JUnit 5: ◦ Has better support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
26. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
27. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
28. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
29. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
30. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
31. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
32. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
33. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
34. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
35. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
36. Influences & Objectives of the Framework • Influences ◦ Responses to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
37. Implementing the Framework
38. Start With a Plan 1. Create a proof-of-concept/proposal PR with: a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint checks and other in-code checks to enforce structure/consistency
39. An Iterative Approach • 9 files changes: ◦ 3 Gradle files updated ◦ 1 Spek test removed ◦ 1 JUnit proof-of-concept test added ◦ 1 structure documentation added ◦ 1 background documentation added ◦ 1 Framework DSL test class added ◦ 1 Framework DSL added …and many zoom calls
40. Framework DSL - Objectives • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
41. Framework DSL - Enforcing Structure • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
42. Framework DSL - Fast Error Checking • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
43. Framework DSL - Shallow Nesting • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // 5+ nested lambdas deep // 1 large (1000+ LoC) lambda
44. Framework DSL - Ease of Debugging • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // WYSIWYG // Spek tests don’t run linearly, // but instead run in two // separate phases - discovery // and execution // JUnit tests run in the order // as you can see in the code and // do not timeout
45. Framework DSL - Readability via Grouping • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy 1 2 3 4 1. Inputs/outputs 2. Given/And 3. When/And 4. Then/And
46. Framework DSL - Readability via Indentation • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
47. Framework DSL - Removing ‘lateinit var’ • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // “Contracts allow a function to explicitly describe its // behavior in a way which is understood by the compiler” // If we know that our ‘Given’ block will be called // exactly once, then the compiler does not require us to // use the ‘lateinit var’ workaround.
48. Framework DSL - Disabling Nesting of Gherkin Blocks • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
49. Framework DSL - Parameterization • How can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
50. Parameterization - Default Argument Sources • @NullSource - Useful for checking if a function supports a single null parameter. • @EmptySource - Pass in an empty List, Set, Map or Array • @ValueSource - Basic lists of short, byte, int, long, float, double, char, boolean, String or Class. ◦ @ValueSource(longs = { 4, 8, 15, 16, 23 }) • @EnumSource - Enums (can filter a subset of the Enums values). ◦ @EnumSource(DeepLinkHost::class) • @CsvSource - Slightly more complex rows/columns of data. ◦ @CsvSource({ "Welcome, 1", "to, 2", "JUnit, 5" }) • @MethodSource - Construct more complex streams of arguments. ◦ @MethodSource("createAppointmentTestCases")
51. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
52. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
53. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
54. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
55. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
56. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
57. Parameterization - Supplying Complex Data • @MethodSource method must be static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
58. Parameterized Tests - Supplying Complex Lists • Supports List, Queue and Set or a variable number of some type. • Primary constructor is useful for pre-existing DataProvider functions. • If your data is a simple/static list, just use @ValueSource.
59. Parameterized Tests - Supplying Complex Lists • For more complex data (e.g. where each row has 3+ values), we recommend to create a data class (in this case called ‘TestCase’) and subclass the ArgumentsListProvider.
60. Parameterized Tests - Supplying Complex Pairs/Maps • Supports Map or a variable number of Pair. • Primary constructor is useful for pre-existing DataProvider functions. • Useful for passing in pairs of input and an expected result - e.g. mapper tests.
61. Parameterized Tests - Supplying Complex Pairs/Maps
62. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦ Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
63. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦ Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
64. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦ Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
65. JUnit Framework: The Lost Levels
66. Parameterization - A First Attempt (Dynamic Tests) // expects function to return a collection of DynamicTest // “If you access fields from the test instance // within a lambda expression for a dynamic test, // those fields will not be reset by callback // methods or extensions between the execution of // individual dynamic tests generated by the same // @TestFactory method.”
67. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’ in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
68. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’ in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
69. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’ in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
70. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’ in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
71. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’ in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
72. Migrating the Tests
73. Migration Rules Spek 2 JUnit 5 Gherkin DSL Feature (meaning Spek files with multiple features are split into multiple JUnit files) Class Scenario @Test function forEach loop @ParameterizedTest lateinit var val private val s by memoized { “s” } private val s = “s” beforeEachTest { } Define vals at top of function or class
74. Migration Rules Two reasons to have a well-defined migration model before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
75. Migration Rules Two reasons to have a well-defined migration model before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
76. Migration Rules Two reasons to have a well-defined migration model before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
77. Converting the Tests Spek conversion Spek conversion Feature maintenance that updates tests develop
78. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
79. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
80. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
81. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
82. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
83. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
84. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
85. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
86. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
87. Converting the Tests • Objectives ◦ Keep time to review a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
88. Enforcing Consistency and Detecting Errors
89. Potential Checks - Lint • White space rule around block sections. ‘Input/Output’, ‘Given/And’, ‘When/And’ and ‘Then/And’ sections should be surrounded by one line of white space. • Ensure that we are not using the@MethodSource annotation for @ParameterizedTest tests. Instead, we should use our helper classes from the DSL to construct streams of input data. • No use of AssertJ ‘Assertions.assertThat’ without statically importing the function for readability • No empty Gherkin blocks
90. Potential Checks - Framework Code • ‘Given’, ‘When’, ‘Then’ and ‘And’ descriptions cannot be blank. If a blank description is provided, the block throws an IllegalArgumentException (implemented). • No duplicate block descriptions. This prevents any copy/paste errors where two blocks (e.g. ‘Given’ and ‘And’ have the same description. This will be enforced at runtime in the Gherkin DSL class. • No blank space at the start or end of descriptions. • Capitalisation (or no capitalisation) of descriptions. • Order of ‘Given’, ‘When’, ‘Then’ and ‘And’ blocks must be correct. This could be enforced at runtime in the Gherkin DSL class by tracking the blocks being executed and checking the neighbours of each block. • Checking if there are no ‘Then’ blocks (usually means we are not asserting anything in our test)
91. Framework Checks - Finding a Balance Framework consistency Developer experience
92. Where Are We Now? 1. Create a proof-of-concept/proposal PR with: a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint and other in-code checks to enforce structure/consistency
93. Documentation • unit-testing-background.md ◦ A full write-up of our requirements/goals, framework design, performance benchmarks and the different approaches considered with rationale for each. ◦ Read this when you want to know how the various strategies we discussed. • unit-test-structure.md ◦ The outcome of the background research containing the proposed framework and structure/style of our tests. ◦ Use this as a reference for the final framework, gherkin conventions, parameterization and guidelines on how to write a JUnit 5 Gherkin test. • unit-tests.md ◦ A more general document of unit testing in the project describing how to run tests, how to write readable tests, naming conventions etc. ◦ Use this as a reference for how to write tests, e.g. how to mock use cases, which assertion framework to use, how to handle Observables.
94. Q&A