Share
Embed
Unit Testing: From Spek 2 to JUnit 5
The slides are from a talk I gave internally at Babylon Health. In this talk, I describe our strategy for migrating our unit tests on Android from Spek 2 to JUnit 5, detailing our reasons for migrating as well as the migration process itself and our thoughts along the way.
Darren Atherton
August 24, 2020
Transcript
94 slides
-
Why Not Spek 2? • Performance ◦ IDE hanging completely
for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor -
Why Not Spek 2? • Performance ◦ IDE hanging completely
for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor -
Why Not Spek 2? • Performance ◦ IDE hanging completely
for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor -
Why Not Spek 2? • Performance ◦ IDE hanging completely
for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor -
Why Not Spek 2? // assertion failed! • Problems running
single tests • Problems running any tests -
Why Not Spek 2? // assertion failed! • Problems running
single tests • Problems running any tests -
Why Not Spek 2? // assertion failed! • Problems running
single tests • Problems running any tests -
Why Not Spek 2? • Timeout and flakiness issues //
assertions failed! -
Why Not Spek 2? • Memoization issues ◦ Hard to
understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking -
Why Not Spek 2? • Memoization issues ◦ Hard to
understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking -
Why Not Spek 2? • Memoization issues ◦ Hard to
understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking -
Why Not Spek 2? • Memoization issues ◦ Hard to
understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking -
Why Not Spek 2? • Memoization issues ◦ Hard to
understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking -
Why Not Spek 2? • lateinit var fields • Nullable
fields Introduces subtle initialisation bugs because fields are not immutable Adds noise to the test as we must deal with the nullable type throughout the test -
Why Not Plain JUnit? • Lack of structure ◦ No
consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple! -
Why Not Plain JUnit? • Lack of structure ◦ No
consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple! -
Why Not Plain JUnit? • Lack of structure ◦ No
consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple! -
Why Not Plain JUnit? • Lack of structure ◦ No
consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple! -
Why Not JUnit 4? • JUnit 5: ◦ Has better
support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc -
Why Not JUnit 4? • JUnit 5: ◦ Has better
support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc -
Why Not JUnit 4? • JUnit 5: ◦ Has better
support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc -
Why Not JUnit 4? • JUnit 5: ◦ Has better
support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc -
Why Not JUnit 4? • JUnit 5: ◦ Has better
support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Influences & Objectives of the Framework • Influences ◦ Responses
to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast -
Start With a Plan 1. Create a proof-of-concept/proposal PR with:
a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint checks and other in-code checks to enforce structure/consistency -
An Iterative Approach • 9 files changes: ◦ 3 Gradle
files updated ◦ 1 Spek test removed ◦ 1 JUnit proof-of-concept test added ◦ 1 structure documentation added ◦ 1 background documentation added ◦ 1 Framework DSL test class added ◦ 1 Framework DSL added …and many zoom calls -
Framework DSL - Objectives • How can we create the
framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Framework DSL - Enforcing Structure • How can we create
the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Framework DSL - Fast Error Checking • How can we
create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Framework DSL - Shallow Nesting • How can we create
the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // 5+ nested lambdas deep // 1 large (1000+ LoC) lambda -
Framework DSL - Ease of Debugging • How can we
create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // WYSIWYG // Spek tests don’t run linearly, // but instead run in two // separate phases - discovery // and execution // JUnit tests run in the order // as you can see in the code and // do not timeout -
Framework DSL - Readability via Grouping • How can we
create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy 1 2 3 4 1. Inputs/outputs 2. Given/And 3. When/And 4. Then/And -
Framework DSL - Readability via Indentation • How can we
create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Framework DSL - Removing ‘lateinit var’ • How can we
create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // “Contracts allow a function to explicitly describe its // behavior in a way which is understood by the compiler” // If we know that our ‘Given’ block will be called // exactly once, then the compiler does not require us to // use the ‘lateinit var’ workaround. -
Framework DSL - Disabling Nesting of Gherkin Blocks • How
can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Framework DSL - Parameterization • How can we create the
framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy -
Parameterization - Default Argument Sources • @NullSource - Useful for
checking if a function supports a single null parameter. • @EmptySource - Pass in an empty List, Set, Map or Array • @ValueSource - Basic lists of short, byte, int, long, float, double, char, boolean, String or Class. ◦ @ValueSource(longs = { 4, 8, 15, 16, 23 }) • @EnumSource - Enums (can filter a subset of the Enums values). ◦ @EnumSource(DeepLinkHost::class) • @CsvSource - Slightly more complex rows/columns of data. ◦ @CsvSource({ "Welcome, 1", "to, 2", "JUnit, 5" }) • @MethodSource - Construct more complex streams of arguments. ◦ @MethodSource("createAppointmentTestCases") -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterization - Supplying Complex Data • @MethodSource method must be
static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype -
Parameterized Tests - Supplying Complex Lists • Supports List, Queue
and Set or a variable number of some type. • Primary constructor is useful for pre-existing DataProvider functions. • If your data is a simple/static list, just use @ValueSource. -
Parameterized Tests - Supplying Complex Lists • For more complex
data (e.g. where each row has 3+ values), we recommend to create a data class (in this case called ‘TestCase’) and subclass the ArgumentsListProvider. -
Parameterized Tests - Supplying Complex Pairs/Maps • Supports Map or
a variable number of Pair. • Primary constructor is useful for pre-existing DataProvider functions. • Useful for passing in pairs of input and an expected result - e.g. mapper tests. -
Repeating Tests & Non-Determinism • Useful for two scenarios: ◦
Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors. -
Repeating Tests & Non-Determinism • Useful for two scenarios: ◦
Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors. -
Repeating Tests & Non-Determinism • Useful for two scenarios: ◦
Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors. -
Parameterization - A First Attempt (Dynamic Tests) // expects function
to return a collection of DynamicTest // “If you access fields from the test instance // within a lambda expression for a dynamic test, // those fields will not be reset by callback // methods or extensions between the execution of // individual dynamic tests generated by the same // @TestFactory method.” -
Adding (and Removing) Coroutine Support • No need for ‘runBlocking’
in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much -
Adding (and Removing) Coroutine Support • No need for ‘runBlocking’
in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much -
Adding (and Removing) Coroutine Support • No need for ‘runBlocking’
in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much -
Adding (and Removing) Coroutine Support • No need for ‘runBlocking’
in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much -
Adding (and Removing) Coroutine Support • No need for ‘runBlocking’
in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much -
Migration Rules Spek 2 JUnit 5 Gherkin DSL Feature (meaning
Spek files with multiple features are split into multiple JUnit files) Class Scenario @Test function forEach loop @ParameterizedTest lateinit var val private val s by memoized { “s” } private val s = “s” beforeEachTest { } Define vals at top of function or class -
Migration Rules Two reasons to have a well-defined migration model
before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another. -
Migration Rules Two reasons to have a well-defined migration model
before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another. -
Migration Rules Two reasons to have a well-defined migration model
before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another. -
Converting the Tests Spek conversion Spek conversion Feature maintenance that
updates tests develop -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Converting the Tests • Objectives ◦ Keep time to review
a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process -
Potential Checks - Lint • White space rule around block
sections. ‘Input/Output’, ‘Given/And’, ‘When/And’ and ‘Then/And’ sections should be surrounded by one line of white space. • Ensure that we are not using the@MethodSource annotation for @ParameterizedTest tests. Instead, we should use our helper classes from the DSL to construct streams of input data. • No use of AssertJ ‘Assertions.assertThat’ without statically importing the function for readability • No empty Gherkin blocks -
Potential Checks - Framework Code • ‘Given’, ‘When’, ‘Then’ and
‘And’ descriptions cannot be blank. If a blank description is provided, the block throws an IllegalArgumentException (implemented). • No duplicate block descriptions. This prevents any copy/paste errors where two blocks (e.g. ‘Given’ and ‘And’ have the same description. This will be enforced at runtime in the Gherkin DSL class. • No blank space at the start or end of descriptions. • Capitalisation (or no capitalisation) of descriptions. • Order of ‘Given’, ‘When’, ‘Then’ and ‘And’ blocks must be correct. This could be enforced at runtime in the Gherkin DSL class by tracking the blocks being executed and checking the neighbours of each block. • Checking if there are no ‘Then’ blocks (usually means we are not asserting anything in our test) -
Where Are We Now? 1. Create a proof-of-concept/proposal PR with:
a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint and other in-code checks to enforce structure/consistency -
Documentation • unit-testing-background.md ◦ A full write-up of our requirements/goals,
framework design, performance benchmarks and the different approaches considered with rationale for each. ◦ Read this when you want to know how the various strategies we discussed. • unit-test-structure.md ◦ The outcome of the background research containing the proposed framework and structure/style of our tests. ◦ Use this as a reference for the final framework, gherkin conventions, parameterization and guidelines on how to write a JUnit 5 Gherkin test. • unit-tests.md ◦ A more general document of unit testing in the project describing how to run tests, how to write readable tests, naming conventions etc. ◦ Use this as a reference for how to write tests, e.g. how to mock use cases, which assertion framework to use, how to handle Observables.