Unit Testing: From Spek 2 to JUnit 5

Unit Testing: From Spek 2 to JUnit 5

The slides are from a talk I gave internally at Babylon Health. In this talk, I describe our strategy for migrating our unit tests on Android from Spek 2 to JUnit 5, detailing our reasons for migrating as well as the migration process itself and our thoughts along the way.

3502084525280a4cf1c12041c53f64db?s=128

Darren Atherton

August 24, 2020
Tweet

Transcript

  1. Unit Testing: From Spek 2 to JUnit 5 Darren Atherton

  2. Solving a Problem

  3. Why Not Spek 2? • Performance ◦ IDE hanging completely

    for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
  4. Why Not Spek 2? • Performance ◦ IDE hanging completely

    for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
  5. Why Not Spek 2? • Performance ◦ IDE hanging completely

    for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
  6. Why Not Spek 2? • Performance ◦ IDE hanging completely

    for some developers ▪ Long files causing the most issues, potentially due to large nested lambdas ◦ Very slow autocompletion in the editor ▪ Lowers productivity and increases feedback loop ◦ Slow execution of Spek tests compared to JUnit ▪ Basic benchmarking performed ▪ More of an added benefit of JUnit than a deciding factor
  7. Why Not Spek 2? // assertion failed! • Problems running

    single tests • Problems running any tests
  8. Why Not Spek 2? // assertion failed! • Problems running

    single tests • Problems running any tests
  9. Why Not Spek 2? // assertion failed! • Problems running

    single tests • Problems running any tests
  10. Why Not Spek 2? • Timeout and flakiness issues //

    assertions failed!
  11. Why Not Spek 2? • Memoization issues ◦ Hard to

    understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
  12. Why Not Spek 2? • Memoization issues ◦ Hard to

    understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
  13. Why Not Spek 2? • Memoization issues ◦ Hard to

    understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
  14. Why Not Spek 2? • Memoization issues ◦ Hard to

    understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
  15. Why Not Spek 2? • Memoization issues ◦ Hard to

    understand when to use each caching mode - especially when onboarding new devs ◦ Inconsistency between where developers apply the memoized fields (plus large team size) ◦ Test code was not deliberate/WYSIWYG ◦ Documentation was lacking
  16. Why Not Spek 2? • lateinit var fields • Nullable

    fields Introduces subtle initialisation bugs because fields are not immutable Adds noise to the test as we must deal with the nullable type throughout the test
  17. Why Not Plain JUnit? • Lack of structure ◦ No

    consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
  18. Why Not Plain JUnit? • Lack of structure ◦ No

    consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
  19. Why Not Plain JUnit? • Lack of structure ◦ No

    consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
  20. Why Not Plain JUnit? • Lack of structure ◦ No

    consistency between all tests in the codebase • Gherkin style only enforceable through comments ◦ Gherkin style allows you to mentally map out intended behaviour • But it is fast and simple!
  21. Why Not JUnit 4? • JUnit 5: ◦ Has better

    support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
  22. Why Not JUnit 4? • JUnit 5: ◦ Has better

    support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
  23. Why Not JUnit 4? • JUnit 5: ◦ Has better

    support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
  24. Why Not JUnit 4? • JUnit 5: ◦ Has better

    support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
  25. Why Not JUnit 4? • JUnit 5: ◦ Has better

    support for function-level parameterized tests ◦ Modern and future-proof ◦ Extensible • Differences not as important as the difference between JUnit and Spek Source: JUnit 4 Parameterized Javadoc
  26. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  27. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  28. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  29. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  30. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  31. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  32. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  33. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  34. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  35. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  36. Influences & Objectives of the Framework • Influences ◦ Responses

    to the developer surveys sent out to the team ◦ More focused meetings with smaller groups ◦ Actual problems we were having with existing approach (Spek) ▪ Flakiness ▪ IDE performance ▪ Execution performance ▪ Problems running tests ▪ Memoization ▪ lateinit var/nullable fields • Objectives of new framework ◦ Structured way of writing tests ◦ Stable (especially when running on CI) ◦ Better IDE performance ◦ Straightforward to read and understand (no out-of-order block execution) ◦ Simple memoization model - e.g. new instance of fields at the class level for each test run ◦ Ability to debug ◦ Fast
  37. Implementing the Framework

  38. Start With a Plan 1. Create a proof-of-concept/proposal PR with:

    a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint checks and other in-code checks to enforce structure/consistency
  39. An Iterative Approach • 9 files changes: ◦ 3 Gradle

    files updated ◦ 1 Spek test removed ◦ 1 JUnit proof-of-concept test added ◦ 1 structure documentation added ◦ 1 background documentation added ◦ 1 Framework DSL test class added ◦ 1 Framework DSL added …and many zoom calls
  40. Framework DSL - Objectives • How can we create the

    framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  41. Framework DSL - Enforcing Structure • How can we create

    the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  42. Framework DSL - Fast Error Checking • How can we

    create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  43. Framework DSL - Shallow Nesting • How can we create

    the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // 5+ nested lambdas deep // 1 large (1000+ LoC) lambda
  44. Framework DSL - Ease of Debugging • How can we

    create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // WYSIWYG // Spek tests don’t run linearly, // but instead run in two // separate phases - discovery // and execution // JUnit tests run in the order // as you can see in the code and // do not timeout
  45. Framework DSL - Readability via Grouping • How can we

    create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy 1 2 3 4 1. Inputs/outputs 2. Given/And 3. When/And 4. Then/And
  46. Framework DSL - Readability via Indentation • How can we

    create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  47. Framework DSL - Removing ‘lateinit var’ • How can we

    create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy // “Contracts allow a function to explicitly describe its // behavior in a way which is understood by the compiler” // If we know that our ‘Given’ block will be called // exactly once, then the compiler does not require us to // use the ‘lateinit var’ workaround.
  48. Framework DSL - Disabling Nesting of Gherkin Blocks • How

    can we create the framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  49. Framework DSL - Parameterization • How can we create the

    framework (using type-safe builders) to: ◦ Add structure to our tests ◦ Enforce fast error checking (such as no empty descriptions) ◦ Ensure IDE performance (shallow nesting of lambdas) ◦ Ensure ease of debugging (no separate discovery and execution phases) ◦ Ensure tests are easy to read/parse ◦ Remove the need for ‘lateinit var’ ◦ Make it impossible to incorrectly nest Gherkin blocks ◦ Make parameterization easy
  50. Parameterization - Default Argument Sources • @NullSource - Useful for

    checking if a function supports a single null parameter. • @EmptySource - Pass in an empty List, Set, Map or Array • @ValueSource - Basic lists of short, byte, int, long, float, double, char, boolean, String or Class. ◦ @ValueSource(longs = { 4, 8, 15, 16, 23 }) • @EnumSource - Enums (can filter a subset of the Enums values). ◦ @EnumSource(DeepLinkHost::class) • @CsvSource - Slightly more complex rows/columns of data. ◦ @CsvSource({ "Welcome, 1", "to, 2", "JUnit, 5" }) • @MethodSource - Construct more complex streams of arguments. ◦ @MethodSource("createAppointmentTestCases")
  51. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  52. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  53. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  54. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  55. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  56. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  57. Parameterization - Supplying Complex Data • @MethodSource method must be

    static, so in Kotlin: ◦ Top-level or in a companion object ◦ Hard to directly pass in existing DataProvider data from the codebase ◦ Far away from the test! • @MethodSource method reference is a String ◦ No ‘clickable’ reference ◦ No autocomplete ◦ Easy to mistype
  58. Parameterized Tests - Supplying Complex Lists • Supports List, Queue

    and Set or a variable number of some type. • Primary constructor is useful for pre-existing DataProvider functions. • If your data is a simple/static list, just use @ValueSource.
  59. Parameterized Tests - Supplying Complex Lists • For more complex

    data (e.g. where each row has 3+ values), we recommend to create a data class (in this case called ‘TestCase’) and subclass the ArgumentsListProvider.
  60. Parameterized Tests - Supplying Complex Pairs/Maps • Supports Map or

    a variable number of Pair. • Primary constructor is useful for pre-existing DataProvider functions. • Useful for passing in pairs of input and an expected result - e.g. mapper tests.
  61. Parameterized Tests - Supplying Complex Pairs/Maps

  62. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦

    Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
  63. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦

    Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
  64. Repeating Tests & Non-Determinism • Useful for two scenarios: ◦

    Can quickly add a large number such as 200 to quickly check for flakiness before committing your test. ◦ Can use a small number such as 5 in combination with fixtures for your input to create a test which also reduces the chance of code being flaky and may catch random code errors.
  65. JUnit Framework: The Lost Levels

  66. Parameterization - A First Attempt (Dynamic Tests) // expects function

    to return a collection of DynamicTest // “If you access fields from the test instance // within a lambda expression for a dynamic test, // those fields will not be reset by callback // methods or extensions between the execution of // individual dynamic tests generated by the same // @TestFactory method.”
  67. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’

    in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
  68. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’

    in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
  69. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’

    in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
  70. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’

    in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
  71. Adding (and Removing) Coroutine Support • No need for ‘runBlocking’

    in any calling code • Can override the CoroutineDispatcher ◦ Solves the problem of the few and not the majority • Slower performance overall • Guiding principle of the framework was lightweight and simple - not doing too much
  72. Migrating the Tests

  73. Migration Rules Spek 2 JUnit 5 Gherkin DSL Feature (meaning

    Spek files with multiple features are split into multiple JUnit files) Class Scenario @Test function forEach loop @ParameterizedTest lateinit var val private val s by memoized { “s” } private val s = “s” beforeEachTest { } Define vals at top of function or class
  74. Migration Rules Two reasons to have a well-defined migration model

    before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
  75. Migration Rules Two reasons to have a well-defined migration model

    before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
  76. Migration Rules Two reasons to have a well-defined migration model

    before migrating from one framework to another: 1. It allows us to write our automated migration script using Kotlin Program Structure Interface (PSI) and Regex in accordance to these rules. The rules in the table show how we want our test code to be translated from input to output. This assists us with validating the migration process. 2. When introducing a large change to a large team, it helps developers to create a mental model of how things work, meaning if we know how to write Spek 2 tests, then we are more likely to be able to write JUnit 5 tests easily by just translating one concept into another.
  77. Converting the Tests Spek conversion Spek conversion Feature maintenance that

    updates tests develop
  78. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  79. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  80. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  81. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  82. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  83. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  84. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  85. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  86. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  87. Converting the Tests • Objectives ◦ Keep time to review

    a migration PR low ◦ Don’t overwhelm the team with PRs ◦ Keep the feature teams moving ◦ Minimise merge conflicts ◦ Expose the team to the new testing conventions • Solutions ◦ Set a max amount of files for a single PR ◦ Provide a slow(er) but steady stream of PRs ◦ Perform only a 1-1 mapping between Spek and JUnit 5 ◦ Involve the whole team in the PR review process
  88. Enforcing Consistency and Detecting Errors

  89. Potential Checks - Lint • White space rule around block

    sections. ‘Input/Output’, ‘Given/And’, ‘When/And’ and ‘Then/And’ sections should be surrounded by one line of white space. • Ensure that we are not using the@MethodSource annotation for @ParameterizedTest tests. Instead, we should use our helper classes from the DSL to construct streams of input data. • No use of AssertJ ‘Assertions.assertThat’ without statically importing the function for readability • No empty Gherkin blocks
  90. Potential Checks - Framework Code • ‘Given’, ‘When’, ‘Then’ and

    ‘And’ descriptions cannot be blank. If a blank description is provided, the block throws an IllegalArgumentException (implemented). • No duplicate block descriptions. This prevents any copy/paste errors where two blocks (e.g. ‘Given’ and ‘And’ have the same description. This will be enforced at runtime in the Gherkin DSL class. • No blank space at the start or end of descriptions. • Capitalisation (or no capitalisation) of descriptions. • Order of ‘Given’, ‘When’, ‘Then’ and ‘And’ blocks must be correct. This could be enforced at runtime in the Gherkin DSL class by tracking the blocks being executed and checking the neighbours of each block. • Checking if there are no ‘Then’ blocks (usually means we are not asserting anything in our test)
  91. Framework Checks - Finding a Balance Framework consistency Developer experience

  92. Where Are We Now? 1. Create a proof-of-concept/proposal PR with:

    a. Gherkin DSL structure in java-test-helpers b. Proposal documentation c. Usage/guidelines documentation d. Unit testing background documentation e. At least one test as a proof of concept - a complex test! 2. Fix the top 10 immediately flaky Spek tests by hand to stabilise develop branch a. Accessible via Gradle Enterprise 3. Create a script to automate conversion of tests from Spek 2 to JUnit 5 4. Batch convert tests and open PRs 30 tests at a time 5. Create Lint and other in-code checks to enforce structure/consistency
  93. Documentation • unit-testing-background.md ◦ A full write-up of our requirements/goals,

    framework design, performance benchmarks and the different approaches considered with rationale for each. ◦ Read this when you want to know how the various strategies we discussed. • unit-test-structure.md ◦ The outcome of the background research containing the proposed framework and structure/style of our tests. ◦ Use this as a reference for the final framework, gherkin conventions, parameterization and guidelines on how to write a JUnit 5 Gherkin test. • unit-tests.md ◦ A more general document of unit testing in the project describing how to run tests, how to write readable tests, naming conventions etc. ◦ Use this as a reference for how to write tests, e.g. how to mock use cases, which assertion framework to use, how to handle Observables.
  94. Q&A