Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Kotlin at Scale — Spek #KotlinConf2017

Testing Kotlin at Scale — Spek #KotlinConf2017

Advanced talk about why and how we migrated thousands of tests from JUnit to Spek and how it helps us write even more tests while keeping them concise and maintainable.

Talk covers testing for both JVM and Android platforms.

Was presented at KotlinConf 2017.

Artem Zinnatullin

November 03, 2017
Tweet

More Decks by Artem Zinnatullin

Other Decks in Programming

Transcript

  1. @artem_zin
    Artem Zinnatullin
    Testing Kotlin at
    Scale: Spek

    View Slide

  2. - Productivity

    View Slide

  3. - Productivity
    - Reviewability

    View Slide

  4. - Productivity
    - Reviewability
    - Maintainability

    View Slide

  5. - Patterns
    - Principles
    - OOP/FP
    - Common Sense

    View Slide

  6. View Slide

  7. But

    View Slide

  8. We focus on production
    code

    View Slide

  9. - Productivity
    - Reviewability
    - Maintainability
    - Developer Happiness






    When it comes to test code

    View Slide

  10. - Patterns
    - Principles
    - OOP/FP
    - Common Sense






    When it comes to test code

    View Slide

  11. Aren’t we being hypocrite to ourselves?

    View Slide

  12. Do we really understand how much
    Effort do we put in Tests?

    View Slide

  13. There is a simple
    metric though

    View Slide

  14. Tests can actually
    take more LOCs than
    production code

    View Slide

  15. RxJava (library):
    - $ cloc src/main — 82 K LOC
    November 2017
    https://github.com/ReactiveX/RxJava

    View Slide

  16. RxJava (library):
    - $ cloc src/main — 82 K LOC
    - $ cloc src/test — 159 K LOC
    November 2017
    https://github.com/ReactiveX/RxJava

    View Slide

  17. 159 / 82 = 1,94 x
    RxJava

    View Slide

  18. RxJava: 159 / 82 = 1,94 x
    OkHttp: 27 / 15 = 1,8 x

    View Slide

  19. RxJava: 159 / 82 = 1,94 x
    OkHttp: 27 / 15 = 1,8 x
    Retrofit: 4,9 / 2,8= 1,75 x

    View Slide

  20. !

    View Slide

  21. Is there something
    common between
    these projects?

    View Slide

  22. - Assertion libraries

    View Slide

  23. - Assertion libraries ✅

    View Slide

  24. - Assertion libraries ✅
    - Mocking libraries

    View Slide

  25. - Assertion libraries ✅
    - Mocking libraries ✅

    View Slide

  26. Test framework
    though?

    View Slide

  27. They all use JUnit (4)
    And it works

    View Slide

  28. JUnit 4 ❤
    - It’s robust

    View Slide

  29. JUnit 4 ❤
    - It’s robust
    - It’s straightforward

    View Slide

  30. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it

    View Slide

  31. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it

    View Slide

  32. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it
    - Everybody is familiar with it

    View Slide

  33. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it
    - Everybody is familiar with it
    - It’s a standard.

    View Slide

  34. But it has problems

    View Slide

  35. “Code Repetition”

    View Slide

  36. @Test
    fun updateCoarseLocationSync_Foregrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(true)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_FG)
    }
    JUnit 4: “Code Repetition” Problem

    View Slide

  37. @Test
    fun updateCoarseLocationSync_NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem

    View Slide

  38. @Test
    fun updateCoarseLocationSync_
    NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(
    false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem

    View Slide

  39. @Test
    fun updateCoarseLocationSync_NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem

    View Slide

  40. “Test case/context
    naming”

    View Slide

  41. @Test
    fun removeAccessTokenShouldStopLoggedinScopeAndStartLoggedOutScope() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit4: “Test case/context naming” Problem

    View Slide

  42. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    JUnit4: “Test case/context naming” Problem
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }

    View Slide

  43. @Test
    fun `remove access token should stop logged in
    scope and start logged out scope`()
    JUnit4: “Test case/context naming” Problem
    Still bad

    View Slide

  44. “What does this test
    test?”

    View Slide

  45. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  46. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  47. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    4 checks

    View Slide

  48. View Slide

  49. Some motivation speech here

    View Slide

  50. Spek

    View Slide

  51. Hadi Hariri @JetBrains
    Spek v0, 2012

    View Slide

  52. Hadi Hariri @JetBrains
    Spek v0, 2012

    View Slide

  53. Hadi Hariri @JetBrains
    Spek v0, 2012
    Kotlin 0.4.297

    View Slide

  54. Hadi Hariri @JetBrains
    Spek v0, 2012

    View Slide

  55. Hadi Hariri @JetBrains
    Spek v0, 2012

    View Slide

  56. 2016

    View Slide

  57. Spek v1.0.25, 2016
    We started to feel JUnit 4 problems really badly

    View Slide

  58. Spek v1.0.25, 2016

    View Slide

  59. Spek v1.0.25, 2016

    View Slide

  60. Spek v1.0.25, 2016
    We started using it

    View Slide

  61. Spek v1.0.25, 2016

    View Slide

  62. Spek v1.0.25, 2016

    View Slide

  63. View Slide

  64. View Slide

  65. I see what you did there, Hadi

    View Slide

  66. Spek 2.x

    View Slide

  67. : BetterTestingFuture
    Spek 2.x

    View Slide

  68. class SuperTypicalJUnitTest {
    val calculator = Calculator()
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    }
    Typical Spec

    View Slide

  69. class CalculatorSpec : Spek({
    val calculator by memoized { Calculator() }
    context("2 + 4") {
    val result by memoized { calculator.add(2, 4) }
    it("equals 6") {
    assertEquals(6, result)
    }
    }
    })
    Typical Spec

    View Slide

  70. Spek: Basic API

    View Slide

  71. context(“2 + 4") {
    }x
    Spek: Basic API

    View Slide

  72. context("2 + 4") {
    }x
    describe(“2 + 4") {
    }x
    Spek: Basic API

    View Slide

  73. context("2 + 4") {
    }x
    describe("2 + 4") {
    }x
    given(“2 + 4") {
    }x
    Spek: Basic API

    View Slide

  74. describe("2 + 4") {
    }x
    context("2 + 4") {
    }x
    given("2 + 4") {
    }x
    group(“2 + 4")
    Spek: Basic API

    View Slide

  75. group("") ~= Test class in JUnit
    Spek: Basic API

    View Slide

  76. group("") ~= Test class in JUnit
    Spek: Basic API
    You can nest groups naturally

    View Slide

  77. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    Spek: Basic API

    View Slide

  78. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API

    View Slide

  79. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API

    View Slide

  80. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API

    View Slide

  81. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API

    View Slide

  82. Spek: Basic API
    it("") = @Test in JUnit

    View Slide

  83. it("") = @Test in JUnit
    Spek: Basic API
    You can have as many `it` in a `group` as needed

    View Slide

  84. Spek: Basic API
    Groups and Tests create natural structure
    that scales very well

    View Slide

  85. val calculator by memoized { Calculator() }
    Spek: Basic API

    View Slide

  86. Spek: Basic API
    You avoid state sharing betwen tests with `memoized`

    View Slide

  87. Let’s rewrite real JUnit test with Spek

    View Slide

  88. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    4 checks

    View Slide

  89. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  90. context("remove access token") {
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  91. context("remove access token") {
    beforeEachTest {
    accessTokenRepository.removeAccessToken()
    }
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  92. context("remove access token") {
    beforeEachTest {
    accessTokenRepository.removeAccessToken()
    }
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  93. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  94. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  95. describe("first scope change") {
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  96. describe("first scope change") {
    val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  97. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem

    View Slide

  98. describe("first scope change") {
    val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
    it("is 'logged in' scope") {
    assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN)
    }
    it("is not started") {
    assertThat(firstScopeChange.started).isFalse()
    }
    }
    Let’s rewrite real JUnit test with Spek

    View Slide

  99. class JUnitTest {
    private var scopeManager = MockScopeManager()
    private val accessTokenRepository = AccessTokenRepository(
    RuntimeEnvironment.application,
    scopeManager
    )
    lateinit var scopeChange1: MockScopeManager.ScopeChange
    lateinit var scopeChange2: MockScopeManager.ScopeChange
    @Before
    fun `remove access token`() {
    accessTokenRepository.removeAccessToken()
    scopeChange1 = scopeManager.scopeChanges[0]
    scopeChange2 = scopeManager.scopeChanges[1]
    }
    @Test
    fun `first scope change is 'logged in' scope`() {
    assertThat(scopeChange1.scope).isEqualTo(LOGGED_IN)
    }
    @Test
    fun `first scope change is not started`() {
    assertThat(scopeChange1.started).isFalse()
    }
    @Test
    fun `second scope change is 'logged out' scope`() {
    assertThat(scopeChange2.scope).isEqualTo(LOGGED_OUT)
    }
    @Test
    fun `first scope change is started`() {
    assertThat(scopeChange2.started).isTrue()
    }
    }
    Let’s rewrite real JUnit test with Spek
    JUnit is not structured, it’s flat

    View Slide

  100. class Spec : Spek({
    val scopeManager by memoized { MockScopeManager() }
    val accessTokenRepository by memoized {
    AccessTokenRepository(RuntimeEnvironment.application, scopeManager)
    }
    context("remove access token") {
    beforeEachTest {
    accessTokenRepository.removeAccessToken()
    }
    describe("first scope change") {
    val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
    it("is 'logged in' scope") {
    assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN)
    }
    it("is not started") {
    assertThat(firstScopeChange.started).isFalse()
    }
    }
    describe("second scope change") {
    val secondScopeChange by memoized { scopeManager.scopeChanges[1] }
    it("is 'logged out' scope") {
    assertThat(secondScopeChange.scope).isEqualTo(LOGGED_OUT)
    }
    it("is started") {
    assertThat(secondScopeChange.started).isTrue()
    }
    }
    }
    })
    Let’s rewrite real JUnit test with Spek
    Spek has structure

    View Slide

  101. Test code can be structured
    with Spek

    View Slide

  102. Let’s rewrite real JUnit test with Spek
    Hierarchical Report in IntelliJ

    View Slide

  103. The missing documentation & samples
    Spek Tips

    View Slide

  104. Spek Tips
    You can iterate stuff

    View Slide

  105. Spek Tips
    Iterate things, Spek is just a code
    class StringExtensionsSpec : Spek({
    listOf(null, "", " ", "\t").forEach { string ->
    context("null or blank string '$string'") {
    val isNullOrBlank by memoized { string.isNullOrBlank() }
    it("is indeed null or blank") {
    assertThat(isNullOrBlank, equalTo(true))
    }
    }
    }
    })

    View Slide

  106. when (spek) ?

    View Slide

  107. Hopefully
    Spek 2.x release plans
    Developer Preview by the end of November

    View Slide

  108. Most likely
    Spek 2.x release plans
    ~month is exactly enough to get sick of how we typically write tests

    View Slide

  109. Spek 2.x release plans
    github.com/spekframework/spek

    View Slide

  110. Spek 2.x release plans
    github.com/spekframework/spek
    twitter.com/artem_zin

    View Slide

  111. Spek 2.x release plans
    github.com/spekframework/spek
    twitter.com/artem_zin

    View Slide

  112. Spek 2.x release plans
    github.com/spekframework/spek
    twitter.com/artem_zin
    End of November

    View Slide

  113. #kotlinconf17
    @artem_zin
    Artem Zinnatullin
    Thank you!
    @hhariri
    Hadi Hariri
    @raniejade
    Ranie Jade Ramiso
    @arturdryomov
    Artur Dryomov

    View Slide