Testing Kotlin at Scale — Spek #KotlinConf2017

Advanced talk about why and how we migrated thousands of tests from JUnit to Spek and how it helps us write even more tests while keeping them concise and maintainable.

Talk covers testing for both JVM and Android platforms.

Was presented at KotlinConf 2017.

Transcript

1. @artem_zin
   Artem Zinnatullin
   Testing Kotlin at
   Scale: Spek
   

   View Slide

2. - Productivity
   

   View Slide

3. - Productivity
   - Reviewability
   

   View Slide

4. - Productivity
   - Reviewability
   - Maintainability
   

   View Slide

5. - Patterns
   - Principles
   - OOP/FP
   - Common Sense
   

   View Slide

6. View Slide

7. But
   

   View Slide

8. We focus on production
   code
   

   View Slide

9. - Productivity
   - Reviewability
   - Maintainability
   - Developer Happiness
   
   
   
   
   
   
   When it comes to test code
   

   View Slide

10. - Patterns
    - Principles
    - OOP/FP
    - Common Sense
    
    
    
    
    
    
    When it comes to test code
    

    View Slide

11. Aren’t we being hypocrite to ourselves?
    

    View Slide

12. Do we really understand how much
    Effort do we put in Tests?
    

    View Slide

13. There is a simple
    metric though
    

    View Slide

14. Tests can actually
    take more LOCs than
    production code
    

    View Slide

15. RxJava (library):
    - $ cloc src/main — 82 K LOC
    November 2017
    https://github.com/ReactiveX/RxJava
    

    View Slide

16. RxJava (library):
    - $ cloc src/main — 82 K LOC
    - $ cloc src/test — 159 K LOC
    November 2017
    https://github.com/ReactiveX/RxJava
    

    View Slide

17. 159 / 82 = 1,94 x
    RxJava
    

    View Slide

18. RxJava: 159 / 82 = 1,94 x
    OkHttp: 27 / 15 = 1,8 x
    

    View Slide

19. RxJava: 159 / 82 = 1,94 x
    OkHttp: 27 / 15 = 1,8 x
    Retrofit: 4,9 / 2,8= 1,75 x
    

    View Slide

20. !
    
    

    View Slide

21. Is there something
    common between
    these projects?
    

    View Slide

22. - Assertion libraries
    

    View Slide

23. - Assertion libraries ✅
    

    View Slide

24. - Assertion libraries ✅
    - Mocking libraries
    

    View Slide

25. - Assertion libraries ✅
    - Mocking libraries ✅
    

    View Slide

26. Test framework
    though?
    

    View Slide

27. They all use JUnit (4)
    And it works
    

    View Slide

28. JUnit 4 ❤
    - It’s robust
    

    View Slide

29. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    

    View Slide

30. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    

    View Slide

31. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it
    

    View Slide

32. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it
    - Everybody is familiar with it
    

    View Slide

33. JUnit 4 ❤
    - It’s robust
    - It’s straightforward
    - You don’t have to debug it
    - All build systems and IDEs support it
    - Everybody is familiar with it
    - It’s a standard.
    

    View Slide

34. But it has problems
    

    View Slide

35. “Code Repetition”
    

    View Slide

36. @Test
    fun updateCoarseLocationSync_Foregrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(true)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_FG)
    }
    JUnit 4: “Code Repetition” Problem
    

    View Slide

37. @Test
    fun updateCoarseLocationSync_NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem
    

    View Slide

38. @Test
    fun updateCoarseLocationSync_
    NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(
    false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem
    

    View Slide

39. @Test
    fun updateCoarseLocationSync_NotForegrounded() {
    `when`(appForegroundDetector.isForegrounded).thenReturn(false)
    val first = AndroidLocationBuilder()
    .withProvider(AndroidLocation.Provider.FUSED)
    .withLat(0.0)
    .withLng(0.0)
    .withTime(1L)
    .build()
    locationIngestService.updateCoarseLocationSync(first)
    val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java)
    verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture())
    val value = argumentCaptor.value
    assertThat(value).isNotNull()
    assertThat(value.locations).isNotNull().hasSize(1)
    assertThat(value.locations[0]).isNotNull()
    assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG)
    }
    JUnit 4: “Code Repetition” Problem
    ⚠
    

    View Slide

40. “Test case/context
    naming”
    

    View Slide

41. @Test
    fun removeAccessTokenShouldStopLoggedinScopeAndStartLoggedOutScope() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit4: “Test case/context naming” Problem
    

    View Slide

42. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    JUnit4: “Test case/context naming” Problem
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    

    View Slide

43. @Test
    fun `remove access token should stop logged in
    scope and start logged out scope`()
    JUnit4: “Test case/context naming” Problem
    Still bad
    

    View Slide

44. “What does this test
    test?”
    

    View Slide

45. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

46. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

47. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    4 checks
    

    View Slide

48. View Slide

49. Some motivation speech here
    

    View Slide

50. Spek
    

    View Slide

51. Hadi Hariri @JetBrains
    Spek v0, 2012
    

    View Slide

52. Hadi Hariri @JetBrains
    Spek v0, 2012
    

    View Slide

53. Hadi Hariri @JetBrains
    Spek v0, 2012
    Kotlin 0.4.297
    

    View Slide

54. Hadi Hariri @JetBrains
    Spek v0, 2012
    

    View Slide

55. Hadi Hariri @JetBrains
    Spek v0, 2012
    

    View Slide

56. View Slide

57. Spek v1.0.25, 2016
    We started to feel JUnit 4 problems really badly
    

    View Slide

58. Spek v1.0.25, 2016
    

    View Slide

59. Spek v1.0.25, 2016
    

    View Slide

60. Spek v1.0.25, 2016
    We started using it
    

    View Slide

61. Spek v1.0.25, 2016
    

    View Slide

62. Spek v1.0.25, 2016
    

    View Slide

63. View Slide

64. View Slide

65. I see what you did there, Hadi
    

    View Slide

66. Spek 2.x
    

    View Slide

67. : BetterTestingFuture
    Spek 2.x
    

    View Slide

68. class SuperTypicalJUnitTest {
    val calculator = Calculator()
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    }
    Typical Spec
    

    View Slide

69. class CalculatorSpec : Spek({
    val calculator by memoized { Calculator() }
    context("2 + 4") {
    val result by memoized { calculator.add(2, 4) }
    it("equals 6") {
    assertEquals(6, result)
    }
    }
    })
    Typical Spec
    

    View Slide

70. Spek: Basic API
    

    View Slide

71. context(“2 + 4") {
    }x
    Spek: Basic API
    

    View Slide

72. context("2 + 4") {
    }x
    describe(“2 + 4") {
    }x
    Spek: Basic API
    

    View Slide

73. context("2 + 4") {
    }x
    describe("2 + 4") {
    }x
    given(“2 + 4") {
    }x
    Spek: Basic API
    

    View Slide

74. describe("2 + 4") {
    }x
    context("2 + 4") {
    }x
    given("2 + 4") {
    }x
    group(“2 + 4")
    Spek: Basic API
    

    View Slide

75. group("") ~= Test class in JUnit
    Spek: Basic API
    

    View Slide

76. group("") ~= Test class in JUnit
    Spek: Basic API
    You can nest groups naturally
    

    View Slide

77. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    Spek: Basic API
    

    View Slide

78. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API
    

    View Slide

79. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API
    

    View Slide

80. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API
    

    View Slide

81. it("equals 6") {
    assertThat(result).isEqualTo(6)
    }
    @Test
    fun `2 + 4 = 6`() {
    val result = calculator.add(2, 4)
    assertEquals(6, result)
    }
    Spek: Basic API
    

    View Slide

82. Spek: Basic API
    it("") = @Test in JUnit
    

    View Slide

83. it("") = @Test in JUnit
    Spek: Basic API
    You can have as many `it` in a `group` as needed
    

    View Slide

84. Spek: Basic API
    Groups and Tests create natural structure
    that scales very well
    

    View Slide

85. val calculator by memoized { Calculator() }
    Spek: Basic API
    

    View Slide

86. Spek: Basic API
    You avoid state sharing betwen tests with `memoized`
    

    View Slide

87. Let’s rewrite real JUnit test with Spek
    

    View Slide

88. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    4 checks
    

    View Slide

89. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

90. context("remove access token") {
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

91. context("remove access token") {
    beforeEachTest {
    accessTokenRepository.removeAccessToken()
    }
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

92. context("remove access token") {
    beforeEachTest {
    accessTokenRepository.removeAccessToken()
    }
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

93. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

94. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

95. describe("first scope change") {
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

96. describe("first scope change") {
    val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

97. @Test
    fun `remove access token should stop logged in scope and start logged out scope`() {
    accessTokenRepository.removeAccessToken()
    val scopeChange1 = scopeManager.scopeChanges[0]
    assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)
    assertThat(scopeChange1.started).isFalse()
    val scopeChange2 = scopeManager.scopeChanges[1]
    assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT)
    assertThat(scopeChange2.started).isTrue()
    }
    JUnit 4: “What does this test test” Problem
    

    View Slide

98. describe("first scope change") {
    val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
    it("is 'logged in' scope") {
    assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN)
    }
    it("is not started") {
    assertThat(firstScopeChange.started).isFalse()
    }
    }
    Let’s rewrite real JUnit test with Spek
    

    View Slide

99. class JUnitTest {
    private var scopeManager = MockScopeManager()
    private val accessTokenRepository = AccessTokenRepository(
    RuntimeEnvironment.application,
    scopeManager
    )
    lateinit var scopeChange1: MockScopeManager.ScopeChange
    lateinit var scopeChange2: MockScopeManager.ScopeChange
    @Before
    fun `remove access token`() {
    accessTokenRepository.removeAccessToken()
    scopeChange1 = scopeManager.scopeChanges[0]
    scopeChange2 = scopeManager.scopeChanges[1]
    }
    @Test
    fun `first scope change is 'logged in' scope`() {
    assertThat(scopeChange1.scope).isEqualTo(LOGGED_IN)
    }
    @Test
    fun `first scope change is not started`() {
    assertThat(scopeChange1.started).isFalse()
    }
    @Test
    fun `second scope change is 'logged out' scope`() {
    assertThat(scopeChange2.scope).isEqualTo(LOGGED_OUT)
    }
    @Test
    fun `first scope change is started`() {
    assertThat(scopeChange2.started).isTrue()
    }
    }
    Let’s rewrite real JUnit test with Spek
    JUnit is not structured, it’s flat
    

    View Slide

100. class Spec : Spek({
     val scopeManager by memoized { MockScopeManager() }
     val accessTokenRepository by memoized {
     AccessTokenRepository(RuntimeEnvironment.application, scopeManager)
     }
     context("remove access token") {
     beforeEachTest {
     accessTokenRepository.removeAccessToken()
     }
     describe("first scope change") {
     val firstScopeChange by memoized { scopeManager.scopeChanges[0] }
     it("is 'logged in' scope") {
     assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN)
     }
     it("is not started") {
     assertThat(firstScopeChange.started).isFalse()
     }
     }
     describe("second scope change") {
     val secondScopeChange by memoized { scopeManager.scopeChanges[1] }
     it("is 'logged out' scope") {
     assertThat(secondScopeChange.scope).isEqualTo(LOGGED_OUT)
     }
     it("is started") {
     assertThat(secondScopeChange.started).isTrue()
     }
     }
     }
     })
     Let’s rewrite real JUnit test with Spek
     Spek has structure
     

     View Slide

101. Test code can be structured
     with Spek
     

     View Slide

102. Let’s rewrite real JUnit test with Spek
     Hierarchical Report in IntelliJ
     

     View Slide

103. The missing documentation & samples
     Spek Tips
     

     View Slide

104. Spek Tips
     You can iterate stuff
     

     View Slide

105. Spek Tips
     Iterate things, Spek is just a code
     class StringExtensionsSpec : Spek({
     listOf(null, "", " ", "\t").forEach { string ->
     context("null or blank string '$string'") {
     val isNullOrBlank by memoized { string.isNullOrBlank() }
     it("is indeed null or blank") {
     assertThat(isNullOrBlank, equalTo(true))
     }
     }
     }
     })
     

     View Slide

106. when (spek) ?
     

     View Slide

107. Hopefully
     Spek 2.x release plans
     Developer Preview by the end of November
     

     View Slide

108. Most likely
     Spek 2.x release plans
     ~month is exactly enough to get sick of how we typically write tests
     

     View Slide

109. Spek 2.x release plans
     github.com/spekframework/spek
     

     View Slide

110. Spek 2.x release plans
     github.com/spekframework/spek
     twitter.com/artem_zin
     

     View Slide

111. Spek 2.x release plans
     github.com/spekframework/spek
     twitter.com/artem_zin
     

     View Slide

112. Spek 2.x release plans
     github.com/spekframework/spek
     twitter.com/artem_zin
     End of November
     

     View Slide

113. #kotlinconf17
     @artem_zin
     Artem Zinnatullin
     Thank you!
     @hhariri
     Hadi Hariri
     @raniejade
     Ranie Jade Ramiso
     @arturdryomov
     Artur Dryomov
     

     View Slide