Testing Kotlin at Scale — Spek #KotlinConf2017

Advanced talk about why and how we migrated thousands of tests from JUnit to Spek and how it helps us write even more tests while keeping them concise and maintainable.

Talk covers testing for both JVM and Android platforms.

Was presented at KotlinConf 2017.

Transcript

1. @artem_zin Artem Zinnatullin Testing Kotlin at Scale: Spek
2. - Productivity
3. - Productivity - Reviewability
4. - Productivity - Reviewability - Maintainability
5. - Patterns - Principles - OOP/FP - Common Sense
7. But
8. We focus on production code
9. - Productivity - Reviewability - Maintainability - Developer Happiness When it comes to test code
10. - Patterns - Principles - OOP/FP - Common Sense When it comes to test code
11. Aren’t we being hypocrite to ourselves?
12. Do we really understand how much Effort do we put in Tests?
13. There is a simple metric though
14. Tests can actually take more LOCs than production code
15. RxJava (library): - $ cloc src/main — 82 K LOC November 2017 https://github.com/ReactiveX/RxJava
16. RxJava (library): - $ cloc src/main — 82 K LOC - $ cloc src/test — 159 K LOC November 2017 https://github.com/ReactiveX/RxJava
17. 159 / 82 = 1,94 x RxJava
18. RxJava: 159 / 82 = 1,94 x OkHttp: 27 / 15 = 1,8 x
19. RxJava: 159 / 82 = 1,94 x OkHttp: 27 / 15 = 1,8 x Retrofit: 4,9 / 2,8= 1,75 x
20. !
21. Is there something common between these projects?
22. - Assertion libraries
23. - Assertion libraries ✅
24. - Assertion libraries ✅ - Mocking libraries
25. - Assertion libraries ✅ - Mocking libraries ✅
26. Test framework though?
27. They all use JUnit (4) And it works
28. JUnit 4 ❤ - It’s robust
29. JUnit 4 ❤ - It’s robust - It’s straightforward
30. JUnit 4 ❤ - It’s robust - It’s straightforward - You don’t have to debug it
31. JUnit 4 ❤ - It’s robust - It’s straightforward - You don’t have to debug it - All build systems and IDEs support it
32. JUnit 4 ❤ - It’s robust - It’s straightforward - You don’t have to debug it - All build systems and IDEs support it - Everybody is familiar with it
33. JUnit 4 ❤ - It’s robust - It’s straightforward - You don’t have to debug it - All build systems and IDEs support it - Everybody is familiar with it - It’s a standard.
34. But it has problems
35. “Code Repetition”
36. @Test fun updateCoarseLocationSync_Foregrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(true) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED) .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_FG) } JUnit 4: “Code Repetition” Problem
37. @Test fun updateCoarseLocationSync_NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(false) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED) .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem
38. @Test fun updateCoarseLocationSync_ NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn( false) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED) .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem
39. @Test fun updateCoarseLocationSync_NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(false) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED) .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem ⚠
40. “Test case/context naming”
41. @Test fun removeAccessTokenShouldStopLoggedinScopeAndStartLoggedOutScope() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit4: “Test case/context naming” Problem
42. @Test fun `remove access token should stop logged in scope and start logged out scope`() { JUnit4: “Test case/context naming” Problem accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() }
43. @Test fun `remove access token should stop logged in scope and start logged out scope`() JUnit4: “Test case/context naming” Problem Still bad
44. “What does this test test?”
45. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
46. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
47. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem 4 checks
49. Some motivation speech here
50. Spek
51. Hadi Hariri @JetBrains Spek v0, 2012
52. Hadi Hariri @JetBrains Spek v0, 2012
53. Hadi Hariri @JetBrains Spek v0, 2012 Kotlin 0.4.297
54. Hadi Hariri @JetBrains Spek v0, 2012
55. Hadi Hariri @JetBrains Spek v0, 2012
57. Spek v1.0.25, 2016 We started to feel JUnit 4 problems really badly
58. Spek v1.0.25, 2016
59. Spek v1.0.25, 2016
60. Spek v1.0.25, 2016 We started using it
61. Spek v1.0.25, 2016
62. Spek v1.0.25, 2016
65. I see what you did there, Hadi
66. Spek 2.x
67. : BetterTestingFuture Spek 2.x
68. class SuperTypicalJUnitTest { val calculator = Calculator() @Test fun `2 + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } } Typical Spec
69. class CalculatorSpec : Spek({ val calculator by memoized { Calculator() } context("2 + 4") { val result by memoized { calculator.add(2, 4) } it("equals 6") { assertEquals(6, result) } } }) Typical Spec
70. Spek: Basic API
71. context(“2 + 4") { }x Spek: Basic API
72. context("2 + 4") { }x describe(“2 + 4") { }x Spek: Basic API
73. context("2 + 4") { }x describe("2 + 4") { }x given(“2 + 4") { }x Spek: Basic API
74. describe("2 + 4") { }x context("2 + 4") { }x given("2 + 4") { }x group(“2 + 4") Spek: Basic API
75. group("") ~= Test class in JUnit Spek: Basic API
76. group("") ~= Test class in JUnit Spek: Basic API You can nest groups naturally
77. it("equals 6") { assertThat(result).isEqualTo(6) } Spek: Basic API
78. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
79. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
80. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
81. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
82. Spek: Basic API it("") = @Test in JUnit
83. it("") = @Test in JUnit Spek: Basic API You can have as many `it` in a `group` as needed
84. Spek: Basic API Groups and Tests create natural structure that scales very well
85. val calculator by memoized { Calculator() } Spek: Basic API
86. Spek: Basic API You avoid state sharing betwen tests with `memoized`
87. Let’s rewrite real JUnit test with Spek
88. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem 4 checks
89. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
90. context("remove access token") { } Let’s rewrite real JUnit test with Spek
91. context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } } Let’s rewrite real JUnit test with Spek
92. context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } } Let’s rewrite real JUnit test with Spek
93. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
94. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
95. describe("first scope change") { } Let’s rewrite real JUnit test with Spek
96. describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0] } } Let’s rewrite real JUnit test with Spek
97. @Test fun `remove access token should stop logged in scope and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
98. describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0] } it("is 'logged in' scope") { assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN) } it("is not started") { assertThat(firstScopeChange.started).isFalse() } } Let’s rewrite real JUnit test with Spek
99. class JUnitTest { private var scopeManager = MockScopeManager() private val accessTokenRepository = AccessTokenRepository( RuntimeEnvironment.application, scopeManager ) lateinit var scopeChange1: MockScopeManager.ScopeChange lateinit var scopeChange2: MockScopeManager.ScopeChange @Before fun `remove access token`() { accessTokenRepository.removeAccessToken() scopeChange1 = scopeManager.scopeChanges[0] scopeChange2 = scopeManager.scopeChanges[1] } @Test fun `first scope change is 'logged in' scope`() { assertThat(scopeChange1.scope).isEqualTo(LOGGED_IN) } @Test fun `first scope change is not started`() { assertThat(scopeChange1.started).isFalse() } @Test fun `second scope change is 'logged out' scope`() { assertThat(scopeChange2.scope).isEqualTo(LOGGED_OUT) } @Test fun `first scope change is started`() { assertThat(scopeChange2.started).isTrue() } } Let’s rewrite real JUnit test with Spek JUnit is not structured, it’s flat
100. class Spec : Spek({ val scopeManager by memoized { MockScopeManager() } val accessTokenRepository by memoized { AccessTokenRepository(RuntimeEnvironment.application, scopeManager) } context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0] } it("is 'logged in' scope") { assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN) } it("is not started") { assertThat(firstScopeChange.started).isFalse() } } describe("second scope change") { val secondScopeChange by memoized { scopeManager.scopeChanges[1] } it("is 'logged out' scope") { assertThat(secondScopeChange.scope).isEqualTo(LOGGED_OUT) } it("is started") { assertThat(secondScopeChange.started).isTrue() } } } }) Let’s rewrite real JUnit test with Spek Spek has structure
101. Test code can be structured with Spek
102. Let’s rewrite real JUnit test with Spek Hierarchical Report in IntelliJ
103. The missing documentation & samples Spek Tips
104. Spek Tips You can iterate stuff
105. Spek Tips Iterate things, Spek is just a code class StringExtensionsSpec : Spek({ listOf(null, "", " ", "\t").forEach { string -> context("null or blank string '$string'") { val isNullOrBlank by memoized { string.isNullOrBlank() } it("is indeed null or blank") { assertThat(isNullOrBlank, equalTo(true)) } } } })
106. when (spek) ?
107. Hopefully Spek 2.x release plans Developer Preview by the end of November
108. Most likely Spek 2.x release plans ~month is exactly enough to get sick of how we typically write tests
109. Spek 2.x release plans github.com/spekframework/spek
110. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin
111. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin
112. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin End of November
113. #kotlinconf17 @artem_zin Artem Zinnatullin Thank you! @hhariri Hadi Hariri @raniejade Ranie Jade Ramiso @arturdryomov Artur Dryomov