Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Kotlin at Scale — Spek #KotlinConf2017

Testing Kotlin at Scale — Spek #KotlinConf2017

Advanced talk about why and how we migrated thousands of tests from JUnit to Spek and how it helps us write even more tests while keeping them concise and maintainable.

Talk covers testing for both JVM and Android platforms.

Was presented at KotlinConf 2017.

1669655d7494301079bbae17c15675b7?s=128

Artem Zinnatullin

November 03, 2017
Tweet

Transcript

  1. @artem_zin Artem Zinnatullin Testing Kotlin at Scale: Spek

  2. - Productivity

  3. - Productivity - Reviewability

  4. - Productivity - Reviewability - Maintainability

  5. - Patterns - Principles - OOP/FP - Common Sense

  6. None
  7. But

  8. We focus on production code

  9. - Productivity - Reviewability - Maintainability - Developer Happiness When

    it comes to test code
  10. - Patterns - Principles - OOP/FP - Common Sense When

    it comes to test code
  11. Aren’t we being hypocrite to ourselves?

  12. Do we really understand how much Effort do we put

    in Tests?
  13. There is a simple metric though

  14. Tests can actually take more LOCs than production code

  15. RxJava (library): - $ cloc src/main — 82 K LOC

    November 2017 https://github.com/ReactiveX/RxJava
  16. RxJava (library): - $ cloc src/main — 82 K LOC

    - $ cloc src/test — 159 K LOC November 2017 https://github.com/ReactiveX/RxJava
  17. 159 / 82 = 1,94 x RxJava

  18. RxJava: 159 / 82 = 1,94 x OkHttp: 27 /

    15 = 1,8 x
  19. RxJava: 159 / 82 = 1,94 x OkHttp: 27 /

    15 = 1,8 x Retrofit: 4,9 / 2,8= 1,75 x
  20. !

  21. Is there something common between these projects?

  22. - Assertion libraries

  23. - Assertion libraries ✅

  24. - Assertion libraries ✅ - Mocking libraries

  25. - Assertion libraries ✅ - Mocking libraries ✅

  26. Test framework though?

  27. They all use JUnit (4) And it works

  28. JUnit 4 ❤ - It’s robust

  29. JUnit 4 ❤ - It’s robust - It’s straightforward

  30. JUnit 4 ❤ - It’s robust - It’s straightforward -

    You don’t have to debug it
  31. JUnit 4 ❤ - It’s robust - It’s straightforward -

    You don’t have to debug it - All build systems and IDEs support it
  32. JUnit 4 ❤ - It’s robust - It’s straightforward -

    You don’t have to debug it - All build systems and IDEs support it - Everybody is familiar with it
  33. JUnit 4 ❤ - It’s robust - It’s straightforward -

    You don’t have to debug it - All build systems and IDEs support it - Everybody is familiar with it - It’s a standard.
  34. But it has problems

  35. “Code Repetition”

  36. @Test fun updateCoarseLocationSync_Foregrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(true) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED)

    .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_FG) } JUnit 4: “Code Repetition” Problem
  37. @Test fun updateCoarseLocationSync_NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(false) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED)

    .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem
  38. @Test fun updateCoarseLocationSync_ NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn( false) val first =

    AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED) .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem
  39. @Test fun updateCoarseLocationSync_NotForegrounded() { `when`(appForegroundDetector.isForegrounded).thenReturn(false) val first = AndroidLocationBuilder() .withProvider(AndroidLocation.Provider.FUSED)

    .withLat(0.0) .withLng(0.0) .withTime(1L) .build() locationIngestService.updateCoarseLocationSync(first) val argumentCaptor = ArgumentCaptor.forClass(IngestLocationsRequestDTO::class.java) verify(locationIngestApi, times(1)).postLocations(argumentCaptor.capture()) val value = argumentCaptor.value assertThat(value).isNotNull() assertThat(value.locations).isNotNull().hasSize(1) assertThat(value.locations[0]).isNotNull() assertThat(value.locations[0].source).isEqualTo(Location.SIGNIFICANT_LOCATION_CHANGE_BG) } JUnit 4: “Code Repetition” Problem ⚠
  40. “Test case/context naming”

  41. @Test fun removeAccessTokenShouldStopLoggedinScopeAndStartLoggedOutScope() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN)

    assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit4: “Test case/context naming” Problem
  42. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { JUnit4: “Test case/context naming” Problem accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() }
  43. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() JUnit4: “Test case/context naming” Problem Still bad
  44. “What does this test test?”

  45. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  46. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  47. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem 4 checks
  48. None
  49. Some motivation speech here

  50. Spek

  51. Hadi Hariri @JetBrains Spek v0, 2012

  52. Hadi Hariri @JetBrains Spek v0, 2012

  53. Hadi Hariri @JetBrains Spek v0, 2012 Kotlin 0.4.297

  54. Hadi Hariri @JetBrains Spek v0, 2012

  55. Hadi Hariri @JetBrains Spek v0, 2012

  56. 2016

  57. Spek v1.0.25, 2016 We started to feel JUnit 4 problems

    really badly
  58. Spek v1.0.25, 2016

  59. Spek v1.0.25, 2016

  60. Spek v1.0.25, 2016 We started using it

  61. Spek v1.0.25, 2016

  62. Spek v1.0.25, 2016

  63. None
  64. None
  65. I see what you did there, Hadi

  66. Spek 2.x

  67. : BetterTestingFuture Spek 2.x

  68. class SuperTypicalJUnitTest { val calculator = Calculator() @Test fun `2

    + 4 = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } } Typical Spec
  69. class CalculatorSpec : Spek({ val calculator by memoized { Calculator()

    } context("2 + 4") { val result by memoized { calculator.add(2, 4) } it("equals 6") { assertEquals(6, result) } } }) Typical Spec
  70. Spek: Basic API

  71. context(“2 + 4") { }x Spek: Basic API

  72. context("2 + 4") { }x describe(“2 + 4") { }x

    Spek: Basic API
  73. context("2 + 4") { }x describe("2 + 4") { }x

    given(“2 + 4") { }x Spek: Basic API
  74. describe("2 + 4") { }x context("2 + 4") { }x

    given("2 + 4") { }x group(“2 + 4") Spek: Basic API
  75. group("") ~= Test class in JUnit Spek: Basic API

  76. group("") ~= Test class in JUnit Spek: Basic API You

    can nest groups naturally
  77. it("equals 6") { assertThat(result).isEqualTo(6) } Spek: Basic API

  78. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4

    = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
  79. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4

    = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
  80. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4

    = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
  81. it("equals 6") { assertThat(result).isEqualTo(6) } @Test fun `2 + 4

    = 6`() { val result = calculator.add(2, 4) assertEquals(6, result) } Spek: Basic API
  82. Spek: Basic API it("") = @Test in JUnit

  83. it("") = @Test in JUnit Spek: Basic API You can

    have as many `it` in a `group` as needed
  84. Spek: Basic API Groups and Tests create natural structure that

    scales very well
  85. val calculator by memoized { Calculator() } Spek: Basic API

  86. Spek: Basic API You avoid state sharing betwen tests with

    `memoized`
  87. Let’s rewrite real JUnit test with Spek

  88. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem 4 checks
  89. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  90. context("remove access token") { } Let’s rewrite real JUnit test

    with Spek
  91. context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } } Let’s

    rewrite real JUnit test with Spek
  92. context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } } Let’s

    rewrite real JUnit test with Spek
  93. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  94. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  95. describe("first scope change") { } Let’s rewrite real JUnit test

    with Spek
  96. describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0]

    } } Let’s rewrite real JUnit test with Spek
  97. @Test fun `remove access token should stop logged in scope

    and start logged out scope`() { accessTokenRepository.removeAccessToken() val scopeChange1 = scopeManager.scopeChanges[0] assertThat(scopeChange1.scope).isEqualTo(PassengerScopes.LOGGED_IN) assertThat(scopeChange1.started).isFalse() val scopeChange2 = scopeManager.scopeChanges[1] assertThat(scopeChange2.scope).isEqualTo(PassengerScopes.LOGGED_OUT) assertThat(scopeChange2.started).isTrue() } JUnit 4: “What does this test test” Problem
  98. describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0]

    } it("is 'logged in' scope") { assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN) } it("is not started") { assertThat(firstScopeChange.started).isFalse() } } Let’s rewrite real JUnit test with Spek
  99. class JUnitTest { private var scopeManager = MockScopeManager() private val

    accessTokenRepository = AccessTokenRepository( RuntimeEnvironment.application, scopeManager ) lateinit var scopeChange1: MockScopeManager.ScopeChange lateinit var scopeChange2: MockScopeManager.ScopeChange @Before fun `remove access token`() { accessTokenRepository.removeAccessToken() scopeChange1 = scopeManager.scopeChanges[0] scopeChange2 = scopeManager.scopeChanges[1] } @Test fun `first scope change is 'logged in' scope`() { assertThat(scopeChange1.scope).isEqualTo(LOGGED_IN) } @Test fun `first scope change is not started`() { assertThat(scopeChange1.started).isFalse() } @Test fun `second scope change is 'logged out' scope`() { assertThat(scopeChange2.scope).isEqualTo(LOGGED_OUT) } @Test fun `first scope change is started`() { assertThat(scopeChange2.started).isTrue() } } Let’s rewrite real JUnit test with Spek JUnit is not structured, it’s flat
  100. class Spec : Spek({ val scopeManager by memoized { MockScopeManager()

    } val accessTokenRepository by memoized { AccessTokenRepository(RuntimeEnvironment.application, scopeManager) } context("remove access token") { beforeEachTest { accessTokenRepository.removeAccessToken() } describe("first scope change") { val firstScopeChange by memoized { scopeManager.scopeChanges[0] } it("is 'logged in' scope") { assertThat(firstScopeChange.scope).isEqualTo(LOGGED_IN) } it("is not started") { assertThat(firstScopeChange.started).isFalse() } } describe("second scope change") { val secondScopeChange by memoized { scopeManager.scopeChanges[1] } it("is 'logged out' scope") { assertThat(secondScopeChange.scope).isEqualTo(LOGGED_OUT) } it("is started") { assertThat(secondScopeChange.started).isTrue() } } } }) Let’s rewrite real JUnit test with Spek Spek has structure
  101. Test code can be structured with Spek

  102. Let’s rewrite real JUnit test with Spek Hierarchical Report in

    IntelliJ
  103. The missing documentation & samples Spek Tips

  104. Spek Tips You can iterate stuff

  105. Spek Tips Iterate things, Spek is just a code class

    StringExtensionsSpec : Spek({ listOf(null, "", " ", "\t").forEach { string -> context("null or blank string '$string'") { val isNullOrBlank by memoized { string.isNullOrBlank() } it("is indeed null or blank") { assertThat(isNullOrBlank, equalTo(true)) } } } })
  106. when (spek) ?

  107. Hopefully Spek 2.x release plans Developer Preview by the end

    of November
  108. Most likely Spek 2.x release plans ~month is exactly enough

    to get sick of how we typically write tests
  109. Spek 2.x release plans github.com/spekframework/spek

  110. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin

  111. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin

  112. Spek 2.x release plans github.com/spekframework/spek twitter.com/artem_zin End of November

  113. #kotlinconf17 @artem_zin Artem Zinnatullin Thank you! @hhariri Hadi Hariri @raniejade

    Ranie Jade Ramiso @arturdryomov Artur Dryomov