Flawless testing for the functional folks

The year is 2019 and there have never been more people writing functional Scala. We’ve grown tired of frameworks, magic (reflection) and omnipotent testing libraries with hundreds of testing styles. I want to fight back against tests that stop on the first failed assertion. Against beforeAll and afterEach. I want to seek vengeance for unexpected, globally configured test parallelism and flaky tests. I want to rethink how we test pure and effective functional code and show another approach to doing it in Scala. In this talk, I will share what I’ve learned trying to make that approach become a reality. We’ll talk about the characteristics of test frameworks, functional API design and making trade-offs.

Transcript

1. Flawless testing for the functional folks Jakub Kozłowski https://bit.ly/2RXH05N
2. Expectation Testing Reality
3. object HelloTests extends TestSuite { val tests = Tests { test("1 + 2 == 3"){ assert(1 + 2 == 3) } test("2 + 1 == 3"){ assert(2 + 1 == 3) } } } Expectation Testing Reality
4. object HelloTests extends TestSuite { val tests = Tests { test("1 + 2 == 3"){ assert(1 + 2 == 3) } test("2 + 1 == 3"){ assert(2 + 1 == 3) } } } implicit val sourceConfig = SourceConfig.singleSession[Int](Destination.queue("Input")) brokerAccess(brokerUrl).use { broker => (Deferred[IO, Unit], Deferred[IO, Unit], Deferred[IO, Unit], Ref[IO].of(Chain.empty[Int])).mapN { (firstReceived, secondReceived, brokerDown, consumedMessages) => def handleMessage(i: Int): IO[Unit] = consumedMessages.modify { alreadyConsumed => val action: IO[Unit] = alreadyConsumed.size match { case 0 => firstReceived.complete(()) *> brokerDown.get case 1 => secondReceived.complete(()) case _ => IO.unit } (alreadyConsumed.append(i), action) }.flatten val runStream = Consumer .stream[IO, Int] { case (_, i) => handleMessage(i).as(Nil) } .compile .drain val manageBroker = firstReceived.get *> //kill broker broker.shutdown *> //the consumer waits for this before trying to commit brokerDown.complete(()) *> //wait a sec to make sure the stream has time to realize the problem IO.sleep(1.second) *> //bring broker back broker.start *> //send another message sendMessages("Input")(2) *> secondReceived.get broker.start *> sendMessages("Input")(1) *> Deferred[IO, Unit].flatMap { stopStream => //run stream until promise is finished, then stop //in the meantime, run manageBroker, then finish promise (runStream race stopStream.get) &> manageBroker.guarantee(stopStream.complete(())) } *> consumedMessages.get.map { _.toList shouldBe List(1, 2) } } }.flatten.unsafeRunSync() Expectation Testing Reality
5. Roadmap Traditional test frameworks Functional testing and the future Functional API design a + b a * b
6. Functional API design
7. Functional API design Algebraic data types + pure functions
8. Algebraic data types
9. Algebraic data types type Book = (Id, String) type Person = (String, List[Person])
10. Algebraic data types type Boolean = True | False type Option a = Some a | None type Either a b = Left a | Right b type List a = Nil | Cons a (List a) type Book = (Id, String) type Person = (String, List[Person])
11. Algebraic data types type Boolean = True | False type Option a = Some a | None type Either a b = Left a | Right b type List a = Nil | Cons a (List a) Coproducts ("one of") type Book = (Id, String) type Person = (String, List[Person]) Products ("all of")
12. Algebraic data types type Boolean = True | False type Option a = Some a | None type Either a b = Left a | Right b type List a = Nil | Cons a (List a) Coproducts ("one of") Also called sum types type Book = (Id, String) type Person = (String, List[Person]) Products ("all of")
13. We can encode any immutable value as an ADT type Nothing type Unit = Unit type Bit = Off | On type Byte = (Bit, Bit, Bit, Bit, Bit, Bit, Bit, Bit) type Char = Byte type String = List[Char]
14. Why products and sums? (A, Option[B]) = Either[(A, B), A]
15. Why products and sums? (A, Option[B]) = Either[(A, B), A] A * (1 + B) = (A * B) + A
16. We can prove type equivalence/isomorphism with basic math Either[A, Option[B]] = Either[Option[A], B] (A, Option[B]) = Either[(A, B), A] Either[A, Either[B, C]] = Either[Either[A, B], C] Either[Option[A], Option[B]] = Either[A, Option[Option[B]]] Home exercise: prove these (and look for them in your code)
17. Case study: messages for message queue - TextMessage (UTF-8 text) - ObjectMessage (Java Serializable objects) - Ping (no payload) Each one has both: - id: Long - timestamp: Instant
18. Object oriented style Some fields are shared - abstracted away as interface/trait trait Message { def id: Long def timestamp: Instant } final case class TextMessage (id: Long, timestamp: Instant, payload: String) extends Message final case class ObjectMessage(id: Long, timestamp: Instant, payload: Serializable) extends Message final case class Ping (id: Long, timestamp: Instant) extends Message This is not an ADT
19. Algebraic data types - naive attempt sealed trait Message extends Product with Serializable { def ident: Long = this match { case TextMessage(i, _, _) => i case ObjectMessage(i, _, _) => i case Ping(i, _) => i } def ts: Instant = ... } final case class TextMessage (id: Long, timestamp: Instant, payload: String) extends Message final case class ObjectMessage(id: Long, timestamp: Instant, payload: Serializable) extends Message final case class Ping (id: Long, timestamp: Instant) extends Message
20. Put common fields together in a case class final case class Message(id: Long, timestamp: Instant, ...) Algebraic data types - reimagined
21. Put common fields together in a case class final case class Message(id: Long, timestamp: Instant, payload: Payload) sealed trait Payload extends Product with Serializable object Payload { final case class Text(value: String) extends Payload final case class Object(value: Serializable) extends Payload case object Zilch extends Payload } Algebraic data types - reimagined Make the rest a coproduct
22. Put common fields together in a case class Algebraic data types - reimagined Make the rest a coproduct Add friendly smart constructors object Message { def text(id: Long, timestamp: Instant, value: String ): Message = Message(id, timestamp, Payload.Text(value)) def obj (id: Long, timestamp: Instant, value: Serializable): Message = Message(id, timestamp, Payload.Object(value)) def ping(id: Long, timestamp: Instant ): Message = Message(id, timestamp, Payload.Zilch) } final case class Message(id: Long, timestamp: Instant, payload: Payload) sealed trait Payload extends Product with Serializable object Payload { final case class Text(value: String) extends Payload final case class Object(value: Serializable) extends Payload case object Zilch extends Payload }
23. Need to know the exact type of a message? def showText(message: TextMessage): IO[Unit] = IO { ... } Before:
24. Need to know the exact type of a message? def showText(message: TextMessage): IO[Unit] = IO { ... } Before: def showText(message: Message): IO[Unit] = IO { } After:
25. Need to know the exact type of a message? def showText(message: TextMessage): IO[Unit] = IO { ... } Before: def showText(message: Message): IO[Unit] = IO { } After:
26. final case class Message[P](id: Long, timestamp: Instant, payload: Payload[P]) object Message { def text(id: Long, timestamp: Instant, value: String) : Message[String] = Message(id, timestamp, Payload.Text(value)) def obj (id: Long, timestamp: Instant, value: Serializable): Message[Serializable] = Message(id, timestamp, Payload.Object(value)) def ping(id: Long, timestamp: Instant) : Message[Unit] = Message(id, timestamp, Payload.Zilch) } sealed trait Payload[P] extends Product with Serializable object Payload { final case class Text (value: String) extends Payload[String] final case class Object(value: Serializable) extends Payload[Serializable] case object Zilch extends Payload[Unit] } Generalized algebraic data types (GADTs) Here: Payload
27. final case class Message[P](id: Long, timestamp: Instant, payload: Payload[P]) object Message { def text(id: Long, timestamp: Instant, value: String) : Message[String] = Message(id, timestamp, Payload.Text(value)) def obj (id: Long, timestamp: Instant, value: Serializable): Message[Serializable] = Message(id, timestamp, Payload.Object(value)) def ping(id: Long, timestamp: Instant) : Message[Unit] = Message(id, timestamp, Payload.Zilch) } sealed trait Payload[P] extends Product with Serializable object Payload { final case class Text (value: String) extends Payload[String] final case class Object(value: Serializable) extends Payload[Serializable] case object Zilch extends Payload[Unit] } Generalized algebraic data types (GADTs) Here: Payload
28. Generalized algebraic data types (GADTs) Here: Payload final case class Message[P](id: Long, timestamp: Instant, payload: Payload[P]) object Message { def text(id: Long, timestamp: Instant, value: String) : Message[String] = Message(id, timestamp, Payload.Text(value)) def obj (id: Long, timestamp: Instant, value: Serializable): Message[Serializable] = Message(id, timestamp, Payload.Object(value)) def ping(id: Long, timestamp: Instant) : Message[Unit] = Message(id, timestamp, Payload.Zilch) } sealed trait Payload[P] extends Product with Serializable object Payload { final case class Text (value: String) extends Payload[String] final case class Object(value: Serializable) extends Payload[Serializable] case object Zilch extends Payload[Unit] }
29. Functions on GADTs
30. Functions on GADTs def showText(message: Message[String]): IO[Unit] = {
31. Functions on GADTs def showText(message: Message[String]): IO[Unit] = { message.payload match { case Text(value) => IO { } } }
32. Functions on GADTs def showText(message: Message[String]): IO[Unit] = { message.payload match { case Text(value) => IO { } } }
33. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) A C B D
34. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) A C B D
35. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type A C B D
36. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload A C B D
37. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] A C B D
38. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary A C B D
39. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload A C B D
40. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload A C B D
41. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ A C B D
42. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ ✅ ✅ ✅ A C B D
43. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ ✅ ✅ ✅ Singleton types in payload abstraction A C B D
44. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ ✅ ✅ ✅ Singleton types in payload abstraction N/A A C B D
45. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ ✅ ✅ ✅ Singleton types in payload abstraction N/A Only if children of Payload support them A C B D
46. ADT vs GADT vs generic type vs generic type with upper bound final case class Message( id: Long, timestamp: Instant, payload: Payload ) final case class Message[P]( id: Long, timestamp: Instant, payload: P ) final case class Message[P]( id: Long, timestamp: Instant, payload: Payload[P] ) (Payload) final case class Message[P <: Payload]( id: Long, timestamp: Instant, payload: P ) ADT (A) GADT (B) Generic (C) Generic with bound (D) Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload Abstract over payload ❌ ✅ ✅ ✅ Singleton types in payload abstraction N/A Only if children of Payload support them ✅ ✅ A C B D
47. Functions class CreditCard(var credit: Int) def pay(amount: PositiveInt, card: CreditCard): Unit = card.credit -= amount.value def program() = { val card = new CreditCard(300) pay(100, card) pay(200, card) println(getBalance(card)) //0 }
48. Functions? class CreditCard(var credit: Int) def pay(amount: PositiveInt, card: CreditCard): Unit = card.credit -= amount.value def program() = { val card = new CreditCard(300) pay(100, card) println(card.credit) //200 }
49. Functions? class CreditCard(var credit: Int) def pay(amount: PositiveInt, card: CreditCard): Unit = card.credit -= amount.value def program() = { val card = new CreditCard(300) pay(100, card) println(card.credit) //200 }
50. Not functions: referential transparency broken class CreditCard(var credit: Int) def pay(amount: PositiveInt, card: CreditCard): Unit = card.credit -= amount.value def program() = { pay(100, new CreditCard(300)) println(new CreditCard(300).credit) //300 }
51. Pure functions case class CreditCard(credit: Int) def pay(amount: PositiveInt, card: CreditCard): CreditCard = card.copy(credit = card.credit - amount.value) def program() = { val card = CreditCard(300) val newCard = pay(100, card) println(newCard.credit) //200 println(card.credit) //300 }
52. Pure functions case class CreditCard(credit: Int) def pay(amount: PositiveInt, card: CreditCard): CreditCard = card.copy(credit = card.credit - amount.value) def program() = { val card = CreditCard(300) val newCard = pay(100, card) println(newCard.credit) //200 println(card.credit) //300 }
53. Pure functions case class CreditCard(credit: Int) def pay(amount: PositiveInt, card: CreditCard): CreditCard = card.copy(credit = card.credit - amount.value) def program() = { val card = CreditCard(300) val newCard = pay(100, card) println(newCard.credit) //200 println(card.credit) //300 } Q: What about chaining operations? A: State monad, beyond the scope of this talk (see https://www.youtube.com/watch?v=Pgo73GfHk0U)
54. Pure functions case class CreditCard(credit: Int) def pay(amount: PositiveInt, card: CreditCard): CreditCard = card.copy(credit = card.credit - amount.value) def program() = { val card = CreditCard(300) val newCard = pay(100, card) println(newCard.credit) //200 println(card.credit) //300 } Q: What about chaining operations? A: State monad, beyond the scope of this talk (see https://www.youtube.com/watch?v=Pgo73GfHk0U)
55. Pure functions case class CreditCard(credit: Int) def pay(amount: PositiveInt, card: CreditCard): CreditCard = card.copy(credit = card.credit - amount.value) def program() = { val newCard = pay(100, CreditCard(300)) println(newCard.credit) //200 println(CreditCard(300).credit) //300 }
56. Alright, hold on dude, these functions are pure because they don't interact with the real world! - You, 2019
57. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world
58. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] = db.update(_ |+| Map(card -> CreditCardDB(amount.value))) def getCredit(card: CardId, db: Database): IO[Int] = db.get.map(_.get(card).map(_.credit).getOrElse(0))
59. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] = db.update(_ |+| Map(card -> CreditCardDB(amount.value))) def getCredit(card: CardId, db: Database): IO[Int] = db.get.map(_.get(card).map(_.credit).getOrElse(0)) def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1)
60. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] = db.update(_ |+| Map(card -> CreditCardDB(amount.value))) def getCredit(card: CardId, db: Database): IO[Int] = db.get.map(_.get(card).map(_.credit).getOrElse(0)) def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db)
61. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] = db.update(_ |+| Map(card -> CreditCardDB(amount.value))) def getCredit(card: CardId, db: Database): IO[Int] = db.get.map(_.get(card).map(_.credit).getOrElse(0)) def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db)
62. def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () } Pure functions in the real world def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] = db.update(_ |+| Map(card -> CreditCardDB(amount.value))) def getCredit(card: CardId, db: Database): IO[Int] = db.get.map(_.get(card).map(_.credit).getOrElse(0)) def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () }
63. Pure functions in the real world def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () }
64. Pure functions in the real world def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) for { _ <- pay(100, card, db) _ <- pay(100, card, db) credit <- getCredit(card, db) _ <- putStrLn(credit) //200 } yield () }
65. Pure functions in the real world def program() = newDatabase(initialCredit = 300).flatMap { db => val card = CardId(1) val pay100 = pay(100, card, db) for { _ <- pay100 _ <- pay100 credit <- getCredit(card, db) _ <- putStrLn(credit) //still 200 } yield () } ✅
66. Traditional testing frameworks DSL Registering suites Parallelism Lifecycle management
67. Traditional testing frameworks DSL - exceptions Registering suites Parallelism Lifecycle management test("list") { val list = List("foo", "bar") //fails ✅ assert(list.size == 3) //never gets run ❌ assert(list.forall(_ == "moo")) }
68. Traditional testing frameworks DSL - exceptions, side effects Registering suites Parallelism Lifecycle management val tests = Tests { test("hello world") { 1 shouldBe 1 } test("hello world 2") { 1 shouldBe 1 } } val tests = Tests { test("hello world") { 1 shouldBe 1 } test("hello world 2") { 1 shouldBe 1 } }
69. Traditional testing frameworks DSL - exceptions, side effects Registering suites Parallelism Lifecycle management val tests = Tests { test("hello world") { 1 shouldBe 1 } test("hello world 2") { 1 shouldBe 1 } } val tests = Tests { test("hello world") { 1 shouldBe 1 } test("hello world 2") { 1 shouldBe 1 } }
70. Traditional testing frameworks DSL - exceptions, side effects Registering suites Parallelism Lifecycle management val tests = Tests { test("hello world") { 1 shouldBe 1 } test("hello world 2") { 1 shouldBe 1 } } val test2 = test("hello world 2") { 1 shouldBe 1 } val tests = Tests { test("hello world") { 1 shouldBe 1 } test2 //nope }
71. def test(body: => Assertion): Unit Functional "Thunk-ctional" programming
72. Traditional testing frameworks DSL - exceptions, side effects Registering suites - automatic Parallelism Lifecycle management //scalatest class ContainerSpec extends WordSpec //specs2 class ContainerSpec extends Specification //minitest object ContainerSpec extends SimpleTestSuite //µtest class ContainerSpec extends TestSuite
73. Traditional testing frameworks DSL - exceptions, side effects Registering suites - automatic Parallelism - globally configured Lifecycle management //per sbt module parallelExecution in Test := false //per build concurrentRestrictions in Global += Tags.exclusive(Tags.Test)
74. Traditional testing frameworks DSL - exceptions, side effects Registering suites - automatic Parallelism - globally configured Lifecycle management - hooks object MyTestSuite extends TestSuite[Int] { private var system: ActorSystem = _ override def setupSuite(): Unit = { system = ActorSystem.create() } override def tearDownSuite(): Unit = { TestKit.shutdownActorSystem(system) system = null } }
75. Traditional testing frameworks DSL - exceptions, side effects Registering suites - automatic Parallelism - globally configured Lifecycle management - hooks
76. Functional testing libraries DSL Registering suites Parallelism Lifecycle management
77. val bigTest = test("big test") { for { //given time <- IO(ZonedDateTime.now()) timeProvider = TimeProvider.const(time) log <- Ref[IO].of(List.empty[AppEvent]) eventSender = EventSender.instance(e => log.update(_ :+ e)) videoService = VideoService.make(eventSender, timeProvider) //when rentResult <- videoService.rent(VideoId(1)) logged <- log.get } yield { val failedToRent = rentResult.shouldBe(Failed(VideoId(1))) val sentEvent = logged.shouldBe(List(AppEvent.FailedRent(time))) //then failedToRent |+| sentEvent } } val suite = tests( pureTest("hello") { testedFunction(1) shouldBe List(1) }, pureTest("hello 2") { val result = testedFunction(2) result.size.shouldBe(2) |+| result.shouldBe(List(2, 2)) }, bigTest ) Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management
78. val bigTest = test("big test") { for { //given time <- IO(ZonedDateTime.now()) timeProvider = TimeProvider.const(time) log <- Ref[IO].of(List.empty[AppEvent]) eventSender = EventSender.instance(e => log.update(_ :+ e)) videoService = VideoService.make(eventSender, timeProvider) //when rentResult <- videoService.rent(VideoId(1)) logged <- log.get } yield { val failedToRent = rentResult.shouldBe(Failed(VideoId(1))) val sentEvent = logged.shouldBe(List(AppEvent.FailedRent(time))) //then failedToRent |+| sentEvent } } val suite = tests( pureTest("hello") { testedFunction(1) shouldBe List(1) }, pureTest("hello 2") { val result = testedFunction(2) result.size.shouldBe(2) |+| result.shouldBe(List(2, 2)) }, bigTest ) Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management
79. s val bigTest = test("big test") { for { //given time <- IO(ZonedDateTime.now()) timeProvider = TimeProvider.const(time) log <- Ref[IO].of(List.empty[AppEvent]) eventSender = EventSender.instance(e => log.update(_ :+ e)) videoService = VideoService.make(eventSender, timeProvider) //when rentResult <- videoService.rent(VideoId(1)) logged <- log.get } yield { val failedToRent = rentResult.shouldBe(Failed(VideoId(1))) val sentEvent = logged.shouldBe(List(AppEvent.FailedRent(time))) //then failedToRent |+| sentEvent } } val suite = tests( pureTest("hello") { testedFunction(1) shouldBe List(1) }, pureTest("hello 2") { val result = testedFunction(2) result.size.shouldBe(2) |+| result.shouldBe(List(2, 2)) }, bigTest ) Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management
80. val bigTest = test("big test") { for { //given time <- IO(ZonedDateTime.now()) timeProvider = TimeProvider.const(time) log <- Ref[IO].of(List.empty[AppEvent]) eventSender = EventSender.instance(e => log.update(_ :+ e)) videoService = VideoService.make(eventSender, timeProvider) //when rentResult <- videoService.rent(VideoId(1)) logged <- log.get } yield { val failedToRent = rentResult.shouldBe(Failed(VideoId(1))) val sentEvent = logged.shouldBe(List(AppEvent.FailedRent(time))) //then failedToRent |+| sentEvent } } val suite = tests( pureTest("hello") { testedFunction(1) shouldBe List(1) }, pureTest("hello 2") { val result = testedFunction(2) result.size.shouldBe(2) |+| result.shouldBe(List(2, 2)) }, bigTest ) Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management
81. val bigTest = test("big test") { for { //given time <- IO(ZonedDateTime.now()) timeProvider = TimeProvider.const(time) log <- Ref[IO].of(List.empty[AppEvent]) eventSender = EventSender.instance(e => log.update(_ :+ e)) videoService = VideoService.make(eventSender, timeProvider) //when rentResult <- videoService.rent(VideoId(1)) logged <- log.get } yield { val failedToRent = rentResult.shouldBe(Failed(VideoId(1))) val sentEvent = logged.shouldBe(List(AppEvent.FailedRent(time))) //then failedToRent |+| sentEvent } } val suite = tests( pureTest("hello") { testedFunction(1) shouldBe List(1) }, pureTest("hello 2") { val result = testedFunction(2) result.size.shouldBe(2) |+| result.shouldBe(List(2, 2)) }, bigTest ) Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management
82. Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management pureTest("simple things") { ((2 + 2) shouldBe 3) |+| (2 shouldBe 3) }
83. Functional testing libraries DSL - purely functional, no exceptions Registering suites Parallelism Lifecycle management pureTest("simple things") { ((2 + 2) shouldBe 3) |+| (2 shouldBe 3) } Never lose failures again
84. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism Lifecycle management object FlawlessTests extends IOApp { def run(args: List[String]): IO[ExitCode] = runTests(args) { val sequentialTests = NonEmptyList.of( GetStatsTest, VisitTests ) Tests.sequence(sequentialTests) } }
85. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism - explicit composition Lifecycle management object FlawlessTests extends IOApp { def run(args: List[String]): IO[ExitCode] = runTests(args) { val sequentialTests = NonEmptyList.of( GetStatsTest, VisitTests ) val parallelTests = NonEmptyList.of( RunnerTests, SyntaxTests ) Tests.sequence(sequentialTests) |+| Tests.parSequence(parallelTests) } }
86. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism - explicit composition Lifecycle management object FlawlessTests extends IOApp { def run(args: List[String]): IO[ExitCode] = runTests(args) { val sequentialTests = NonEmptyList.of( GetStatsTest, VisitTests ) val parallelTests = NonEmptyList.of( RunnerTests, SyntaxTests ) Tests.sequence(sequentialTests) |+| Tests.parSequence(parallelTests) } } These run sequentially
87. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism - explicit composition Lifecycle management object FlawlessTests extends IOApp { def run(args: List[String]): IO[ExitCode] = runTests(args) { val sequentialTests = NonEmptyList.of( GetStatsTest, VisitTests ) val parallelTests = NonEmptyList.of( RunnerTests, SyntaxTests ) Tests.sequence(sequentialTests) |+| Tests.parSequence(parallelTests) } } These run in parallel
88. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism - explicit composition Lifecycle management object FlawlessTests extends IOApp { def run(args: List[String]): IO[ExitCode] = runTests(args) { val sequentialTests = NonEmptyList.of( GetStatsTest, VisitTests ) val parallelTests = NonEmptyList.of( RunnerTests, SyntaxTests ) Tests.sequence(sequentialTests) |+| Tests.parSequence(parallelTests) } } Sequential composition by default
89. Functional testing libraries DSL - purely functional, no exceptions Registering suites - manual Parallelism - explicit composition Lifecycle management - Resource monad val makeActorSystem: Resource[IO, ActorSystem] = Resource.make(IO(ActorSystem())) { as => IO.fromFuture(IO(as.shutdown)) } val suite = Tests.resource(makeActorSystem).use { as => tests( test("system") { ... } ) }
90. The future
91. The future import flawless._
92. Your tests as a recursive data structure val suite: Tests[NonEmptyList[SuiteResult]] = { tests( test("flaky") { IO(Random.nextBoolean()).map(_ shouldBe true) }, pureTest("obvious") { 1 shouldBe 1 } ).lift[NonEmptyList] |+| Tests.parSequence { val results = List(..., ..., ...) NonEmptyList.of(1, 2, 3).map { i => lazyTest(s"fib(${1000*i})") { fib(i * 1000) shouldBe results(i) } } } }.via(findFlaky(maxAttempts = 1000))
93. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast)
94. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast) Would retry effectful tests
95. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast) Would cache pure tests until recompiled
96. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast) Would limit running time of each test in the tree
97. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast) Would skip tests with resources / tests with IO
98. Traverse the test tree how you want .via(findFlaky(maxAttempts = 1000)) .via(catchNonFatal) .via(retry(schedule)) .via(cached(...)) .via(timeoutEach(5.seconds)) .via(disabled) .via(fast) Would skip tests with resources / tests with IO (or all tests!)
99. Experiment more with the API Find the right abstractions Constraints vs liberties See what's possible Gather feedback Find other ridiculous ideas that might make sense TODO
100. Appendix: what changed since last time I gave this talk @jdegoes @adamgfraser
101. ZIO Test vs flawless testM("parallel offers and sequential takes") { effect.map(l => assert(l.toSet, equalTo(values.toSet))) } test("parallel offers and sequential takes") { effect.map(l => l.toSet shouldBe values.toSet) }
102. ZIO Test vs flawless testM("parallel offers and sequential takes") { effect.map(l => assert(l.toSet, equalTo(values.toSet))) } test("parallel offers and sequential takes") { effect.map(l => l.toSet shouldBe values.toSet) }
103. ZIO Test vs flawless ZIO Test flawless Purely functional ✅ ✅ Tests as a data structure ✅ ✅ Aspects as functions ✅ ✅
104. ZIO Test vs flawless ZIO Test flawless Purely functional ✅ ✅ Tests as a data structure ✅ ✅ Aspects as functions ✅ ✅ Minimal ❌ ✅ (so far!) Arbitrary nesting of tests ✅ ❌ (yet) Available now ✅ ⏳
105. WIP Demo time?
106. Thank you blog.kubukoz.com @kubukoz Slides: https://bit.ly/2RXH05N Code: https://git.io/fj6oc