Flawless testing for the functional folks

Flawless testing
for the functional folks
Jakub Kozłowski
https://bit.ly/2RXH05N

View Slide

Expectation
Testing
Reality

View Slide

object HelloTests extends TestSuite {
val tests = Tests {
test("1 + 2 == 3"){
assert(1 + 2 == 3)
}
test("2 + 1 == 3"){
assert(2 + 1 == 3)
}
}
}
Expectation
Testing
Reality

View Slide

object HelloTests extends TestSuite {
val tests = Tests {
test("1 + 2 == 3"){
assert(1 + 2 == 3)
}
test("2 + 1 == 3"){
assert(2 + 1 == 3)
}
}
}
implicit val sourceConfig = SourceConfig.singleSession[Int](Destination.queue("Input"))
brokerAccess(brokerUrl).use { broker =>
(Deferred[IO, Unit], Deferred[IO, Unit], Deferred[IO, Unit], Ref[IO].of(Chain.empty[Int])).mapN {
(firstReceived, secondReceived, brokerDown, consumedMessages) =>
def handleMessage(i: Int): IO[Unit] =
consumedMessages.modify { alreadyConsumed =>
val action: IO[Unit] = alreadyConsumed.size match {
case 0 => firstReceived.complete(()) *> brokerDown.get
case 1 => secondReceived.complete(())
case _ => IO.unit
}
(alreadyConsumed.append(i), action)
}.flatten
val runStream = Consumer
.stream[IO, Int] { case (_, i) => handleMessage(i).as(Nil) }
.compile
.drain
val manageBroker =
firstReceived.get *>
//kill broker
broker.shutdown *>
//the consumer waits for this before trying to commit
brokerDown.complete(()) *>
//wait a sec to make sure the stream has time to realize the problem
IO.sleep(1.second) *>
//bring broker back
broker.start *>
//send another message
sendMessages("Input")(2) *>
secondReceived.get
broker.start *> sendMessages("Input")(1) *> Deferred[IO, Unit].flatMap { stopStream =>
//run stream until promise is finished, then stop
//in the meantime, run manageBroker, then finish promise
(runStream race stopStream.get) &> manageBroker.guarantee(stopStream.complete(()))
} *> consumedMessages.get.map {
_.toList shouldBe List(1, 2)
}
}
}.flatten.unsafeRunSync()
Expectation
Testing
Reality

View Slide

Roadmap
Traditional
test frameworks
Functional testing
and the future
Functional
API design
a + b
a * b

View Slide

Functional API design

View Slide

Functional API design
Algebraic data types + pure functions

View Slide

Algebraic data types

View Slide

type Book = (Id, String)
type Person = (String, List[Person])

View Slide

type Boolean = True | False
type Option a = Some a | None
type Either a b = Left a | Right b
type List a = Nil | Cons a (List a)

View Slide

Coproducts ("one of")
Products ("all of")

View Slide

Coproducts ("one of")
Also called sum types
Products ("all of")

View Slide

We can encode any immutable value as an ADT
type Nothing
type Unit = Unit
type Bit = Off | On
type Byte = (Bit, Bit, Bit, Bit, Bit, Bit, Bit, Bit)
type Char = Byte
type String = List[Char]

View Slide

Why products and sums?
(A, Option[B]) = Either[(A, B), A]

View Slide

Why products and sums?
A * (1 + B) = (A * B) + A

View Slide

We can prove type equivalence/isomorphism with basic math
Either[A, Option[B]] = Either[Option[A], B]
Either[A, Either[B, C]] = Either[Either[A, B], C]
Either[Option[A], Option[B]] = Either[A, Option[Option[B]]]
Home exercise: prove these (and look for them in your code)

View Slide

Case study: messages for message queue
- TextMessage (UTF-8 text)
- ObjectMessage (Java Serializable objects)
- Ping (no payload)
Each one has both:
- id: Long
- timestamp: Instant

View Slide

Object oriented style
Some ﬁelds are shared - abstracted away as interface/trait
trait Message {
def id: Long
def timestamp: Instant
}
final case class TextMessage (id: Long, timestamp: Instant, payload: String) extends Message
final case class ObjectMessage(id: Long, timestamp: Instant, payload: Serializable) extends Message
final case class Ping (id: Long, timestamp: Instant) extends Message
This is not an ADT

View Slide

Algebraic data types - naive attempt
sealed trait Message extends Product with Serializable {
def ident: Long = this match {
case TextMessage(i, _, _) => i
case ObjectMessage(i, _, _) => i
case Ping(i, _) => i
}
def ts: Instant = ...
}
final case class TextMessage (id: Long, timestamp: Instant, payload: String) extends Message
final case class ObjectMessage(id: Long, timestamp: Instant, payload: Serializable) extends Message
final case class Ping (id: Long, timestamp: Instant) extends Message

View Slide

Put common ﬁelds together in a case class
final case class Message(id: Long, timestamp: Instant, ...)
Algebraic data types - reimagined

View Slide

final case class Message(id: Long, timestamp: Instant, payload: Payload)
sealed trait Payload extends Product with Serializable
object Payload {
final case class Text(value: String) extends Payload
final case class Object(value: Serializable) extends Payload
case object Zilch extends Payload
}
Make the rest a coproduct

View Slide

Make the rest a coproduct
Add friendly smart constructors
object Message {
def text(id: Long, timestamp: Instant, value: String ): Message = Message(id, timestamp, Payload.Text(value))
def obj (id: Long, timestamp: Instant, value: Serializable): Message = Message(id, timestamp, Payload.Object(value))
def ping(id: Long, timestamp: Instant ): Message = Message(id, timestamp, Payload.Zilch)
}
final case class Message(id: Long, timestamp: Instant, payload: Payload)
sealed trait Payload extends Product with Serializable
object Payload {
final case class Text(value: String) extends Payload
final case class Object(value: Serializable) extends Payload
case object Zilch extends Payload
}

View Slide

Need to know the exact type of a message?
def showText(message: TextMessage): IO[Unit] = IO {
...
}
Before:

View Slide

Need to know the exact type of a message?
def showText(message: TextMessage): IO[Unit] = IO {
...
}
Before:
def showText(message: Message): IO[Unit] = IO {
}
After:

View Slide

final case class Message[P](id: Long, timestamp: Instant, payload: Payload[P])
object Message {
def text(id: Long, timestamp: Instant, value: String) : Message[String] = Message(id, timestamp, Payload.Text(value))
def obj (id: Long, timestamp: Instant, value: Serializable): Message[Serializable] = Message(id, timestamp, Payload.Object(value))
def ping(id: Long, timestamp: Instant) : Message[Unit] = Message(id, timestamp, Payload.Zilch)
}
sealed trait Payload[P] extends Product with Serializable
object Payload {
final case class Text (value: String) extends Payload[String]
final case class Object(value: Serializable) extends Payload[Serializable]
case object Zilch extends Payload[Unit]
}
Generalized algebraic data types (GADTs)
Here: Payload

View Slide

Generalized algebraic data types (GADTs)
Here: Payload
final case class Message[P](id: Long, timestamp: Instant, payload: Payload[P])
object Message {
def text(id: Long, timestamp: Instant, value: String) : Message[String] = Message(id, timestamp, Payload.Text(value))
def obj (id: Long, timestamp: Instant, value: Serializable): Message[Serializable] = Message(id, timestamp, Payload.Object(value))
def ping(id: Long, timestamp: Instant) : Message[Unit] = Message(id, timestamp, Payload.Zilch)
}
sealed trait Payload[P] extends Product with Serializable
object Payload {
final case class Text (value: String) extends Payload[String]
final case class Object(value: Serializable) extends Payload[Serializable]
case object Zilch extends Payload[Unit]
}

View Slide

Functions on GADTs

View Slide

Functions on GADTs
def showText(message: Message[String]): IO[Unit] = {

View Slide

Functions on GADTs
def showText(message: Message[String]): IO[Unit] = {
message.payload match {
case Text(value) => IO {
}
}
}

View Slide

ADT vs GADT vs generic type vs generic type with upper bound
final case class Message(
id: Long,
timestamp: Instant,
payload: Payload
)
final case class Message[P](
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
final case class Message[P <: Payload](
id: Long,
timestamp: Instant,
payload: P
)
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
ADT (A) GADT (B) Generic (C) Generic with bound (D)
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Constrain payload type
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Constrain payload type extends Payload
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Constrain payload type extends Payload ∃ A . A extends Payload[P]
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Constrain payload type extends Payload ∃ A . A extends Payload[P] Arbitrary extends Payload
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Abstract over payload
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Abstract over payload ❌
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Abstract over payload ❌ ✅ ✅ ✅
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
Singleton types in payload
abstraction
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
abstraction
N/A
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
abstraction
N/A
Only if children of Payload
support them
A
C
B
D

View Slide

id: Long,
timestamp: Instant,
payload: Payload
)
id: Long,
timestamp: Instant,
payload: P
)
id: Long,
timestamp: Instant,
payload: Payload[P]
)
(Payload)
id: Long,
timestamp: Instant,
payload: P
)
abstraction
N/A
Only if children of Payload
support them
✅ ✅
A
C
B
D

View Slide

Functions
class CreditCard(var credit: Int)
def pay(amount: PositiveInt, card: CreditCard): Unit =
card.credit -= amount.value
def program() = {
val card = new CreditCard(300)
pay(100, card)
pay(200, card)
println(getBalance(card)) //0
}

View Slide

Functions?
def program() = {
val card = new CreditCard(300)
pay(100, card)
println(card.credit) //200
}

View Slide

Not functions: referential transparency broken
def program() = {
pay(100, new CreditCard(300))
println(new CreditCard(300).credit) //300
}

View Slide

Pure functions
case class CreditCard(credit: Int)
def pay(amount: PositiveInt, card: CreditCard): CreditCard =
card.copy(credit = card.credit - amount.value)
def program() = {
val card = CreditCard(300)
val newCard = pay(100, card)
println(newCard.credit) //200
}

View Slide

Pure functions
def program() = {
val card = CreditCard(300)
val newCard = pay(100, card)
}
Q: What about chaining operations?
A: State monad, beyond the scope of this talk (see https://www.youtube.com/watch?v=Pgo73GfHk0U)

View Slide

Pure functions
def program() = {
val newCard = pay(100, CreditCard(300))
println(CreditCard(300).credit) //300
}

View Slide

Alright, hold on dude, these functions are pure because they
don't interact with the real world!
- You, 2019

View Slide

def program() = newDatabase(initialCredit = 300).flatMap { db =>
val card = CardId(1)
for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
credit <- getCredit(card, db)
_ <- putStrLn(credit) //200
} yield ()
}
Pure functions in the real world

View Slide

for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}
def pay(amount: PositiveInt, card: CardId, db: Database): IO[Unit] =
db.update(_ |+| Map(card -> CreditCardDB(amount.value)))
def getCredit(card: CardId, db: Database): IO[Int] =
db.get.map(_.get(card).map(_.credit).getOrElse(0))

View Slide

for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}

View Slide

for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}
for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)

View Slide

for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}
for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}

View Slide

for {
_ <- pay(100, card, db)
_ <- pay(100, card, db)
} yield ()
}

View Slide

val pay100 = pay(100, card, db)
for {
_ <- pay100
_ <- pay100
_ <- putStrLn(credit) //still 200
} yield ()
}
✅

View Slide

Traditional testing frameworks
DSL
Registering suites
Parallelism
Lifecycle management

View Slide

DSL - exceptions
Registering suites
Parallelism
test("list") {
val list = List("foo", "bar")
//fails ✅
assert(list.size == 3)
//never gets run ❌
assert(list.forall(_ == "moo"))
}

View Slide

DSL - exceptions, side eﬀects
Registering suites
Parallelism
val tests = Tests {
test("hello world") {
1 shouldBe 1
}
test("hello world 2") {
1 shouldBe 1
}
}
val tests = Tests {
1 shouldBe 1
}
1 shouldBe 1
}
}

View Slide

Registering suites
Parallelism
val tests = Tests {
1 shouldBe 1
}
1 shouldBe 1
}
}
val test2 = test("hello world 2") {
1 shouldBe 1
}
val tests = Tests {
1 shouldBe 1
}
test2 //nope
}

View Slide

def test(body: => Assertion): Unit
Functional "Thunk-ctional" programming

View Slide

Registering suites - automatic
Parallelism
//scalatest
class ContainerSpec extends WordSpec
//specs2
class ContainerSpec extends Specification
//minitest
object ContainerSpec extends SimpleTestSuite
//µtest
class ContainerSpec extends TestSuite

View Slide

Parallelism - globally conﬁgured
//per sbt module
parallelExecution in Test := false
//per build
concurrentRestrictions in Global +=
Tags.exclusive(Tags.Test)

View Slide

Lifecycle management - hooks
object MyTestSuite extends TestSuite[Int] {
private var system: ActorSystem = _
override def setupSuite(): Unit = {
system = ActorSystem.create()
}
override def tearDownSuite(): Unit = {
TestKit.shutdownActorSystem(system)
system = null
}
}

View Slide

Lifecycle management - hooks

View Slide

Functional testing libraries
DSL
Registering suites
Parallelism

View Slide

val bigTest = test("big test") {
for {
//given
time <- IO(ZonedDateTime.now())
timeProvider = TimeProvider.const(time)
log <- Ref[IO].of(List.empty[AppEvent])
eventSender = EventSender.instance(e => log.update(_ :+ e))
videoService = VideoService.make(eventSender, timeProvider)
//when
rentResult <- videoService.rent(VideoId(1))
logged <- log.get
} yield {
val failedToRent =
rentResult.shouldBe(Failed(VideoId(1)))
val sentEvent =
logged.shouldBe(List(AppEvent.FailedRent(time)))
//then
failedToRent |+| sentEvent
}
}
val suite = tests(
pureTest("hello") {
testedFunction(1) shouldBe List(1)
},
pureTest("hello 2") {
val result = testedFunction(2)
result.size.shouldBe(2) |+|
result.shouldBe(List(2, 2))
},
bigTest
)
DSL - purely functional, no exceptions
Registering suites
Parallelism

View Slide

s
for {
//given
//when
logged <- log.get
} yield {
val failedToRent =
val sentEvent =
//then
}
}
val suite = tests(
pureTest("hello") {
},
},
bigTest
)
Registering suites
Parallelism

View Slide

for {
//given
//when
logged <- log.get
} yield {
val failedToRent =
val sentEvent =
//then
}
}
val suite = tests(
pureTest("hello") {
},
},
bigTest
)
Registering suites
Parallelism

View Slide

Registering suites
Parallelism
pureTest("simple things") {
((2 + 2) shouldBe 3) |+| (2 shouldBe 3)
}

View Slide

Registering suites
Parallelism
pureTest("simple things") {
((2 + 2) shouldBe 3) |+| (2 shouldBe 3)
}
Never lose failures again

View Slide

Registering suites - manual
Parallelism
object FlawlessTests extends IOApp {
def run(args: List[String]): IO[ExitCode] = runTests(args) {
val sequentialTests = NonEmptyList.of(
GetStatsTest,
VisitTests
)
Tests.sequence(sequentialTests)
}
}

View Slide

Parallelism - explicit composition
GetStatsTest,
VisitTests
)
val parallelTests = NonEmptyList.of(
RunnerTests,
SyntaxTests
)
Tests.sequence(sequentialTests) |+|
Tests.parSequence(parallelTests)
}
}

View Slide

GetStatsTest,
VisitTests
)
RunnerTests,
SyntaxTests
)
}
}
These run sequentially

View Slide

GetStatsTest,
VisitTests
)
RunnerTests,
SyntaxTests
)
}
} These run in parallel

View Slide

GetStatsTest,
VisitTests
)
RunnerTests,
SyntaxTests
)
}
}
Sequential composition by default

View Slide

Lifecycle management - Resource monad
val makeActorSystem: Resource[IO, ActorSystem] =
Resource.make(IO(ActorSystem())) { as =>
IO.fromFuture(IO(as.shutdown))
}
val suite = Tests.resource(makeActorSystem).use { as =>
tests(
test("system") {
...
}
)
}

View Slide

The future

View Slide

The future
import flawless._

View Slide

Your tests as a recursive data structure
val suite: Tests[NonEmptyList[SuiteResult]] = {
tests(
test("flaky") {
IO(Random.nextBoolean()).map(_ shouldBe true)
},
pureTest("obvious") {
1 shouldBe 1
}
).lift[NonEmptyList] |+|
Tests.parSequence {
val results = List(..., ..., ...)
NonEmptyList.of(1, 2, 3).map { i =>
lazyTest(s"fib(${1000*i})") {
fib(i * 1000) shouldBe results(i)
}
}
}
}.via(findFlaky(maxAttempts = 1000))

View Slide

Traverse the test tree how you want
.via(findFlaky(maxAttempts = 1000))
.via(catchNonFatal)
.via(retry(schedule))
.via(cached(...))
.via(timeoutEach(5.seconds))
.via(disabled)
.via(fast)

View Slide

.via(catchNonFatal)
.via(cached(...))
.via(disabled)
.via(fast)
Would retry effectful tests

View Slide

.via(catchNonFatal)
.via(cached(...))
.via(disabled)
.via(fast)
Would cache pure tests until recompiled

View Slide

.via(catchNonFatal)
.via(cached(...))
.via(disabled)
.via(fast)
Would limit running time of each test in the tree

View Slide

.via(catchNonFatal)
.via(cached(...))
.via(disabled)
.via(fast) Would skip tests with resources / tests with IO

View Slide

.via(catchNonFatal)
.via(cached(...))
.via(disabled)
.via(fast) Would skip tests with resources / tests with IO (or all tests!)

View Slide

Experiment more with the API
Find the right abstractions
Constraints vs liberties
See what's possible
Gather feedback
Find other ridiculous ideas that might make sense
TODO

View Slide

Appendix: what changed since last time I gave this talk
@jdegoes @adamgfraser

View Slide

ZIO Test vs ﬂawless
testM("parallel offers and sequential takes") {
effect.map(l => assert(l.toSet, equalTo(values.toSet)))
}
test("parallel offers and sequential takes") {
effect.map(l => l.toSet shouldBe values.toSet)
}

View Slide

ZIO Test ﬂawless
Purely functional ✅ ✅
Tests as a data structure ✅ ✅
Aspects as functions ✅ ✅

View Slide

ZIO Test ﬂawless
Purely functional ✅ ✅
Tests as a data structure ✅ ✅
Aspects as functions ✅ ✅
Minimal ❌ ✅ (so far!)
Arbitrary nesting of tests ✅ ❌ (yet)
Available now ✅
⏳

View Slide

WIP
Demo time?

View Slide

Thank you
blog.kubukoz.com
@kubukoz
Slides: https://bit.ly/2RXH05N
Code: https://git.io/fj6oc

View Slide

Flawless testing for the functional folks

Flawless testing for the functional folks

Jakub Kozłowski

More Decks by Jakub Kozłowski

Other Decks in Technology

Featured

Transcript