Run Wild, Run Free!

(A team's journey over Scala's FP emerging patterns)

https://github.com/47deg/run-wild-run-free

Working at a Scala consultancy, I have been privileged to observe diverse developer experiences when learning and putting Scala to practice in real world projects.

Many people trip on different areas, but some are more prominent and seem to cause the most headaches for people adopting Scala. We can fix that.

In this talk, I will present some of these patterns, anti-patterns, and pitfalls we have observed in the wild.

We will discuss some of the architectural and technical decisions we have made to cope with these issues, as well as the importance of Principled Typed Functional Programming for newcomers and our industry as a whole.

The purpose of this talk is to make you empathize with the pains that newcomers to the language face, and what we as a community can do to change this.

We'll go over some code examples that can be problematic and why folks trip on those frequently. We will also cover some of the Functional Programming patterns that may be utilized to achieve more flexible architectures and abstractions for real world applications.

Expect to see code around scala.concurrent.Future pitfalls, exception handling, cats transformer stacks, other cats datatypes, and Free monads.

You may be interested in this talk if:

You want wider spread adoption of FP Scala.
You are a Scala newcomer running into issues with Scala.
You are an FP Scala developer looking for code examples and inspiration around FP solutions to common application architecture problems.

Transcript

1. Run Wild, Run Free!
   A team's journey over Scala's FP emerging patterns
   @raulraja @47deg 2
   

   View Slide

2. What is this about?
   — Code samples illustrating common issues folks run
   into with Scala.
   Compiled since 2013 with about ~10 teams with 5 - 6
   devs per team
   + many conf conversations.
   @raulraja @47deg 3
   

   View Slide

3. What is this about?
   A code centric report on:
   — What a few modern Scala based code bases may look like in the wild
   since 2013.
   — How some of those codebases may progress toward a more FP style.
   — Hurdles & caveats Scala devs & FP newcomers face when attempting
   the above.
   — Emerging FP Patterns.
   — An opening discussion of what we, as a community, can do to make
   newcomers life easier.
   @raulraja @47deg 4
   

   View Slide

4. DISCLAIMER: What it's NOT about!
   — A critique of your server as a function
   — A critique to [scala.concurrent.Future] or any
   other Future like impl.
   @raulraja @47deg 5
   

   View Slide

5. Your server as a function
   import rwrf.utils._
   import scala.concurrent.Future
   type Service[Req, Rep] = Req => Future[Rep]
   A lot of current architectures are based on
   variations of this.
   @raulraja @47deg 6
   

   View Slide

6. Let's build something
   type UserId = Long
   type AddressId = Long
   case class User(userId: UserId, addressId: AddressId)
   case class Address(addressId: AddressId)
   @raulraja @47deg 7
   

   View Slide

7. Exceptions
   Purposely unsealed to resemble most exceptions in
   the Throwable hierarchy
   case class NotFound(msg : String) extends RuntimeException(msg)
   case class DuplicateFound(msg : String) extends RuntimeException(msg)
   case class TimeoutException(msg : String) extends RuntimeException(msg)
   case class HostNotFoundException(msg : String) extends RuntimeException(msg)
   @raulraja @47deg 8
   

   View Slide

8. DB backend
   def fetchRemoteUser(userId: UserId) : User =
   throw NotFound(s"user not found with id : $userId")
   def fetchRemoteAddress(addressId: AddressId) : Address =
   throw DuplicateFound(s"address duplicate found")
   @raulraja @47deg 9
   

   View Slide

9. Services
   Here is where people start tripping
   val fetchUser: Service[UserId, User] =
   (userId: UserId) => Future {
   fetchRemoteUser(userId)
   }
   @raulraja @47deg 10
   

   View Slide

10. Services
    Let's try again
    import scala.concurrent.ExecutionContext.Implicits.global
    val fetchUser: Service[UserId, User] =
    (userId: UserId) => Future(fetchRemoteUser(userId))
    val fetchAddress: Service[AddressId, Address] =
    (addressId: AddressId) => Future(fetchRemoteAddress(addressId))
    val fetchUserInfo: Service[UserId, (User, Address)] =
    (userId: UserId) =>
    for {
    user <- fetchUser(userId)
    address <- fetchAddress(user.addressId)
    } yield (user, address)
    @raulraja @47deg 11
    

    View Slide

11. Error handling
    What if something goes wrong?
    val fetchUserInfo: Service[UserId, (User, Address)] =
    (userId: UserId) =>
    for {
    user <- fetchUser(userId) //not found
    address <- fetchAddress(user.addressId) //duplicate found
    } yield (user, address)
    @raulraja @47deg 12
    

    View Slide

12. Error handling
    At this point folks branch out and they either
    — Attempt to Future#recover for both known and
    unforeseen exceptions.
    — Model known exceptional cases via nested Option,
    Either, Try...
    @raulraja @47deg 13
    

    View Slide

13. Error handling
    Those who attempt to recover may succeed
    val result = fetchUserInfo(1L) recover {
    case _ : NotFound => User(1L, 10L)
    }
    @raulraja @47deg 14
    

    View Slide

14. Error handling
    But if you don't know the impl details and attempt
    to recover
    val result = fetchAddress(1L) recover {
    case _ : NotFound => Address(1L)
    }
    @raulraja @47deg 15
    

    View Slide

15. Error handling
    Recovering becomes a game of guess and match
    fetchAddress(1L) recover {
    case _ : NotFound => Address(1L)
    case _ : DuplicateFound => Address(1L)
    }
    @raulraja @47deg 16
    

    View Slide

16. Error handling
    What starts as a trivial piece of code quickly becomes
    import scala.util.control._
    fetchAddress(1L) recover {
    case _ : NotFound => ???
    case _ : DuplicateFound => ???
    case _ : TimeoutException => ???
    case _ : HostNotFoundException => ???
    case NonFatal(e) => ???
    }
    @raulraja @47deg 17
    

    View Slide

17. Error handling
    With most Future#recover style apps I've seen...
    — Http 500 production errors due to uncaught
    exceptions
    — Future#recover is abused for both known/
    unforeseen exceptions
    — Partial Functions + Unsealed hierarchies = Late
    night fun!
    @raulraja @47deg 18
    

    View Slide

18. Error handling
    Let's mock the DB again
    val existingUserId = 1L
    val nonExistingUserId = 2L
    def fetchRemoteUser(userId: UserId) : User =
    if (userId == existingUserId) User(userId, 10L) else null
    def fetchRemoteAddress(addressId: AddressId) : Address =
    Address(addressId)
    @raulraja @47deg 19
    

    View Slide

19. Error handling
    Folks realize they need safeguards
    val fetchUser: Service[UserId, Option[User]] =
    (userId: UserId) => Future(Option(fetchRemoteUser(userId)))
    val fetchAddress: Service[AddressId, Option[Address]] =
    (addressId: AddressId) => Future(Option(fetchRemoteAddress(addressId)))
    @raulraja @47deg 20
    

    View Slide

20. Error handling
    The issue of having a nested type quickly surfaces
    val fetchUserInfo: Service[UserId, (User, Address)] =
    (userId: UserId) =>
    for {
    user <- fetchUser(userId)
    address <- fetchAddress(user.addressId)
    } yield (user, address)
    @raulraja @47deg 21
    

    View Slide

21. Error handling
    Most folks wrestle with the for comprehension but end up
    doing:
    val fetchUserInfo: Service[UserId, Option[(User, Address)]] =
    (userId: UserId) =>
    fetchUser(userId) flatMap {
    case Some(user) => fetchAddress(user.addressId) flatMap {
    case Some(address) => Future.successful(Some((user, address)))
    case None => Future.successful(None)
    }
    case None => Future.successful(None)
    }
    @raulraja @47deg 22
    

    View Slide

22. Error handling
    PM hands out new requirements...
    We need ALL THE INFO for a User in that endpoint! :O
    @raulraja @47deg 23
    

    View Slide

23. Error handling
    As new requirements are added
    val fetchUserInfo: Service[UserId, Option[(User, Address, PostalCode, Region, Country)]] =
    (userId: UserId) =>
    fetchUser(userId) flatMap {
    case Some(user) => fetchAddress(user.addressId) flatMap {
    case Some(address) => fetchPostalCode(address.postalCodeId) flatMap {
    case Some(postalCode) => fetchRegion(postalCode.regionId) flatMap {
    case Some(region) => fetchCountry(region.countryId) flatMap {
    case Some(country) =>
    Future.successful(Some((user, address, postalCode, region, country)))
    case None => Future.successful(None)
    }
    case None => Future.successful(None)
    }
    case None => Future.successful(None)
    }
    case None => Future.successful(None)
    }
    case None => Future.successful(None)
    }
    @raulraja @47deg 24
    

    View Slide

24. Error handling
    Code starts looking like
    
    @raulraja @47deg 25
    

    View Slide

25. Error handling
    Code starts looking like
    If optCashReportDay.Value = True Then
    DoCashReportDay
    Else
    If optCashReportWeek.Value = True Then
    DoCashReportWeek
    Else
    If optCashReportMonth.Value = True Then
    DoCashReportMonth
    Else
    If optCashReportAnnual.Value = True Then
    DoCashReportAnnual
    Else
    If optBondReportDay.Value = True Then
    DoBondReportDay
    // Goes on forever....
    @raulraja @47deg 26
    

    View Slide

26. Error handling
    At this point it's a game of choosing your own
    adventure:
    — It compiles and runs, I don't care! (˽°□°҂
    ˽Ɨ ˍʓˍ
    — We should ask for help, things are getting out of
    control!
    — Someones says the word ☠ Monad Transformers ☠
    @raulraja @47deg 27
    

    View Slide

27. Error handling
    OptionT, EitherT, Validated some of the reasons
    folks get interested in FP in Scala.
    @raulraja @47deg 28
    

    View Slide

28. Error handling
    Those that survive are happy again that their code looks nicer again
    import cats.data.OptionT
    import cats.instances.future._
    val fetchUserInfo: Service[UserId, Option[(User, Address)]] =
    (userId: UserId) => {
    val resT = for {
    user <- OptionT(fetchUser(userId))
    address <- OptionT(fetchAddress(user.addressId))
    } yield (user, address)
    resT.value
    }
    val existingUser = fetchUserInfo(existingUserId).await
    val nonExistingUser = fetchUserInfo(nonExistingUserId).await
    @raulraja @47deg 29
    

    View Slide

29. Error handling
    As they dig deeper in FP land they learn about the
    importance of ADTs and sealed hierarchies!
    sealed abstract class AppException(msg : String) extends Product with Serializable
    final case class NotFound(msg : String) extends AppException(msg)
    final case class DuplicateFound(msg : String) extends AppException(msg)
    final case class TimeoutException(msg : String) extends AppException(msg)
    final case class HostNotFoundException(msg : String) extends AppException(msg)
    @raulraja @47deg 30
    

    View Slide

30. Error handling
    Exceptional cases start showing up in return types.
    import cats.data.Xor
    import cats.syntax.xor._
    implicit class FutureOptionOps[A](fa : Future[Option[A]]) {
    def toXor[L](e : L): Future[L Xor A] =
    fa map (_.fold(e.left[A])(x => x.right[L]))
    }
    val fetchUser: Service[UserId, NotFound Xor User] = {
    (userId: UserId) =>
    Future(Option(fetchRemoteUser(userId)))
    .toXor(NotFound(s"User $userId not found"))
    }
    val fetchAddress: Service[AddressId, NotFound Xor Address] = {
    (addressId: AddressId) =>
    Future(Option(fetchRemoteAddress(addressId)))
    .toXor(NotFound(s"Address $addressId not found"))
    }
    @raulraja @47deg 31
    

    View Slide

31. Error handling
    They grow beyond OptionT and start using other transformers.
    import cats.data.XorT
    val fetchUserInfo: Service[UserId, NotFound Xor (User, Address)] = {
    (userId: UserId) =>
    val resT = for {
    user <- XorT(fetchUser(userId))
    address <- XorT(fetchAddress(user.addressId))
    } yield (user, address)
    resT.value
    }
    val existingUser = fetchUserInfo(existingUserId).await
    val nonExistingUser = fetchUserInfo(nonExistingUserId).await
    @raulraja @47deg 32
    

    View Slide

32. Non determinism
    Common mistakes when refactoring
    import org.scalacheck._
    import org.scalacheck.Prop.{forAll, BooleanOperators}
    def sideEffect(latency: Int): Future[Long] =
    Future { Thread.sleep(latency); System.currentTimeMillis }
    def latencyGen: Gen[(Int, Int)] = for {
    a <- Gen.choose(10, 100)
    b <- Gen.choose(10, 100)
    } yield (a, b)
    @raulraja @47deg 33
    

    View Slide

33. Non determinism
    Effects are sequentially performed
    val test = forAll(latencyGen) { latency =>
    val ops = for {
    a <- sideEffect(latency._1)
    b <- sideEffect(latency._2)
    } yield (a, b)
    val (read, write) = ops.await
    read < write
    }
    @raulraja @47deg 34
    

    View Slide

34. Non determinism
    Effects may be executed in the wrong order
    val test = forAll(latencyGen) { latency =>
    val op1 = sideEffect(latency._1)
    val op2 = sideEffect(latency._2)
    val ops = for {
    a <- op1
    b <- op2
    } yield (a, b)
    val (read, write) = ops.await
    read < write
    }
    @raulraja @47deg 35
    

    View Slide

35. What others things are folks reporting?
    — Wrong order of Effects (When someone recommends
    moving )
    — Random deadlocks (Custom ExecutionContexts)
    — General confusion as to why most combinators require
    an implicit EC
    when one was already provided to Future#apply.
    — A lot of reusable code becomes non reusable because
    it's inside a Future
    @raulraja @47deg 36
    

    View Slide

36. Code reuse?
    Can we make our code work in the context of other
    types beside Future?
    @raulraja @47deg 37
    

    View Slide

37. Abstracting over the return type
    As it stands our services are coupled to Future.
    type Service[A, B] = A => Future[B]
    @raulraja @47deg 38
    

    View Slide

38. Abstracting over the return type
    But they don't have to.
    type Service[F[_], A, B] = A => F[B]
    @raulraja @47deg 39
    

    View Slide

39. Abstracting over the return type
    We just need a way to lift a thunk: => A to an F[_]
    import simulacrum.typeclass
    @typeclass trait Capture[F[_]] {
    def capture[A](a: => A): F[A]
    }
    @raulraja @47deg 40
    

    View Slide

40. Abstracting over the return type
    We'll add one instance per type we are wanting to
    support
    import scala.concurrent.ExecutionContext
    implicit def futureCapture(implicit ec : ExecutionContext) : Capture[Future] =
    new Capture[Future] {
    override def capture[A](a: => A): Future[A] = Future(a)(ec)
    }
    @raulraja @47deg 41
    

    View Slide

41. Abstracting over the return type
    import monix.eval.Task
    import monix.cats._
    implicit val taskCapture : Capture[Task] =
    new Capture[Task] {
    override def capture[A](a: => A): Task[A] = Task.evalOnce(a)
    }
    @raulraja @47deg 42
    

    View Slide

42. Abstracting over the return type
    import scala.util.Try
    implicit val tryCapture : Capture[Try] =
    new Capture[Try] {
    override def capture[A](a: => A): Try[A] = Try(a)
    }
    @raulraja @47deg 43
    

    View Slide

43. Abstracting over the return type
    Our services now are parametrized to any F[_] for which a
    Capture
    instance is found.
    import cats.{Functor, Foldable, Monoid}
    import cats.implicits._
    implicit class FGOps[F[_]: Functor, G[_]: Foldable, A](fa : F[G[A]]) {
    def toXor[L](e : L): F[L Xor A] =
    Functor[F].map(fa) { g =>
    Foldable[G].foldLeft[A, L Xor A](g, e.left[A])((_, b) => b.right[L])
    }
    }
    @raulraja @47deg 44
    

    View Slide

44. Abstracting over the return type
    Our services now are parametrized to any F[_] for which a Capture
    instance is found.
    class Services[F[_] : Capture : Functor] {
    val fetchUser: Service[F, UserId, NotFound Xor User] = {
    (userId: UserId) =>
    Capture[F]
    .capture(Option(fetchRemoteUser(userId)))
    .toXor(NotFound(s"User $userId not found"))
    }
    val fetchAddress: Service[F, AddressId, NotFound Xor Address] = {
    (addressId: AddressId) =>
    Capture[F]
    .capture(Option(fetchRemoteAddress(addressId)))
    .toXor(NotFound(s"Address $addressId not found"))
    }
    }
    object Services {
    import cats._
    def apply[F[_] : Capture: Monad: RecursiveTailRecM] : Services[F] = new Services[F]
    implicit def instance[F[_]: Capture: Monad: RecursiveTailRecM]: Services[F] = apply[F]
    }
    @raulraja @47deg 45
    

    View Slide

45. Abstracting over the return type
    Code becomes reusable regardless of the target
    runtime
    val futureS = Services[Future].fetchUser(existingUserId)
    val taskS = Services[Task].fetchUser(existingUserId)
    val tryS = Services[Try].fetchUser(existingUserId)
    @raulraja @47deg 46
    

    View Slide

46. Abstracting over the return type
    Other alternatives available (Rapture Modes and
    Result)
    @raulraja @47deg 47
    

    View Slide

47. Abstracting over implementations
    Now that we can run our code to any F[_] that can
    capture a lazy computation we may want to abstract
    over
    implementations too.
    @raulraja @47deg 48
    

    View Slide

48. Abstracting over implementations
    Free Monads / Applicatives is what has worked best for us.
    — Free of interpretation allowing multiple runtimes.
    — Composable via Coproduct.
    — Lots of boilerplate that can be improved.
    — Supports abstracting over return types.
    — Getting momentum with multiple posts and libs supporting
    the pattern.
    (Freek, cats, ...)
    @raulraja @47deg 49
    

    View Slide

49. Abstracting over implementations
    Let's refactor our services to run on Free?
    @raulraja @47deg 50
    

    View Slide

50. Abstracting over implementations
    Define your Algebra
    sealed abstract class ServiceOp[A] extends Product with Serializable
    final case class FetchUser(userId: UserId) extends ServiceOp[NotFound Xor User]
    final case class FetchAddress(addressId: AddressId) extends ServiceOp[NotFound Xor Address]
    @raulraja @47deg 51
    

    View Slide

51. Abstracting over implementations
    Lift your Algebra to Free
    import cats.free.Free
    type ServiceIO[A] = Free[ServiceOp, A]
    object ServiceOps {
    def fetchUser(userId: UserId): ServiceIO[NotFound Xor User] =
    Free.liftF(FetchUser(userId))
    def fetchAddress(addressId: AddressId): ServiceIO[NotFound Xor Address] =
    Free.liftF(FetchAddress(addressId))
    }
    @raulraja @47deg 52
    

    View Slide

52. Abstracting over implementations
    Write 1 or many interpreters that may be swapped at runtime
    import cats._
    import cats.implicits._
    import Services._
    def interpreter[M[_] : Capture : Monad : RecursiveTailRecM]
    (implicit impl: Services[M]): ServiceOp ~> M = new (ServiceOp ~> M) {
    override def apply[A](fa: ServiceOp[A]): M[A] = {
    val result = fa match {
    case FetchUser(userId) => impl.fetchUser(userId)
    case FetchAddress(addressId) => impl.fetchAddress(addressId)
    }
    result.asInstanceOf[M[A]]
    }
    }
    @raulraja @47deg 53
    

    View Slide

53. Abstracting over implementations
    Write programs using the smart constructors and
    combining them at will
    def fetchUserInfo(userId: UserId): ServiceIO[NotFound Xor (User, Address)] = {
    val resT = for {
    user <- XorT(ServiceOps.fetchUser(userId))
    address <- XorT(ServiceOps.fetchAddress(user.addressId))
    } yield (user, address)
    resT.value
    }
    @raulraja @47deg 54
    

    View Slide

54. Abstracting over implementations
    Run your programs to any target implementation and
    runtime
    val tryResult = fetchUserInfo(existingUserId).foldMap(interpreter[Try])
    val taskResult = fetchUserInfo(existingUserId).foldMap(interpreter[Task])
    val futureResult = fetchUserInfo(existingUserId).foldMap(interpreter[Future])
    @raulraja @47deg 55
    

    View Slide

55. Patterns
    Recommendations for others that have worked for us:
    — Algebraic design and sealed hierarchies for safer
    exceptions control.
    (Don't match and Guess).
    — Abstract over return types for code reuse.
    — Abstract over implementations to increase
    flexibility and composition.
    @raulraja @47deg 56
    

    View Slide

56. Conclusion
    — Most Scala newcomers are NOT exposed to Typed FP
    when they start.
    — There are repeating patterns of failure and
    frustration among newcommers.
    — Scala is not a Functional Programming Language but
    we can make it.
    — Expose newcomers to Scala to typed FP for a brither
    Future
    @raulraja @47deg 57
    

    View Slide

57. Wishes for the Future
    — Make Scala more FP friendly.
    @raulraja @47deg
    https://speakerdeck.com/raulraja/run-wild-run-free
    @raulraja @47deg 58
    

    View Slide

58. Acknowledgments
    — 47 Degrees
    — Background photo
    — Algebraic Data Types
    — Cats
    — Rapture
    — scala.concurrent.Future
    — Your server as a function
    @raulraja @47deg 59
    

    View Slide