Run Wild, Run Free!

Run Wild, Run Free!

(A team's journey over Scala's FP emerging patterns)

https://github.com/47deg/run-wild-run-free

Working at a Scala consultancy, I have been privileged to observe diverse developer experiences when learning and putting Scala to practice in real world projects.

Many people trip on different areas, but some are more prominent and seem to cause the most headaches for people adopting Scala. We can fix that.

In this talk, I will present some of these patterns, anti-patterns, and pitfalls we have observed in the wild.

We will discuss some of the architectural and technical decisions we have made to cope with these issues, as well as the importance of Principled Typed Functional Programming for newcomers and our industry as a whole.

The purpose of this talk is to make you empathize with the pains that newcomers to the language face, and what we as a community can do to change this.

We'll go over some code examples that can be problematic and why folks trip on those frequently. We will also cover some of the Functional Programming patterns that may be utilized to achieve more flexible architectures and abstractions for real world applications.

Expect to see code around scala.concurrent.Future pitfalls, exception handling, cats transformer stacks, other cats datatypes, and Free monads.

You may be interested in this talk if:

You want wider spread adoption of FP Scala.
You are a Scala newcomer running into issues with Scala.
You are an FP Scala developer looking for code examples and inspiration around FP solutions to common application architecture problems.

Ad2476bf0540dfaa0fc30cb62c8e07da?s=128

Raúl Raja Martínez

September 12, 2016
Tweet

Transcript

  1. Run Wild, Run Free! A team's journey over Scala's FP

    emerging patterns @raulraja @47deg 2
  2. What is this about? — Code samples illustrating common issues

    folks run into with Scala. Compiled since 2013 with about ~10 teams with 5 - 6 devs per team + many conf conversations. @raulraja @47deg 3
  3. What is this about? A code centric report on: —

    What a few modern Scala based code bases may look like in the wild since 2013. — How some of those codebases may progress toward a more FP style. — Hurdles & caveats Scala devs & FP newcomers face when attempting the above. — Emerging FP Patterns. — An opening discussion of what we, as a community, can do to make newcomers life easier. @raulraja @47deg 4
  4. DISCLAIMER: What it's NOT about! — A critique of your

    server as a function — A critique to [scala.concurrent.Future] or any other Future like impl. @raulraja @47deg 5
  5. Your server as a function import rwrf.utils._ import scala.concurrent.Future type

    Service[Req, Rep] = Req => Future[Rep] A lot of current architectures are based on variations of this. @raulraja @47deg 6
  6. Let's build something type UserId = Long type AddressId =

    Long case class User(userId: UserId, addressId: AddressId) case class Address(addressId: AddressId) @raulraja @47deg 7
  7. Exceptions Purposely unsealed to resemble most exceptions in the Throwable

    hierarchy case class NotFound(msg : String) extends RuntimeException(msg) case class DuplicateFound(msg : String) extends RuntimeException(msg) case class TimeoutException(msg : String) extends RuntimeException(msg) case class HostNotFoundException(msg : String) extends RuntimeException(msg) @raulraja @47deg 8
  8. DB backend def fetchRemoteUser(userId: UserId) : User = throw NotFound(s"user

    not found with id : $userId") def fetchRemoteAddress(addressId: AddressId) : Address = throw DuplicateFound(s"address duplicate found") @raulraja @47deg 9
  9. Services Here is where people start tripping val fetchUser: Service[UserId,

    User] = (userId: UserId) => Future { fetchRemoteUser(userId) } @raulraja @47deg 10
  10. Services Let's try again import scala.concurrent.ExecutionContext.Implicits.global val fetchUser: Service[UserId, User]

    = (userId: UserId) => Future(fetchRemoteUser(userId)) val fetchAddress: Service[AddressId, Address] = (addressId: AddressId) => Future(fetchRemoteAddress(addressId)) val fetchUserInfo: Service[UserId, (User, Address)] = (userId: UserId) => for { user <- fetchUser(userId) address <- fetchAddress(user.addressId) } yield (user, address) @raulraja @47deg 11
  11. Error handling What if something goes wrong? val fetchUserInfo: Service[UserId,

    (User, Address)] = (userId: UserId) => for { user <- fetchUser(userId) //not found address <- fetchAddress(user.addressId) //duplicate found } yield (user, address) @raulraja @47deg 12
  12. Error handling At this point folks branch out and they

    either — Attempt to Future#recover for both known and unforeseen exceptions. — Model known exceptional cases via nested Option, Either, Try... @raulraja @47deg 13
  13. Error handling Those who attempt to recover may succeed val

    result = fetchUserInfo(1L) recover { case _ : NotFound => User(1L, 10L) } @raulraja @47deg 14
  14. Error handling But if you don't know the impl details

    and attempt to recover val result = fetchAddress(1L) recover { case _ : NotFound => Address(1L) } @raulraja @47deg 15
  15. Error handling Recovering becomes a game of guess and match

    fetchAddress(1L) recover { case _ : NotFound => Address(1L) case _ : DuplicateFound => Address(1L) } @raulraja @47deg 16
  16. Error handling What starts as a trivial piece of code

    quickly becomes import scala.util.control._ fetchAddress(1L) recover { case _ : NotFound => ??? case _ : DuplicateFound => ??? case _ : TimeoutException => ??? case _ : HostNotFoundException => ??? case NonFatal(e) => ??? } @raulraja @47deg 17
  17. Error handling With most Future#recover style apps I've seen... —

    Http 500 production errors due to uncaught exceptions — Future#recover is abused for both known/ unforeseen exceptions — Partial Functions + Unsealed hierarchies = Late night fun! @raulraja @47deg 18
  18. Error handling Let's mock the DB again val existingUserId =

    1L val nonExistingUserId = 2L def fetchRemoteUser(userId: UserId) : User = if (userId == existingUserId) User(userId, 10L) else null def fetchRemoteAddress(addressId: AddressId) : Address = Address(addressId) @raulraja @47deg 19
  19. Error handling Folks realize they need safeguards val fetchUser: Service[UserId,

    Option[User]] = (userId: UserId) => Future(Option(fetchRemoteUser(userId))) val fetchAddress: Service[AddressId, Option[Address]] = (addressId: AddressId) => Future(Option(fetchRemoteAddress(addressId))) @raulraja @47deg 20
  20. Error handling The issue of having a nested type quickly

    surfaces val fetchUserInfo: Service[UserId, (User, Address)] = (userId: UserId) => for { user <- fetchUser(userId) address <- fetchAddress(user.addressId) } yield (user, address) @raulraja @47deg 21
  21. Error handling Most folks wrestle with the for comprehension but

    end up doing: val fetchUserInfo: Service[UserId, Option[(User, Address)]] = (userId: UserId) => fetchUser(userId) flatMap { case Some(user) => fetchAddress(user.addressId) flatMap { case Some(address) => Future.successful(Some((user, address))) case None => Future.successful(None) } case None => Future.successful(None) } @raulraja @47deg 22
  22. Error handling PM hands out new requirements... We need ALL

    THE INFO for a User in that endpoint! :O @raulraja @47deg 23
  23. Error handling As new requirements are added val fetchUserInfo: Service[UserId,

    Option[(User, Address, PostalCode, Region, Country)]] = (userId: UserId) => fetchUser(userId) flatMap { case Some(user) => fetchAddress(user.addressId) flatMap { case Some(address) => fetchPostalCode(address.postalCodeId) flatMap { case Some(postalCode) => fetchRegion(postalCode.regionId) flatMap { case Some(region) => fetchCountry(region.countryId) flatMap { case Some(country) => Future.successful(Some((user, address, postalCode, region, country))) case None => Future.successful(None) } case None => Future.successful(None) } case None => Future.successful(None) } case None => Future.successful(None) } case None => Future.successful(None) } @raulraja @47deg 24
  24. Error handling Code starts looking like <img src="../../../../../../../../../../../../../../../../../assets/logo.png" /> @raulraja

    @47deg 25
  25. Error handling Code starts looking like If optCashReportDay.Value = True

    Then DoCashReportDay Else If optCashReportWeek.Value = True Then DoCashReportWeek Else If optCashReportMonth.Value = True Then DoCashReportMonth Else If optCashReportAnnual.Value = True Then DoCashReportAnnual Else If optBondReportDay.Value = True Then DoBondReportDay // Goes on forever.... @raulraja @47deg 26
  26. Error handling At this point it's a game of choosing

    your own adventure: — It compiles and runs, I don't care! (˽°□°҂ ˽Ɨ ˍʓˍ — We should ask for help, things are getting out of control! — Someones says the word ☠ Monad Transformers ☠ @raulraja @47deg 27
  27. Error handling OptionT, EitherT, Validated some of the reasons folks

    get interested in FP in Scala. @raulraja @47deg 28
  28. Error handling Those that survive are happy again that their

    code looks nicer again import cats.data.OptionT import cats.instances.future._ val fetchUserInfo: Service[UserId, Option[(User, Address)]] = (userId: UserId) => { val resT = for { user <- OptionT(fetchUser(userId)) address <- OptionT(fetchAddress(user.addressId)) } yield (user, address) resT.value } val existingUser = fetchUserInfo(existingUserId).await val nonExistingUser = fetchUserInfo(nonExistingUserId).await @raulraja @47deg 29
  29. Error handling As they dig deeper in FP land they

    learn about the importance of ADTs and sealed hierarchies! sealed abstract class AppException(msg : String) extends Product with Serializable final case class NotFound(msg : String) extends AppException(msg) final case class DuplicateFound(msg : String) extends AppException(msg) final case class TimeoutException(msg : String) extends AppException(msg) final case class HostNotFoundException(msg : String) extends AppException(msg) @raulraja @47deg 30
  30. Error handling Exceptional cases start showing up in return types.

    import cats.data.Xor import cats.syntax.xor._ implicit class FutureOptionOps[A](fa : Future[Option[A]]) { def toXor[L](e : L): Future[L Xor A] = fa map (_.fold(e.left[A])(x => x.right[L])) } val fetchUser: Service[UserId, NotFound Xor User] = { (userId: UserId) => Future(Option(fetchRemoteUser(userId))) .toXor(NotFound(s"User $userId not found")) } val fetchAddress: Service[AddressId, NotFound Xor Address] = { (addressId: AddressId) => Future(Option(fetchRemoteAddress(addressId))) .toXor(NotFound(s"Address $addressId not found")) } @raulraja @47deg 31
  31. Error handling They grow beyond OptionT and start using other

    transformers. import cats.data.XorT val fetchUserInfo: Service[UserId, NotFound Xor (User, Address)] = { (userId: UserId) => val resT = for { user <- XorT(fetchUser(userId)) address <- XorT(fetchAddress(user.addressId)) } yield (user, address) resT.value } val existingUser = fetchUserInfo(existingUserId).await val nonExistingUser = fetchUserInfo(nonExistingUserId).await @raulraja @47deg 32
  32. Non determinism Common mistakes when refactoring import org.scalacheck._ import org.scalacheck.Prop.{forAll,

    BooleanOperators} def sideEffect(latency: Int): Future[Long] = Future { Thread.sleep(latency); System.currentTimeMillis } def latencyGen: Gen[(Int, Int)] = for { a <- Gen.choose(10, 100) b <- Gen.choose(10, 100) } yield (a, b) @raulraja @47deg 33
  33. Non determinism Effects are sequentially performed val test = forAll(latencyGen)

    { latency => val ops = for { a <- sideEffect(latency._1) b <- sideEffect(latency._2) } yield (a, b) val (read, write) = ops.await read < write } @raulraja @47deg 34
  34. Non determinism Effects may be executed in the wrong order

    val test = forAll(latencyGen) { latency => val op1 = sideEffect(latency._1) val op2 = sideEffect(latency._2) val ops = for { a <- op1 b <- op2 } yield (a, b) val (read, write) = ops.await read < write } @raulraja @47deg 35
  35. What others things are folks reporting? — Wrong order of

    Effects (When someone recommends moving ) — Random deadlocks (Custom ExecutionContexts) — General confusion as to why most combinators require an implicit EC when one was already provided to Future#apply. — A lot of reusable code becomes non reusable because it's inside a Future @raulraja @47deg 36
  36. Code reuse? Can we make our code work in the

    context of other types beside Future? @raulraja @47deg 37
  37. Abstracting over the return type As it stands our services

    are coupled to Future. type Service[A, B] = A => Future[B] @raulraja @47deg 38
  38. Abstracting over the return type But they don't have to.

    type Service[F[_], A, B] = A => F[B] @raulraja @47deg 39
  39. Abstracting over the return type We just need a way

    to lift a thunk: => A to an F[_] import simulacrum.typeclass @typeclass trait Capture[F[_]] { def capture[A](a: => A): F[A] } @raulraja @47deg 40
  40. Abstracting over the return type We'll add one instance per

    type we are wanting to support import scala.concurrent.ExecutionContext implicit def futureCapture(implicit ec : ExecutionContext) : Capture[Future] = new Capture[Future] { override def capture[A](a: => A): Future[A] = Future(a)(ec) } @raulraja @47deg 41
  41. Abstracting over the return type import monix.eval.Task import monix.cats._ implicit

    val taskCapture : Capture[Task] = new Capture[Task] { override def capture[A](a: => A): Task[A] = Task.evalOnce(a) } @raulraja @47deg 42
  42. Abstracting over the return type import scala.util.Try implicit val tryCapture

    : Capture[Try] = new Capture[Try] { override def capture[A](a: => A): Try[A] = Try(a) } @raulraja @47deg 43
  43. Abstracting over the return type Our services now are parametrized

    to any F[_] for which a Capture instance is found. import cats.{Functor, Foldable, Monoid} import cats.implicits._ implicit class FGOps[F[_]: Functor, G[_]: Foldable, A](fa : F[G[A]]) { def toXor[L](e : L): F[L Xor A] = Functor[F].map(fa) { g => Foldable[G].foldLeft[A, L Xor A](g, e.left[A])((_, b) => b.right[L]) } } @raulraja @47deg 44
  44. Abstracting over the return type Our services now are parametrized

    to any F[_] for which a Capture instance is found. class Services[F[_] : Capture : Functor] { val fetchUser: Service[F, UserId, NotFound Xor User] = { (userId: UserId) => Capture[F] .capture(Option(fetchRemoteUser(userId))) .toXor(NotFound(s"User $userId not found")) } val fetchAddress: Service[F, AddressId, NotFound Xor Address] = { (addressId: AddressId) => Capture[F] .capture(Option(fetchRemoteAddress(addressId))) .toXor(NotFound(s"Address $addressId not found")) } } object Services { import cats._ def apply[F[_] : Capture: Monad: RecursiveTailRecM] : Services[F] = new Services[F] implicit def instance[F[_]: Capture: Monad: RecursiveTailRecM]: Services[F] = apply[F] } @raulraja @47deg 45
  45. Abstracting over the return type Code becomes reusable regardless of

    the target runtime val futureS = Services[Future].fetchUser(existingUserId) val taskS = Services[Task].fetchUser(existingUserId) val tryS = Services[Try].fetchUser(existingUserId) @raulraja @47deg 46
  46. Abstracting over the return type Other alternatives available (Rapture Modes

    and Result) @raulraja @47deg 47
  47. Abstracting over implementations Now that we can run our code

    to any F[_] that can capture a lazy computation we may want to abstract over implementations too. @raulraja @47deg 48
  48. Abstracting over implementations Free Monads / Applicatives is what has

    worked best for us. — Free of interpretation allowing multiple runtimes. — Composable via Coproduct. — Lots of boilerplate that can be improved. — Supports abstracting over return types. — Getting momentum with multiple posts and libs supporting the pattern. (Freek, cats, ...) @raulraja @47deg 49
  49. Abstracting over implementations Let's refactor our services to run on

    Free? @raulraja @47deg 50
  50. Abstracting over implementations Define your Algebra sealed abstract class ServiceOp[A]

    extends Product with Serializable final case class FetchUser(userId: UserId) extends ServiceOp[NotFound Xor User] final case class FetchAddress(addressId: AddressId) extends ServiceOp[NotFound Xor Address] @raulraja @47deg 51
  51. Abstracting over implementations Lift your Algebra to Free import cats.free.Free

    type ServiceIO[A] = Free[ServiceOp, A] object ServiceOps { def fetchUser(userId: UserId): ServiceIO[NotFound Xor User] = Free.liftF(FetchUser(userId)) def fetchAddress(addressId: AddressId): ServiceIO[NotFound Xor Address] = Free.liftF(FetchAddress(addressId)) } @raulraja @47deg 52
  52. Abstracting over implementations Write 1 or many interpreters that may

    be swapped at runtime import cats._ import cats.implicits._ import Services._ def interpreter[M[_] : Capture : Monad : RecursiveTailRecM] (implicit impl: Services[M]): ServiceOp ~> M = new (ServiceOp ~> M) { override def apply[A](fa: ServiceOp[A]): M[A] = { val result = fa match { case FetchUser(userId) => impl.fetchUser(userId) case FetchAddress(addressId) => impl.fetchAddress(addressId) } result.asInstanceOf[M[A]] } } @raulraja @47deg 53
  53. Abstracting over implementations Write programs using the smart constructors and

    combining them at will def fetchUserInfo(userId: UserId): ServiceIO[NotFound Xor (User, Address)] = { val resT = for { user <- XorT(ServiceOps.fetchUser(userId)) address <- XorT(ServiceOps.fetchAddress(user.addressId)) } yield (user, address) resT.value } @raulraja @47deg 54
  54. Abstracting over implementations Run your programs to any target implementation

    and runtime val tryResult = fetchUserInfo(existingUserId).foldMap(interpreter[Try]) val taskResult = fetchUserInfo(existingUserId).foldMap(interpreter[Task]) val futureResult = fetchUserInfo(existingUserId).foldMap(interpreter[Future]) @raulraja @47deg 55
  55. Patterns Recommendations for others that have worked for us: —

    Algebraic design and sealed hierarchies for safer exceptions control. (Don't match and Guess). — Abstract over return types for code reuse. — Abstract over implementations to increase flexibility and composition. @raulraja @47deg 56
  56. Conclusion — Most Scala newcomers are NOT exposed to Typed

    FP when they start. — There are repeating patterns of failure and frustration among newcommers. — Scala is not a Functional Programming Language but we can make it. — Expose newcomers to Scala to typed FP for a brither Future @raulraja @47deg 57
  57. Wishes for the Future — Make Scala more FP friendly.

    @raulraja @47deg https://speakerdeck.com/raulraja/run-wild-run-free @raulraja @47deg 58
  58. Acknowledgments — 47 Degrees — Background photo — Algebraic Data

    Types — Cats — Rapture — scala.concurrent.Future — Your server as a function @raulraja @47deg 59