Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Connecting the dots - building and structuring ...

Connecting the dots - building and structuring a functional application in Scala

Functional programming relies on building programs from orthogonal, composable blocks. That's likely one of the reasons why full-blown application frameworks haven't gained much traction in the functional ecosystem.

However, we still need to structure our code and wire up our applications in a way that lets us keep them modular, testable and simply pleasant to work with - in this talk, we will learn how to do just that!

Using an application that integrates with several third-party services to process data in a streaming fashion, and expose its results to downstream clients, we will walk through the architecture design and testing setup for a functional app on the Typelevel stack.

Jakub Kozłowski

May 05, 2021
Tweet

More Decks by Jakub Kozłowski

Other Decks in Programming

Transcript

  1. CONNECTING THE DOTS BUILDING AND STRUCTURING A FUNCTIONAL APPLICATION IN

    SCALA JAKUB KOZŁOWSKI, DISNEY STREAMING YOW! LAMBDA JAM 2021 Photo by Kumiko SHIMIZU on Unsplash
  2. PROBLEM STATEMENT We want to build an application There are

    some sources of data (databases, APIs, event streams)
  3. PROBLEM STATEMENT We want to build an application There are

    some sources of data (databases, APIs, event streams) We need to serve HTTP traffic
  4. PROBLEM STATEMENT We want to build an application There are

    some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally
  5. PROBLEM STATEMENT We want to build an application There are

    some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally We want to do it with FP
  6. DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def

    server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes
  7. DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def

    server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) }
  8. DEPENDENCY GRAPH This could be us, but the real world

    exists... def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) }
  9. def getConnection(db: Database): Connection def returnConnection(conn: Connection): Unit def doWork(conn:

    Connection): Result TRY-FINALLY val result = { val c = getConnection(db) try doWork(c) finally returnConnection(c) }
  10. def getConnection(db: Database): IO[Connection] def returnConnection(conn: Connection): IO[Unit] def doWork(conn:

    Connection): IO[Result] TRY-FINALLY IN FP val result: IO[Result] = getConnection(db).bracket { c !=> doWork(c) }(returnConnection)
  11. WHY AM I TALKING ABOUT THIS? def database: Database def

    businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes
  12. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def

    businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes]
  13. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def

    businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = database.flatMap { db !=> val logic = businessLogic(db) server(logic).flatMap { srv !=> backgroundProcesses(logic).map(p !=> (srv, p)) } }
  14. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def

    businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = for { db !<- database logic = businessLogic(db) srv !<- server(logic) processes !<- backgroundProcesses(logic) } yield (srv, processes)
  15. HOW IS A BACKGROUND PROCESS A RESOURCE? Watch this space:

    yt.kubukoz.com -> "Background processing in functional Scala" playlist
  16. ALGEBRAS / CAPABILITY TRAITS Tagless Final style Interfaces parameterised by

    an effect Capability traits - lawless type classes
  17. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over

    Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly
  18. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over

    Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly
  19. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over

    Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly - multiple instances in app: all explicit
  20. IMAGE PROCESSING APP CASE STUDY Project Goals Search images from

    a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time
  21. IMAGE PROCESSING APP CASE STUDY Project Goals Search images from

    a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time github.com/kubukoz/dropbox-demo
  22. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC

    BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic
  23. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC

    BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API
  24. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC

    BY-SA 4.0 Jakub Nabrdalik - Hexagonal Architecture in practice https://www.youtube.com/watch?v=sOaS83Ir8Ck Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API
  25. PROJECT STRUCTURE shared - contains common vocabulary Used by adapters

    and core logic imagesource, ocr, indexer - modules root - contains core logic + http module Standard sbt pattern for "main" sources
  26. OCR MODULE ProcessRunner - capability trait for running system processes

    Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages)
  27. OCR MODULE ProcessRunner - capability trait for running system processes

    Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages) TestOCRInstances - contains test fakes for OCR for usage in tests of higher-level components (processes)
  28. PROCESS RUNNER package com.kubukoz.process trait ProcessRunner[F[_]] { def run(program: List[String]):

    Resource[F, ProcessRunner.Running[F]] } object ProcessRunner { def apply[F[_]](implicit F: ProcessRunner[F]): ProcessRunner[F] = F implicit def instance[F[_]: Async]: ProcessRunner[F] = !!... }
  29. TESSERACT package com.kubukoz.ocr.tesseract private[ocr] trait Tesseract[F[_]] { def decode(input: fs2.Stream[F,

    Byte], languages: List[String]): F[String] } object Tesseract { def apply[F[_]](implicit F: Tesseract[F]): Tesseract[F] = F def instance[F[_]: ProcessRunner: Logger: Concurrent](implicit SC: fs2.Compiler[F, F]): Tesseract[F] = !!... }
  30. OCR

  31. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]

    } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR
  32. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]

    } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!...
  33. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]

    } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) }
  34. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]

    } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) } def module[F[_]: Concurrent: ProcessRunner: Logger](config: Config): OCR[F] = { implicit val tesseract = Tesseract.instance[F] OCR.tesseractInstance[F](config) }
  35. package com.kubukoz.ocr object TestOCRInstances { !// decodeText("hello".getBytes) !== "hello" def

    simple[F[_]: Functor](implicit SC: fs2.Compiler[F, F]): OCR[F] = _.through(fs2.text.utf8Decode[F]).compile.string.map(DecodedText(_)) } TEST INSTANCE
  36. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) }
  37. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config)
  38. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] }
  39. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]
  40. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue)
  41. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]]
  42. object Application { final case class Config( indexer: Indexer.Config, imageSource:

    ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]] server !<- HttpServer.instance[F](config.http) yield server
  43. A COUPLE GUIDELINES TESTING Test the contract, not the implementation

    Prefer fakes over mocks/stubs Test your fakes with the same suite as the real things
  44. TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*>

    !*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) }
  45. TESTING Await blogpost for more ;) index.schedule(Path("/hello")) val file =

    fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*> !*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) }
  46. TIPS Use Resource + IO for stateful dependencies Define clear

    responsibilities for modules Design for replacement
  47. TIPS Use Resource + IO for stateful dependencies Define clear

    responsibilities for modules Design for replacement Look for abstractions
  48. TIPS Use Resource + IO for stateful dependencies Define clear

    responsibilities for modules Design for replacement Look for abstractions Prototype early
  49. TIPS Use Resource + IO for stateful dependencies Define clear

    responsibilities for modules Design for replacement Look for abstractions Prototype early Draw some diagrams, see if you have too many arrows ;)
  50. LEARN MORE Check out the sources: github.com/kubukoz/dropbox-demo Read "Practical FP

    in Scala" by Gabriel Volpe github.com/scala-steward-org/scala-steward github.com/branchtalk-io/backend github.com/kubukoz/spotify-next github.com/pitgull/pitgull leanpub.com/pfp-scala