Share
Embed
Connecting the dots - building and structuring a functional application in Scala
Functional programming relies on building programs from orthogonal, composable blocks. That's likely one of the reasons why full-blown application frameworks haven't gained much traction in the functional ecosystem.
However, we still need to structure our code and wire up our applications in a way that lets us keep them modular, testable and simply pleasant to work with - in this talk, we will learn how to do just that!
Using an application that integrates with several third-party services to process data in a streaming fashion, and expose its results to downstream clients, we will walk through the architecture design and testing setup for a functional app on the Typelevel stack.
Jakub Kozłowski
May 05, 2021
Transcript
95 slides
-
CONNECTING THE DOTS BUILDING AND STRUCTURING A FUNCTIONAL APPLICATION IN
SCALA JAKUB KOZŁOWSKI, DISNEY STREAMING YOW! LAMBDA JAM 2021 Photo by Kumiko SHIMIZU on Unsplash -
PROBLEM STATEMENT We want to build an application There are
some sources of data (databases, APIs, event streams) -
PROBLEM STATEMENT We want to build an application There are
some sources of data (databases, APIs, event streams) We need to serve HTTP traffic -
PROBLEM STATEMENT We want to build an application There are
some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally -
PROBLEM STATEMENT We want to build an application There are
some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally We want to do it with FP -
DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def
server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes -
DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def
server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) } -
DEPENDENCY GRAPH This could be us, but the real world
exists... def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) } -
RESOURCES. But we can just kill our application when it
quits, right? -
def getConnection(db: Database): Connection def returnConnection(conn: Connection): Unit def doWork(conn:
Connection): Result TRY-FINALLY -
def getConnection(db: Database): Connection def returnConnection(conn: Connection): Unit def doWork(conn:
Connection): Result TRY-FINALLY val result = { val c = getConnection(db) try doWork(c) finally returnConnection(c) } -
def getConnection(db: Database): IO[Connection] def returnConnection(conn: Connection): IO[Unit] def doWork(conn:
Connection): IO[Result] TRY-FINALLY IN FP val result: IO[Result] = getConnection(db).bracket { c !=> doWork(c) }(returnConnection) -
WHY AM I TALKING ABOUT THIS? def database: Database def
businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes -
WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def
businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] -
WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def
businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = database.flatMap { db !=> val logic = businessLogic(db) server(logic).flatMap { srv !=> backgroundProcesses(logic).map(p !=> (srv, p)) } } -
WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def
businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = for { db !<- database logic = businessLogic(db) srv !<- server(logic) processes !<- backgroundProcesses(logic) } yield (srv, processes) -
HOW IS A BACKGROUND PROCESS A RESOURCE? Watch this space:
yt.kubukoz.com -> "Background processing in functional Scala" playlist -
ALGEBRAS / CAPABILITY TRAITS Tagless Final style Interfaces parameterised by
an effect Capability traits - lawless type classes -
RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over
Sync/Async Implicit or explicit? -
RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over
Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly -
RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over
Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly -
RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over
Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly - multiple instances in app: all explicit -
IMAGE PROCESSING APP CASE STUDY Project Goals Search images from
a datasource by the text on them (OCR) -
IMAGE PROCESSING APP CASE STUDY Project Goals Search images from
a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time -
IMAGE PROCESSING APP CASE STUDY Project Goals Search images from
a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time github.com/kubukoz/dropbox-demo -
TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC
BY-SA 4.0 Or... just sensible architecture. -
TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC
BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic -
TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC
BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API -
TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC
BY-SA 4.0 Jakub Nabrdalik - Hexagonal Architecture in practice https://www.youtube.com/watch?v=sOaS83Ir8Ck Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API -
PROJECT STRUCTURE shared - contains common vocabulary Used by adapters
and core logic imagesource, ocr, indexer - modules root - contains core logic + http module Standard sbt pattern for "main" sources -
OCR MODULE ProcessRunner - capability trait for running system processes
Tesseract - runs a Tesseract process -
OCR MODULE ProcessRunner - capability trait for running system processes
Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages) -
OCR MODULE ProcessRunner - capability trait for running system processes
Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages) TestOCRInstances - contains test fakes for OCR for usage in tests of higher-level components (processes) -
PROCESS RUNNER package com.kubukoz.process trait ProcessRunner[F[_]] { def run(program: List[String]):
Resource[F, ProcessRunner.Running[F]] } object ProcessRunner { def apply[F[_]](implicit F: ProcessRunner[F]): ProcessRunner[F] = F implicit def instance[F[_]: Async]: ProcessRunner[F] = !!... } -
TESSERACT package com.kubukoz.ocr.tesseract private[ocr] trait Tesseract[F[_]] { def decode(input: fs2.Stream[F,
Byte], languages: List[String]): F[String] } object Tesseract { def apply[F[_]](implicit F: Tesseract[F]): Tesseract[F] = F def instance[F[_]: ProcessRunner: Logger: Concurrent](implicit SC: fs2.Compiler[F, F]): Tesseract[F] = !!... } -
package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
} object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR -
package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
} object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... -
package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
} object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) } -
package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
} object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) } def module[F[_]: Concurrent: ProcessRunner: Logger](config: Config): OCR[F] = { implicit val tesseract = Tesseract.instance[F] OCR.tesseractInstance[F](config) } -
package com.kubukoz.ocr object TestOCRInstances { !// decodeText("hello".getBytes) !== "hello" def
simple[F[_]: Functor](implicit SC: fs2.Compiler[F, F]): OCR[F] = _.through(fs2.text.utf8Decode[F]).compile.string.map(DecodedText(_)) } TEST INSTANCE -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]] -
object Application { final case class Config( indexer: Indexer.Config, imageSource:
ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]] server !<- HttpServer.instance[F](config.http) yield server -
A COUPLE GUIDELINES TESTING Test the contract, not the implementation
Prefer fakes over mocks/stubs -
A COUPLE GUIDELINES TESTING Test the contract, not the implementation
Prefer fakes over mocks/stubs Test your fakes with the same suite as the real things -
TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*>
!*> indexer.search("hello").compile.toList -
TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*>
!*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) } -
TESTING Await blogpost for more ;) index.schedule(Path("/hello")) val file =
fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*> !*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) } -
TIPS Use Resource + IO for stateful dependencies Define clear
responsibilities for modules -
TIPS Use Resource + IO for stateful dependencies Define clear
responsibilities for modules Design for replacement -
TIPS Use Resource + IO for stateful dependencies Define clear
responsibilities for modules Design for replacement Look for abstractions -
TIPS Use Resource + IO for stateful dependencies Define clear
responsibilities for modules Design for replacement Look for abstractions Prototype early -
TIPS Use Resource + IO for stateful dependencies Define clear
responsibilities for modules Design for replacement Look for abstractions Prototype early Draw some diagrams, see if you have too many arrows ;) -
LEARN MORE Check out the sources: github.com/kubukoz/dropbox-demo Read "Practical FP
in Scala" by Gabriel Volpe github.com/scala-steward-org/scala-steward github.com/branchtalk-io/backend github.com/kubukoz/spotify-next github.com/pitgull/pitgull leanpub.com/pfp-scala -
THANK YOU 📰 blog.kubukoz.com 🐦 @kubukoz Slides: speakerdeck.com/kubukoz Code: git.io/JONEj
Find me on YouTube! (yt.kubukoz.com)