Connecting the dots - building and structuring a functional application in Scala

Functional programming relies on building programs from orthogonal, composable blocks. That's likely one of the reasons why full-blown application frameworks haven't gained much traction in the functional ecosystem.

However, we still need to structure our code and wire up our applications in a way that lets us keep them modular, testable and simply pleasant to work with - in this talk, we will learn how to do just that!

Using an application that integrates with several third-party services to process data in a streaming fashion, and expose its results to downstream clients, we will walk through the architecture design and testing setup for a functional app on the Typelevel stack.

Transcript

1. CONNECTING THE DOTS BUILDING AND STRUCTURING A FUNCTIONAL APPLICATION IN SCALA JAKUB KOZŁOWSKI, DISNEY STREAMING YOW! LAMBDA JAM 2021 Photo by Kumiko SHIMIZU on Unsplash
2. PROBLEM STATEMENT
3. PROBLEM STATEMENT We want to build an application
4. PROBLEM STATEMENT We want to build an application There are some sources of data (databases, APIs, event streams)
5. PROBLEM STATEMENT We want to build an application There are some sources of data (databases, APIs, event streams) We need to serve HTTP traffic
6. PROBLEM STATEMENT We want to build an application There are some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally
7. PROBLEM STATEMENT We want to build an application There are some sources of data (databases, APIs, event streams) We need to serve HTTP traffic Some things need to run in the background additionally We want to do it with FP
8. DEPENDENCY GRAPH TYPICAL APPLICATION
9. DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes
10. DEPENDENCY GRAPH def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) }
11. DEPENDENCY GRAPH This could be us, but the real world exists... def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes def build: (Server, Processes) = { val logic = businessLogic(database) (server(logic), backgroundProcesses(logic)) }
12. RESOURCES.
13. RESOURCES. But we can just kill our application when it quits, right?
14. CONNECTION POOLS?
15. CONNECTION POOLS? But we can use try-finally, right?
16. def getConnection(db: Database): Connection def returnConnection(conn: Connection): Unit def doWork(conn: Connection): Result TRY-FINALLY
17. def getConnection(db: Database): Connection def returnConnection(conn: Connection): Unit def doWork(conn: Connection): Result TRY-FINALLY val result = { val c = getConnection(db) try doWork(c) finally returnConnection(c) }
18. def getConnection(db: Database): IO[Connection] def returnConnection(conn: Connection): IO[Unit] def doWork(conn: Connection): IO[Result] TRY-FINALLY IN FP val result: IO[Result] = getConnection(db).bracket { c !=> doWork(c) }(returnConnection)
19. MORE RESOURCES? Can't keep nesting bracket forever
20. MORE RESOURCES? Can't keep nesting bracket forever ABSTRACTION? How to hide details?
21. RESOURCE DATA STRUCTURE
22. RUNNING A RESOURCE?
23. RUNNING A RESOURCE?
24. COMPOSITION?
25. WHY AM I TALKING ABOUT THIS? def database: Database def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Server def backgroundProcesses(logic: BusinessLogic): Processes
26. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes]
27. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = database.flatMap { db !=> val logic = businessLogic(db) server(logic).flatMap { srv !=> backgroundProcesses(logic).map(p !=> (srv, p)) } }
28. WHY AM I TALKING ABOUT THIS? def database: Resource[Database] def businessLogic(db: Database): BusinessLogic def server(logic: BusinessLogic): Resource[Server] def backgroundProcesses(logic: BusinessLogic): Resource[Processes] def build: Resource[(Server, Processes)] = for { db !<- database logic = businessLogic(db) srv !<- server(logic) processes !<- backgroundProcesses(logic) } yield (srv, processes)
29. HOW IS A BACKGROUND PROCESS A RESOURCE? Watch this space: yt.kubukoz.com -> "Background processing in functional Scala" playlist
30. DEPENDENCY GRAPH AS A RESOURCE
31. ALGEBRAS / CAPABILITY TRAITS
32. ALGEBRAS / CAPABILITY TRAITS Tagless Final style
33. ALGEBRAS / CAPABILITY TRAITS Tagless Final style Interfaces parameterised by an effect
34. ALGEBRAS / CAPABILITY TRAITS Tagless Final style Interfaces parameterised by an effect Capability traits - lawless type classes
35. RULES OF THUMB
36. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async
37. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async Implicit or explicit?
38. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly
39. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly
40. RULES OF THUMB Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async Implicit or explicit? - 1 instance per type: implicit definition, pass implicitly - has possible test instance: explicit definition, pass implicitly - multiple instances in app: all explicit
41. IMAGE PROCESSING APP CASE STUDY
42. IMAGE PROCESSING APP CASE STUDY Project Goals
43. IMAGE PROCESSING APP CASE STUDY Project Goals Search images from a datasource by the text on them (OCR)
44. IMAGE PROCESSING APP CASE STUDY Project Goals Search images from a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time
45. IMAGE PROCESSING APP CASE STUDY Project Goals Search images from a datasource by the text on them (OCR) Live OCR is too slow, so we'll index ahead of time github.com/kubukoz/dropbox-demo
46. DATA FLOW (LINEAR)
47. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0
48. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0 Or... just sensible architecture.
49. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic
50. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0 Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API
51. TANGENT: HEXAGONAL ARCHITECTURE Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0 Jakub Nabrdalik - Hexagonal Architecture in practice https://www.youtube.com/watch?v=sOaS83Ir8Ck Or... just sensible architecture. Keep vendor/implementation-specific details hidden and away from core logic Only talk to these via adapters with a simple API
52. DEPENDENCY GRAPH
53. PROJECT STRUCTURE shared - contains common vocabulary Used by adapters and core logic imagesource, ocr, indexer - modules root - contains core logic + http module Standard sbt pattern for "main" sources
54. OCR MODULE
55. OCR MODULE ProcessRunner - capability trait for running system processes
56. OCR MODULE ProcessRunner - capability trait for running system processes Tesseract - runs a Tesseract process
57. OCR MODULE ProcessRunner - capability trait for running system processes Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages)
58. OCR MODULE ProcessRunner - capability trait for running system processes Tesseract - runs a Tesseract process OCR - wraps Tesseract and specifies config options (languages) TestOCRInstances - contains test fakes for OCR for usage in tests of higher-level components (processes)
59. PROCESS RUNNER package com.kubukoz.process trait ProcessRunner[F[_]] { def run(program: List[String]): Resource[F, ProcessRunner.Running[F]] } object ProcessRunner { def apply[F[_]](implicit F: ProcessRunner[F]): ProcessRunner[F] = F implicit def instance[F[_]: Async]: ProcessRunner[F] = !!... }
60. TESSERACT package com.kubukoz.ocr.tesseract private[ocr] trait Tesseract[F[_]] { def decode(input: fs2.Stream[F, Byte], languages: List[String]): F[String] } object Tesseract { def apply[F[_]](implicit F: Tesseract[F]): Tesseract[F] = F def instance[F[_]: ProcessRunner: Logger: Concurrent](implicit SC: fs2.Compiler[F, F]): Tesseract[F] = !!... }
61. OCR
62. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR
63. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!...
64. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) }
65. package com.kubukoz.ocr trait OCR[F[_]] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] } object OCR { def apply[F[_]](implicit F: OCR[F]): OCR[F] = F } OCR final case class Config(languages: List[String]) def config[F[_]]: ConfigValue[F, Config] = !!... private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] { def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] = Tesseract[F].decode(file, config.languages).map(DecodedText(_)) } def module[F[_]: Concurrent: ProcessRunner: Logger](config: Config): OCR[F] = { implicit val tesseract = Tesseract.instance[F] OCR.tesseractInstance[F](config) }
66. package com.kubukoz.ocr object TestOCRInstances { !// decodeText("hello".getBytes) !== "hello" def simple[F[_]: Functor](implicit SC: fs2.Compiler[F, F]): OCR[F] = _.through(fs2.text.utf8Decode[F]).compile.string.map(DecodedText(_)) } TEST INSTANCE
67. WIRING IT ALL UP
68. WIRING IT ALL UP
69. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) }
70. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config)
71. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] }
72. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]
73. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue)
74. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]]
75. object Application { final case class Config( indexer: Indexer.Config, imageSource: ImageSource.Config, processQueue: ProcessQueue.Config, ocr: OCR.Config, http: HttpServer.Config, ) } def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = ( Indexer.config[F], ImageSource.config[F], ProcessQueue.config[F], OCR.config[F], HttpServer.config[F], ).parMapN(Config) def run[F[_]: Async: Logger](config: Config): Resource[F, Server] = for { implicit0(client: Client[F]) !<- HttpClient.instance[F] } implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer) implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]] processQueue !<- ProcessQueue.instance(config.processQueue) implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]] implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]] implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]] server !<- HttpServer.instance[F](config.http) yield server
76. A COUPLE GUIDELINES TESTING
77. A COUPLE GUIDELINES TESTING Test the contract, not the implementation
78. A COUPLE GUIDELINES TESTING Test the contract, not the implementation Prefer fakes over mocks/stubs
79. A COUPLE GUIDELINES TESTING Test the contract, not the implementation Prefer fakes over mocks/stubs Test your fakes with the same suite as the real things
80. TESTING
81. TESTING index.schedule(Path("/hello"))
82. TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*>
83. TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*> !*> indexer.search("hello").compile.toList
84. TESTING index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*> !*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) }
85. TESTING Await blogpost for more ;) index.schedule(Path("/hello")) val file = fakeFile("hello world", "/hello/world") imageSource.uploadFile(file.fileData) !*> !*> indexer.search("hello").compile.toList { }.map { results !=> expect(results !== List(file.fileDocument)) }
86. SUMMARY
87. TIPS
88. TIPS Use Resource + IO for stateful dependencies
89. TIPS Use Resource + IO for stateful dependencies Define clear responsibilities for modules
90. TIPS Use Resource + IO for stateful dependencies Define clear responsibilities for modules Design for replacement
91. TIPS Use Resource + IO for stateful dependencies Define clear responsibilities for modules Design for replacement Look for abstractions
92. TIPS Use Resource + IO for stateful dependencies Define clear responsibilities for modules Design for replacement Look for abstractions Prototype early
93. TIPS Use Resource + IO for stateful dependencies Define clear responsibilities for modules Design for replacement Look for abstractions Prototype early Draw some diagrams, see if you have too many arrows ;)
94. LEARN MORE Check out the sources: github.com/kubukoz/dropbox-demo Read "Practical FP in Scala" by Gabriel Volpe github.com/scala-steward-org/scala-steward github.com/branchtalk-io/backend github.com/kubukoz/spotify-next github.com/pitgull/pitgull leanpub.com/pfp-scala
95. THANK YOU 📰 blog.kubukoz.com 🐦 @kubukoz Slides: speakerdeck.com/kubukoz Code: git.io/JONEj Find me on YouTube! (yt.kubukoz.com)