Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Connecting the dots - building and structuring a functional application in Scala

Connecting the dots - building and structuring a functional application in Scala

Functional programming relies on building programs from orthogonal, composable blocks. That's likely one of the reasons why full-blown application frameworks haven't gained much traction in the functional ecosystem.

However, we still need to structure our code and wire up our applications in a way that lets us keep them modular, testable and simply pleasant to work with - in this talk, we will learn how to do just that!

Using an application that integrates with several third-party services to process data in a streaming fashion, and expose its results to downstream clients, we will walk through the architecture design and testing setup for a functional app on the Typelevel stack.

Jakub Kozłowski

May 05, 2021
Tweet

More Decks by Jakub Kozłowski

Other Decks in Programming

Transcript

  1. CONNECTING THE DOTS
    BUILDING AND STRUCTURING A FUNCTIONAL APPLICATION IN SCALA
    JAKUB KOZŁOWSKI, DISNEY STREAMING
    YOW! LAMBDA JAM 2021
    Photo by Kumiko SHIMIZU on Unsplash

    View full-size slide

  2. PROBLEM STATEMENT

    View full-size slide

  3. PROBLEM STATEMENT
    We want to build an application

    View full-size slide

  4. PROBLEM STATEMENT
    We want to build an application
    There are some sources of data (databases, APIs, event streams)

    View full-size slide

  5. PROBLEM STATEMENT
    We want to build an application
    There are some sources of data (databases, APIs, event streams)
    We need to serve HTTP traffic

    View full-size slide

  6. PROBLEM STATEMENT
    We want to build an application
    There are some sources of data (databases, APIs, event streams)
    We need to serve HTTP traffic
    Some things need to run in the background additionally

    View full-size slide

  7. PROBLEM STATEMENT
    We want to build an application
    There are some sources of data (databases, APIs, event streams)
    We need to serve HTTP traffic
    Some things need to run in the background additionally
    We want to do it with FP

    View full-size slide

  8. DEPENDENCY GRAPH
    TYPICAL APPLICATION

    View full-size slide

  9. DEPENDENCY GRAPH
    def database: Database
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Server
    def backgroundProcesses(logic: BusinessLogic): Processes

    View full-size slide

  10. DEPENDENCY GRAPH
    def database: Database
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Server
    def backgroundProcesses(logic: BusinessLogic): Processes
    def build: (Server, Processes) = {
    val logic = businessLogic(database)
    (server(logic), backgroundProcesses(logic))
    }

    View full-size slide

  11. DEPENDENCY GRAPH
    This could be us, but the real world exists...
    def database: Database
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Server
    def backgroundProcesses(logic: BusinessLogic): Processes
    def build: (Server, Processes) = {
    val logic = businessLogic(database)
    (server(logic), backgroundProcesses(logic))
    }

    View full-size slide

  12. RESOURCES.
    But we can just kill our application when it quits, right?

    View full-size slide

  13. CONNECTION POOLS?

    View full-size slide

  14. CONNECTION POOLS?
    But we can use try-finally, right?

    View full-size slide

  15. def getConnection(db: Database): Connection
    def returnConnection(conn: Connection): Unit
    def doWork(conn: Connection): Result
    TRY-FINALLY

    View full-size slide

  16. def getConnection(db: Database): Connection
    def returnConnection(conn: Connection): Unit
    def doWork(conn: Connection): Result
    TRY-FINALLY
    val result = {
    val c = getConnection(db)
    try doWork(c)
    finally returnConnection(c)
    }

    View full-size slide

  17. def getConnection(db: Database): IO[Connection]
    def returnConnection(conn: Connection): IO[Unit]
    def doWork(conn: Connection): IO[Result]
    TRY-FINALLY IN FP
    val result: IO[Result] =
    getConnection(db).bracket { c !=>
    doWork(c)
    }(returnConnection)

    View full-size slide

  18. MORE RESOURCES?
    Can't keep nesting bracket forever

    View full-size slide

  19. MORE RESOURCES?
    Can't keep nesting bracket forever
    ABSTRACTION?
    How to hide details?

    View full-size slide

  20. RESOURCE DATA STRUCTURE

    View full-size slide

  21. RUNNING A RESOURCE?

    View full-size slide

  22. RUNNING A RESOURCE?

    View full-size slide

  23. COMPOSITION?

    View full-size slide

  24. WHY AM I TALKING ABOUT THIS?
    def database: Database
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Server
    def backgroundProcesses(logic: BusinessLogic): Processes

    View full-size slide

  25. WHY AM I TALKING ABOUT THIS?
    def database: Resource[Database]
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Resource[Server]
    def backgroundProcesses(logic: BusinessLogic): Resource[Processes]

    View full-size slide

  26. WHY AM I TALKING ABOUT THIS?
    def database: Resource[Database]
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Resource[Server]
    def backgroundProcesses(logic: BusinessLogic): Resource[Processes]
    def build: Resource[(Server, Processes)] = database.flatMap { db !=>
    val logic = businessLogic(db)
    server(logic).flatMap { srv !=>
    backgroundProcesses(logic).map(p !=> (srv, p))
    }
    }

    View full-size slide

  27. WHY AM I TALKING ABOUT THIS?
    def database: Resource[Database]
    def businessLogic(db: Database): BusinessLogic
    def server(logic: BusinessLogic): Resource[Server]
    def backgroundProcesses(logic: BusinessLogic): Resource[Processes]
    def build: Resource[(Server, Processes)] =
    for {
    db !<- database
    logic = businessLogic(db)
    srv !<- server(logic)
    processes !<- backgroundProcesses(logic)
    } yield (srv, processes)

    View full-size slide

  28. HOW IS A BACKGROUND PROCESS A RESOURCE?
    Watch this space: yt.kubukoz.com -> "Background processing in functional Scala" playlist

    View full-size slide

  29. DEPENDENCY GRAPH AS A RESOURCE

    View full-size slide

  30. ALGEBRAS / CAPABILITY TRAITS

    View full-size slide

  31. ALGEBRAS / CAPABILITY TRAITS
    Tagless Final style

    View full-size slide

  32. ALGEBRAS / CAPABILITY TRAITS
    Tagless Final style
    Interfaces parameterised by an effect

    View full-size slide

  33. ALGEBRAS / CAPABILITY TRAITS
    Tagless Final style
    Interfaces parameterised by an effect
    Capability traits - lawless type
    classes

    View full-size slide

  34. RULES OF THUMB

    View full-size slide

  35. RULES OF THUMB
    Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async

    View full-size slide

  36. RULES OF THUMB
    Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async
    Implicit or explicit?

    View full-size slide

  37. RULES OF THUMB
    Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async
    Implicit or explicit?
    - 1 instance per type: implicit definition, pass implicitly

    View full-size slide

  38. RULES OF THUMB
    Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async
    Implicit or explicit?
    - 1 instance per type: implicit definition, pass implicitly
    - has possible test instance: explicit definition, pass implicitly

    View full-size slide

  39. RULES OF THUMB
    Prefer capability traits (Files[F], Network[F], Console[F]) over Sync/Async
    Implicit or explicit?
    - 1 instance per type: implicit definition, pass implicitly
    - has possible test instance: explicit definition, pass implicitly
    - multiple instances in app: all explicit

    View full-size slide

  40. IMAGE PROCESSING APP
    CASE STUDY

    View full-size slide

  41. IMAGE PROCESSING APP
    CASE STUDY
    Project Goals

    View full-size slide

  42. IMAGE PROCESSING APP
    CASE STUDY
    Project Goals
    Search images from a datasource by the text on them (OCR)

    View full-size slide

  43. IMAGE PROCESSING APP
    CASE STUDY
    Project Goals
    Search images from a datasource by the text on them (OCR)
    Live OCR is too slow, so we'll index ahead of time

    View full-size slide

  44. IMAGE PROCESSING APP
    CASE STUDY
    Project Goals
    Search images from a datasource by the text on them (OCR)
    Live OCR is too slow, so we'll index ahead of time
    github.com/kubukoz/dropbox-demo

    View full-size slide

  45. DATA FLOW
    (LINEAR)

    View full-size slide

  46. TANGENT: HEXAGONAL
    ARCHITECTURE
    Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0

    View full-size slide

  47. TANGENT: HEXAGONAL
    ARCHITECTURE
    Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0
    Or... just sensible architecture.

    View full-size slide

  48. TANGENT: HEXAGONAL
    ARCHITECTURE
    Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0
    Or... just sensible architecture.
    Keep vendor/implementation-specific details hidden
    and away from core logic

    View full-size slide

  49. TANGENT: HEXAGONAL
    ARCHITECTURE
    Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0
    Or... just sensible architecture.
    Keep vendor/implementation-specific details hidden
    and away from core logic
    Only talk to these via adapters with a simple API

    View full-size slide

  50. TANGENT: HEXAGONAL
    ARCHITECTURE
    Hexagonal Architecture by Cth027, licensed under CC BY-SA 4.0
    Jakub Nabrdalik - Hexagonal Architecture in practice
    https://www.youtube.com/watch?v=sOaS83Ir8Ck
    Or... just sensible architecture.
    Keep vendor/implementation-specific details hidden
    and away from core logic
    Only talk to these via adapters with a simple API

    View full-size slide

  51. DEPENDENCY
    GRAPH

    View full-size slide

  52. PROJECT STRUCTURE
    shared - contains common vocabulary
    Used by adapters and core logic
    imagesource, ocr, indexer - modules
    root - contains core logic + http module
    Standard sbt pattern for "main" sources

    View full-size slide

  53. OCR MODULE
    ProcessRunner - capability trait for running system processes

    View full-size slide

  54. OCR MODULE
    ProcessRunner - capability trait for running system processes
    Tesseract - runs a Tesseract process

    View full-size slide

  55. OCR MODULE
    ProcessRunner - capability trait for running system processes
    Tesseract - runs a Tesseract process
    OCR - wraps Tesseract and specifies config options (languages)

    View full-size slide

  56. OCR MODULE
    ProcessRunner - capability trait for running system processes
    Tesseract - runs a Tesseract process
    OCR - wraps Tesseract and specifies config options (languages)
    TestOCRInstances - contains test fakes for OCR for usage in tests
    of higher-level components (processes)

    View full-size slide

  57. PROCESS RUNNER
    package com.kubukoz.process
    trait ProcessRunner[F[_]] {
    def run(program: List[String]): Resource[F, ProcessRunner.Running[F]]
    }
    object ProcessRunner {
    def apply[F[_]](implicit F: ProcessRunner[F]): ProcessRunner[F] = F
    implicit def instance[F[_]: Async]: ProcessRunner[F] = !!...
    }

    View full-size slide

  58. TESSERACT
    package com.kubukoz.ocr.tesseract
    private[ocr] trait Tesseract[F[_]] {
    def decode(input: fs2.Stream[F, Byte], languages: List[String]): F[String]
    }
    object Tesseract {
    def apply[F[_]](implicit F: Tesseract[F]): Tesseract[F] = F
    def instance[F[_]: ProcessRunner: Logger: Concurrent](implicit SC: fs2.Compiler[F, F]): Tesseract[F] = !!...
    }

    View full-size slide

  59. package com.kubukoz.ocr
    trait OCR[F[_]] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
    }
    object OCR {
    def apply[F[_]](implicit F: OCR[F]): OCR[F] = F
    }
    OCR

    View full-size slide

  60. package com.kubukoz.ocr
    trait OCR[F[_]] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
    }
    object OCR {
    def apply[F[_]](implicit F: OCR[F]): OCR[F] = F
    }
    OCR
    final case class Config(languages: List[String])
    def config[F[_]]: ConfigValue[F, Config] = !!...

    View full-size slide

  61. package com.kubukoz.ocr
    trait OCR[F[_]] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
    }
    object OCR {
    def apply[F[_]](implicit F: OCR[F]): OCR[F] = F
    }
    OCR
    final case class Config(languages: List[String])
    def config[F[_]]: ConfigValue[F, Config] = !!...
    private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] =
    Tesseract[F].decode(file, config.languages).map(DecodedText(_))
    }

    View full-size slide

  62. package com.kubukoz.ocr
    trait OCR[F[_]] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText]
    }
    object OCR {
    def apply[F[_]](implicit F: OCR[F]): OCR[F] = F
    }
    OCR
    final case class Config(languages: List[String])
    def config[F[_]]: ConfigValue[F, Config] = !!...
    private[ocr] def tesseractInstance[F[_]: Tesseract: Functor](config: Config): OCR[F] = new OCR[F] {
    def decodeText(file: fs2.Stream[F, Byte]): F[DecodedText] =
    Tesseract[F].decode(file, config.languages).map(DecodedText(_))
    }
    def module[F[_]: Concurrent: ProcessRunner: Logger](config: Config): OCR[F] = {
    implicit val tesseract = Tesseract.instance[F]
    OCR.tesseractInstance[F](config)
    }

    View full-size slide

  63. package com.kubukoz.ocr
    object TestOCRInstances {
    !// decodeText("hello".getBytes) !== "hello"
    def simple[F[_]: Functor](implicit SC: fs2.Compiler[F, F]): OCR[F] =
    _.through(fs2.text.utf8Decode[F]).compile.string.map(DecodedText(_))
    }
    TEST INSTANCE

    View full-size slide

  64. WIRING IT ALL UP

    View full-size slide

  65. WIRING IT ALL UP

    View full-size slide

  66. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }

    View full-size slide

  67. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)

    View full-size slide

  68. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)
    def run[F[_]: Async: Logger](config: Config): Resource[F, Server] =
    for {
    implicit0(client: Client[F]) !<- HttpClient.instance[F]
    }

    View full-size slide

  69. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)
    def run[F[_]: Async: Logger](config: Config): Resource[F, Server] =
    for {
    implicit0(client: Client[F]) !<- HttpClient.instance[F]
    }
    implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource
    implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer)
    implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]

    View full-size slide

  70. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)
    def run[F[_]: Async: Logger](config: Config): Resource[F, Server] =
    for {
    implicit0(client: Client[F]) !<- HttpClient.instance[F]
    }
    implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource
    implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer)
    implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]
    processQueue !<- ProcessQueue.instance(config.processQueue)

    View full-size slide

  71. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)
    def run[F[_]: Async: Logger](config: Config): Resource[F, Server] =
    for {
    implicit0(client: Client[F]) !<- HttpClient.instance[F]
    }
    implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource
    implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer)
    implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]
    processQueue !<- ProcessQueue.instance(config.processQueue)
    implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]]
    implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]]
    implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]]

    View full-size slide

  72. object Application {
    final case class Config(
    indexer: Indexer.Config,
    imageSource: ImageSource.Config,
    processQueue: ProcessQueue.Config,
    ocr: OCR.Config,
    http: HttpServer.Config,
    )
    }
    def config[F[_]: ApplicativeThrow]: ConfigValue[F, Config] = (
    Indexer.config[F],
    ImageSource.config[F],
    ProcessQueue.config[F],
    OCR.config[F],
    HttpServer.config[F],
    ).parMapN(Config)
    def run[F[_]: Async: Logger](config: Config): Resource[F, Server] =
    for {
    implicit0(client: Client[F]) !<- HttpClient.instance[F]
    }
    implicit0(imageSource: ImageSource[F]) !<- ImageSource.module[F](config.imageSource).toResource
    implicit0(indexer: Indexer[F]) !<- Indexer.module[F](config.indexer)
    implicit0(ocr: OCR[F]) !<- OCR.module[F](config.ocr).pure[Resource[F, *]]
    processQueue !<- ProcessQueue.instance(config.processQueue)
    implicit0(index: Index[F]) !<- Index.instance[F](processQueue).pure[Resource[F, *]]
    implicit0(download: Download[F]) !<- Download.instance[F].pure[Resource[F, *]]
    implicit0(search: Search[F]) !<- Search.instance[F](serverInfo.get).pure[Resource[F, *]]
    server !<- HttpServer.instance[F](config.http)
    yield server

    View full-size slide

  73. A COUPLE GUIDELINES
    TESTING

    View full-size slide

  74. A COUPLE GUIDELINES
    TESTING
    Test the contract, not the implementation

    View full-size slide

  75. A COUPLE GUIDELINES
    TESTING
    Test the contract, not the implementation
    Prefer fakes over mocks/stubs

    View full-size slide

  76. A COUPLE GUIDELINES
    TESTING
    Test the contract, not the implementation
    Prefer fakes over mocks/stubs
    Test your fakes with the same suite as the real things

    View full-size slide

  77. TESTING
    index.schedule(Path("/hello"))

    View full-size slide

  78. TESTING
    index.schedule(Path("/hello"))
    val file = fakeFile("hello world", "/hello/world")
    imageSource.uploadFile(file.fileData) !*>

    View full-size slide

  79. TESTING
    index.schedule(Path("/hello"))
    val file = fakeFile("hello world", "/hello/world")
    imageSource.uploadFile(file.fileData) !*>
    !*>
    indexer.search("hello").compile.toList

    View full-size slide

  80. TESTING
    index.schedule(Path("/hello"))
    val file = fakeFile("hello world", "/hello/world")
    imageSource.uploadFile(file.fileData) !*>
    !*>
    indexer.search("hello").compile.toList
    {
    }.map { results !=>
    expect(results !== List(file.fileDocument))
    }

    View full-size slide

  81. TESTING
    Await blogpost for more ;)
    index.schedule(Path("/hello"))
    val file = fakeFile("hello world", "/hello/world")
    imageSource.uploadFile(file.fileData) !*>
    !*>
    indexer.search("hello").compile.toList
    {
    }.map { results !=>
    expect(results !== List(file.fileDocument))
    }

    View full-size slide

  82. TIPS
    Use Resource + IO for stateful dependencies

    View full-size slide

  83. TIPS
    Use Resource + IO for stateful dependencies
    Define clear responsibilities for modules

    View full-size slide

  84. TIPS
    Use Resource + IO for stateful dependencies
    Define clear responsibilities for modules
    Design for replacement

    View full-size slide

  85. TIPS
    Use Resource + IO for stateful dependencies
    Define clear responsibilities for modules
    Design for replacement
    Look for abstractions

    View full-size slide

  86. TIPS
    Use Resource + IO for stateful dependencies
    Define clear responsibilities for modules
    Design for replacement
    Look for abstractions
    Prototype early

    View full-size slide

  87. TIPS
    Use Resource + IO for stateful dependencies
    Define clear responsibilities for modules
    Design for replacement
    Look for abstractions
    Prototype early
    Draw some diagrams, see if you have too many arrows ;)

    View full-size slide

  88. LEARN MORE
    Check out the sources: github.com/kubukoz/dropbox-demo
    Read "Practical FP in Scala" by Gabriel Volpe
    github.com/scala-steward-org/scala-steward
    github.com/branchtalk-io/backend
    github.com/kubukoz/spotify-next
    github.com/pitgull/pitgull leanpub.com/pfp-scala

    View full-size slide

  89. THANK YOU
    📰 blog.kubukoz.com
    🐦 @kubukoz
    Slides: speakerdeck.com/kubukoz
    Code: git.io/JONEj
    Find me on YouTube! (yt.kubukoz.com)

    View full-size slide