$30 off During Our Annual Pro Sale. View Details »

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018

Airframe is a collection of lightweight libraries useful for building full-fledged applications in Scala. https://github.com/wvlet/airframe

Taro L. Saito

November 17, 2018
Tweet

More Decks by Taro L. Saito

Other Decks in Technology

Transcript

  1. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Taro L. Saito, Ph.D.
    GitHub: @xerial
    Arm Treasure Data
    Airframe: Lightweight Building
    Blocks for Scala
    November 16th, 2018
    Scale By The Bay 2018 - Unconference
    1

    View Slide

  2. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    About Me: Taro L. Saito (Leo)
    ● An Engineer with Research Background
    ● Ph.D., University of Tokyo
    ● DBMS & Genome Science
    ● Leading Query Engine Team
    ● Active OSS Developer
    ● airframe
    ● sqlite-jdbc
    ■ More than 1000 GitHub stars
    ● snappy-java
    ■ Compression library used in
    Spark, Parquet
    ● sbt-sonatype
    ■ Used in 2000+ Scala projects
    ● ...
    2

    View Slide

  3. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe
    ● Lightweight Building Blocks for Scala
    ● Essential libraries for building any applications
    ● Used in production for 2+ years
    ● Based on my code collection since 2009
    ● Initially written in Java
    ● Gradually migrated to Scala
    ● Repackaged into wvlet.airframe in 2016
    ● For maintainability
    ● 18 Modules
    ● Simplifying your daily programming in Scala
    3

    View Slide

  4. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    18 Airframe Modules
    ● Bootstrap
    ● airframe-config Configuration loader
    ● airframe-launcher Command-line program launcher
    ● Object Serialization
    ● airframe-codec encoder/decoder SPI + standard codecs
    ● airframe-msgpack pure-Scala MessagePack implementation
    ● airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object
    ● Monitoring & Debugging
    ● airframe-log Logging
    ● airframe-metrics Human-readable metrics for time, date, data size, etc
    ● airframe-jmx Object metrics provider through JMX
    ● Building Service Objects
    ● airframe Dependency injection
    ● airframe-surface Object type inspector
    ● Misc:
    ● airframe-control, airframe-jdbc, airframe-json, airframe-http, etc.
    4

    View Slide

  5. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Utilities for Debugging Applications
    5

    View Slide

  6. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Debugging Applications: Airframe Log
    ● Airframe Log: A Modern Logging Library for Scala (Medium Blog)
    ● ANSI color, source code location support
    6

    View Slide

  7. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe Metrics
    ● Human Readable Data Format (Duration, DataSize, etc.)
    ● Handy Time Window String Support
    7
    “-1d”
    “-1w”
    “-7d”

    View Slide

  8. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe JMX
    ● Checking the internal states of remote JVM processes
    ● JMX clients
    ● jconsole has JMX metric monitor
    ● Airframe JMX -> DataDog -> Dashboard
    8

    View Slide

  9. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe Launcher
    ● Using Class & Function Command Line Programs
    9

    View Slide

  10. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Dependency Injection with Airframe
    10

    View Slide

  11. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Today’s Goals
    ● Lean How to Use Airframe DI (Dependency Injection)
    ● Understand what can be simplified with DI
    ● Learn 5 Airframe DI Design Patterns
    ● That improve the thought processes in programming
    11

    View Slide

  12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    What is Dependency Injection (DI)?
    ● Many Articles…
    ● Inversion of Control Containers and the Dependency Injection pattern. Martin
    Fowler (2004)
    ● StackOverflow, Wikipedia, …
    ● Many Frameworks...
    ● Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc.
    ● No framework approaches also exist (Pure-Scala DI)
    ● Recent Definition:
    ● Dependency Injection is the process of creating the static, stateless graph of
    service objects, where each service is parameterised by its dependencies.
    ■ What is Dependency Injection? by Adam Warski (2018)
    ● However, it’s still difficult to understand what is DI
    12

    View Slide

  13. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Simplifying DI with Airframe
    ● Airframe Usage
    ● import wvlet.airframe._
    ● Simple 3 Step DI
    ● bind
    ● design
    ● build
    13

    View Slide

  14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Design Patterns In Airframe
    ● Pattern 0 (basic): Building Service Objects with Auto-Wiring
    ● Pattern 1: Configuring Modules
    ● Pattern 2: Switching Bindings for Tests
    ● Pattern 3: Managing Lifecycle in FILO order
    ● Pattern 4: Bundling Service Traits
    ● Flower-bundle pattern
    ● Pattern 5: Binding Factory
    14

    View Slide

  15. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Constructor vs In-Trait Injections
    ● Constructor Injection
    ● In-Trait Injection
    15

    View Slide

  16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Building Service Objects
    ● When coding A and B
    ● You can focus on only direct dependencies
    ● You can forget about indirect dependencies
    ● Airframe DI builds A, B, and direct/indirect dependencies on your behalf.
    A
    DB
    Connection
    Pool
    DB
    Client
    DB Monitor
    Fluentd
    Logger
    HttpClient
    B
    16
    You can forget this part

    View Slide

  17. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Code Example
    ● Focus on Logic
    17
    A
    DB
    Client
    Fluentd
    Logger
    B

    View Slide

  18. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Hand Wiring vs Auto Wiring
    18

    View Slide

  19. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Configuring Modules
    ● How to pass configuration objects to corresponding modules?
    ● new A(new DBClient(new ConnectionPool(new DB(new DBConfig(...)), new
    ConnectionPoolConfig(...), new DBClientConfig(...)))
    ● Things You Need to Remember
    ● Argument orders (or argument names in Scala) of individual modules
    ● How to instantiate modules
    A
    DB
    Connection
    Pool
    DB
    Client
    DB Monitor
    Fluentd
    Logger
    HttpClient
    B
    19
    DB
    Config
    Connection
    Pool Config
    HttpClient
    Config

    View Slide

  20. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 1: Adding Config
    ● Put config to the closest place in the code
    20

    View Slide

  21. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 2: Switching Bindings For Testing
    ● In Airframe Design
    ● You can replace DB and FluentdLogger to In-Memory Impl
    ● How to build A and B differs, but the same code can be used
    A
    Memory
    DB
    Connection
    Pool
    DB
    Client
    DB Monitor
    Fluentd
    Logger
    In-memory
    Logger
    B
    21
    Overriding Design for Testing

    View Slide

  22. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 2: Switching Bindings For Testing
    22

    View Slide

  23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 3: Managing Lifecycle
    ● onStart, onShutdown hooks
    ● JSR-250 annotations
    23

    View Slide

  24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Complex FILO Order Resource Management
    ● FILO := First-In Last-Out
    ● Airframe registers onStart and onShutdown lifecycle hooks when creating
    instances
    ● When closing sessions, onShutdown will be called in the reverse order
    ● Dependencies forms DAG
    ● Dependencies will be generated when creating new service objects
    A
    DB
    Connection
    Pool
    DB
    Client
    DB Monitor
    Fluentd
    Logger
    HttpClient
    B
    1
    3
    4
    5
    6
    7
    2
    8
    Shared
    Resource
    24

    View Slide

  25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 4: Bundling Service Traits
    ● Flower-Bundle Pattern
    ● Create composable services
    25

    View Slide

  26. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Pattern 5: Bind Factory
    ● Create a factory that change partial dependencies
    ● e.g., creating differently configured instances of Databases
    26

    View Slide

  27. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    3 Things You Can Forget With Airframe DI
    ● 1. How to Build Service Objects
    ● config, auto-wiring, flower bundle pattern
    ● 2. How to Manage Resource Lifecycle
    ● FILO order
    ● 3. How to Use DI Itself (!!)
    ● Only need to understand bind, design, and build.
    27

    View Slide

  28. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Summary: Reducing Code Complexity with Airframe DI
    ● You can effectively forget about:
    ● How to build service objects
    ● How to manage resources in FILO order
    ● How to use DI itself
    A
    DB
    Connection
    Pool
    DB
    Client
    DB Monitor
    Fluentd
    Logger
    HttpClient
    B
    1
    3
    4
    5
    6
    7
    2
    8
    28
    Implementation Details

    View Slide

  29. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe Internals
    29

    View Slide

  30. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Details
    ● bind[X]
    ● X becomes singleton
    ● Usually there is no need to declare bind[X].toSingleton
    ■ Unless you want to initialize X early with production mode
    ● bindInstance[X]
    ● When you need to call new X(d1, d2, …) every time
    ● bindFactory[A1 => X]
    ● For customizing A1 in X
    ● Design
    ● Immutable & Serializable
    ● design1 + design2 + ….
    ■ Overriding the previous design (adding order is important)
    ● Session
    ● withLazyMode/withProductionMode
    ● noLifeCycleLogging
    30

    View Slide

  31. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Injecting Airframe Session with Scala Macros
    31

    View Slide

  32. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Airframe Surface: Object Shape Inspector
    ● Reading Type Signatures From ScalaSig
    ● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files
    ● Surface supports
    ● type alias, tagged type
    ● higher-kinded types
    class A (data:List[B])
    class A
    data: List[java.lang.Object]
    class A
    data: List[java.lang.Object]
    ScalaSig: data:List[B]
    javac
    scalac
    Surface.of[A]
    data: List[B]
    scala.reflect.runtime.
    universe.TypeTag
    Type Erasure
    32
    ???

    View Slide

  33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Future Work & Summary
    33

    View Slide

  34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Current State of Airframe
    ● Version 0.73 (As of November 2018)
    ● We already had 40+ releases in 2018
    ● Automated Release
    ● Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js
    ● ‘sbt release’ command took 3 hours
    ■ Sequential steps:
    ○ compile -> test -> package -> upload x 18 modules x 4 Scala versions
    ● Now a new version can be released in 10 minutes on Travis CI
    ● Blog
    ● 3 Tips for Maintaining Scala Projects
    ● Future Work
    ● Adding child session support
    ● Support multiple constructor argument blocks
    34

    View Slide

  35. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Philosophy: Simplicity By Design
    ● “Simplicity” by Philippe Dufour
    ● A clock made by a legendary
    watchmaker in Switzerland
    ● Every part of the clock is built
    by himself
    ● Airframe
    ● Provides simplicity for
    application developers
    35

    View Slide

  36. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
    Summary
    ● To understand DI, think about what you can
    simplify
    ● How to build objects
    ● How to manage resources (FILO)
    ● Learning DI framework itself
    ● 5 Airframe Design Patterns
    ● Pattern 1: Configuring Modules
    ● Pattern 2: Overriding Bindings for Tests
    ● Pattern 3: Managing Lifecycle in FILO order
    ● Pattern 4: Bundling Service Traits
    ■ Flower-bundle pattern
    ● Pattern 5: Binding Factory
    Don’t Forget Adding GitHub Star!
    wvlet/airframe
    36

    View Slide

  37. Confidential © Arm 2017
    Confidential © Arm 2017
    Confidential © Arm 2017
    Thank You!
    Danke!
    Merci!
    谢谢!
    ありがとう!
    Gracias!
    Kiitos!
    37

    View Slide