Tweet
Tweet

Slide 1

Slide 1 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Taro L. Saito, Ph.D. GitHub: @xerial Arm Treasure Data Airframe: Lightweight Building Blocks for Scala November 16th, 2018 Scale By The Bay 2018 - Unconference 1

Slide 2

Slide 2 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) ● An Engineer with Research Background ● Ph.D., University of Tokyo ● DBMS & Genome Science ● Leading Query Engine Team ● Active OSS Developer ● airframe ● sqlite-jdbc ■ More than 1000 GitHub stars ● snappy-java ■ Compression library used in Spark, Parquet ● sbt-sonatype ■ Used in 2000+ Scala projects ● ... 2

Slide 3

Slide 3 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe ● Lightweight Building Blocks for Scala ● Essential libraries for building any applications ● Used in production for 2+ years ● Based on my code collection since 2009 ● Initially written in Java ● Gradually migrated to Scala ● Repackaged into wvlet.airframe in 2016 ● For maintainability ● 18 Modules ● Simplifying your daily programming in Scala 3

Slide 4

Slide 4 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 18 Airframe Modules ● Bootstrap ● airframe-config Configuration loader ● airframe-launcher Command-line program launcher ● Object Serialization ● airframe-codec encoder/decoder SPI + standard codecs ● airframe-msgpack pure-Scala MessagePack implementation ● airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object ● Monitoring & Debugging ● airframe-log Logging ● airframe-metrics Human-readable metrics for time, date, data size, etc ● airframe-jmx Object metrics provider through JMX ● Building Service Objects ● airframe Dependency injection ● airframe-surface Object type inspector ● Misc: ● airframe-control, airframe-jdbc, airframe-json, airframe-http, etc. 4

Slide 5

Slide 5 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Utilities for Debugging Applications 5

Slide 6

Slide 6 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Debugging Applications: Airframe Log ● Airframe Log: A Modern Logging Library for Scala (Medium Blog) ● ANSI color, source code location support 6

Slide 7

Slide 7 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Metrics ● Human Readable Data Format (Duration, DataSize, etc.) ● Handy Time Window String Support 7 “-1d” “-1w” “-7d”

Slide 8

Slide 8 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe JMX ● Checking the internal states of remote JVM processes ● JMX clients ● jconsole has JMX metric monitor ● Airframe JMX -> DataDog -> Dashboard 8

Slide 9

Slide 9 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Launcher ● Using Class & Function Command Line Programs 9

Slide 10

Slide 10 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Dependency Injection with Airframe 10

Slide 11

Slide 11 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Today’s Goals ● Lean How to Use Airframe DI (Dependency Injection) ● Understand what can be simplified with DI ● Learn 5 Airframe DI Design Patterns ● That improve the thought processes in programming 11

Slide 12

Slide 12 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. What is Dependency Injection (DI)? ● Many Articles… ● Inversion of Control Containers and the Dependency Injection pattern. Martin Fowler (2004) ● StackOverflow, Wikipedia, … ● Many Frameworks... ● Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc. ● No framework approaches also exist (Pure-Scala DI) ● Recent Definition: ● Dependency Injection is the process of creating the static, stateless graph of service objects, where each service is parameterised by its dependencies. ■ What is Dependency Injection? by Adam Warski (2018) ● However, it’s still difficult to understand what is DI 12

Slide 13

Slide 13 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Simplifying DI with Airframe ● Airframe Usage ● import wvlet.airframe._ ● Simple 3 Step DI ● bind ● design ● build 13

Slide 14

Slide 14 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Design Patterns In Airframe ● Pattern 0 (basic): Building Service Objects with Auto-Wiring ● Pattern 1: Configuring Modules ● Pattern 2: Switching Bindings for Tests ● Pattern 3: Managing Lifecycle in FILO order ● Pattern 4: Bundling Service Traits ● Flower-bundle pattern ● Pattern 5: Binding Factory 14

Slide 15

Slide 15 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Constructor vs In-Trait Injections ● Constructor Injection ● In-Trait Injection 15

Slide 16

Slide 16 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Building Service Objects ● When coding A and B ● You can focus on only direct dependencies ● You can forget about indirect dependencies ● Airframe DI builds A, B, and direct/indirect dependencies on your behalf. A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 16 You can forget this part

Slide 17

Slide 17 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Code Example ● Focus on Logic 17 A DB Client Fluentd Logger B

Slide 18

Slide 18 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Hand Wiring vs Auto Wiring 18

Slide 19

Slide 19 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Configuring Modules ● How to pass configuration objects to corresponding modules? ● new A(new DBClient(new ConnectionPool(new DB(new DBConfig(...)), new ConnectionPoolConfig(...), new DBClientConfig(...))) ● Things You Need to Remember ● Argument orders (or argument names in Scala) of individual modules ● How to instantiate modules A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 19 DB Config Connection Pool Config HttpClient Config

Slide 20

Slide 20 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 1: Adding Config ● Put config to the closest place in the code 20

Slide 21

Slide 21 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing ● In Airframe Design ● You can replace DB and FluentdLogger to In-Memory Impl ● How to build A and B differs, but the same code can be used A Memory DB Connection Pool DB Client DB Monitor Fluentd Logger In-memory Logger B 21 Overriding Design for Testing

Slide 22

Slide 22 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing 22

Slide 23

Slide 23 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 3: Managing Lifecycle ● onStart, onShutdown hooks ● JSR-250 annotations 23

Slide 24

Slide 24 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Complex FILO Order Resource Management ● FILO := First-In Last-Out ● Airframe registers onStart and onShutdown lifecycle hooks when creating instances ● When closing sessions, onShutdown will be called in the reverse order ● Dependencies forms DAG ● Dependencies will be generated when creating new service objects A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 1 3 4 5 6 7 2 8 Shared Resource 24

Slide 25

Slide 25 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 4: Bundling Service Traits ● Flower-Bundle Pattern ● Create composable services 25

Slide 26

Slide 26 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 5: Bind Factory ● Create a factory that change partial dependencies ● e.g., creating differently configured instances of Databases 26

Slide 27

Slide 27 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 3 Things You Can Forget With Airframe DI ● 1. How to Build Service Objects ● config, auto-wiring, flower bundle pattern ● 2. How to Manage Resource Lifecycle ● FILO order ● 3. How to Use DI Itself (!!) ● Only need to understand bind, design, and build. 27

Slide 28

Slide 28 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary: Reducing Code Complexity with Airframe DI ● You can effectively forget about: ● How to build service objects ● How to manage resources in FILO order ● How to use DI itself A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 1 3 4 5 6 7 2 8 28 Implementation Details

Slide 29

Slide 29 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Internals 29

Slide 30

Slide 30 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Details ● bind[X] ● X becomes singleton ● Usually there is no need to declare bind[X].toSingleton ■ Unless you want to initialize X early with production mode ● bindInstance[X] ● When you need to call new X(d1, d2, …) every time ● bindFactory[A1 => X] ● For customizing A1 in X ● Design ● Immutable & Serializable ● design1 + design2 + …. ■ Overriding the previous design (adding order is important) ● Session ● withLazyMode/withProductionMode ● noLifeCycleLogging 30

Slide 31

Slide 31 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Injecting Airframe Session with Scala Macros 31

Slide 32

Slide 32 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Surface: Object Shape Inspector ● Reading Type Signatures From ScalaSig ● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files ● Surface supports ● type alias, tagged type ● higher-kinded types class A (data:List[B]) class A data: List[java.lang.Object] class A data: List[java.lang.Object] ScalaSig: data:List[B] javac scalac Surface.of[A] data: List[B] scala.reflect.runtime. universe.TypeTag Type Erasure 32 ???

Slide 33

Slide 33 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Future Work & Summary 33

Slide 34

Slide 34 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Current State of Airframe ● Version 0.73 (As of November 2018) ● We already had 40+ releases in 2018 ● Automated Release ● Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js ● ‘sbt release’ command took 3 hours ■ Sequential steps: ○ compile -> test -> package -> upload x 18 modules x 4 Scala versions ● Now a new version can be released in 10 minutes on Travis CI ● Blog ● 3 Tips for Maintaining Scala Projects ● Future Work ● Adding child session support ● Support multiple constructor argument blocks 34

Slide 35

Slide 35 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Philosophy: Simplicity By Design ● “Simplicity” by Philippe Dufour ● A clock made by a legendary watchmaker in Switzerland ● Every part of the clock is built by himself ● Airframe ● Provides simplicity for application developers 35

Slide 36

Slide 36 text

Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary ● To understand DI, think about what you can simplify ● How to build objects ● How to manage resources (FILO) ● Learning DI framework itself ● 5 Airframe Design Patterns ● Pattern 1: Configuring Modules ● Pattern 2: Overriding Bindings for Tests ● Pattern 3: Managing Lifecycle in FILO order ● Pattern 4: Bundling Service Traits ■ Flower-bundle pattern ● Pattern 5: Binding Factory Don’t Forget Adding GitHub Star! wvlet/airframe 36

Slide 37

Slide 37 text

Confidential © Arm 2017 Confidential © Arm 2017 Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 37