Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018

Airframe is a collection of lightweight libraries useful for building full-fledged applications in Scala. https://github.com/wvlet/airframe

Transcript

1. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Taro L. Saito, Ph.D. GitHub: @xerial Arm Treasure Data Airframe: Lightweight Building Blocks for Scala November 16th, 2018 Scale By The Bay 2018 - Unconference 1
2. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) • An Engineer with Research Background • Ph.D., University of Tokyo • DBMS & Genome Science • Leading Query Engine Team • Active OSS Developer • airframe • sqlite-jdbc ▪ More than 1000 GitHub stars • snappy-java ▪ Compression library used in Spark, Parquet • sbt-sonatype ▪ Used in 2000+ Scala projects • ... 2
3. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe • Lightweight Building Blocks for Scala • Essential libraries for building any applications • Used in production for 2+ years • Based on my code collection since 2009 • Initially written in Java • Gradually migrated to Scala • Repackaged into wvlet.airframe in 2016 • For maintainability • 18 Modules • Simplifying your daily programming in Scala 3
4. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 18 Airframe Modules • Bootstrap • airframe-config Configuration loader • airframe-launcher Command-line program launcher • Object Serialization • airframe-codec encoder/decoder SPI + standard codecs • airframe-msgpack pure-Scala MessagePack implementation • airframe-tablet CSV/TSV/JSON/JDBC ResultSet <-> Object • Monitoring & Debugging • airframe-log Logging • airframe-metrics Human-readable metrics for time, date, data size, etc • airframe-jmx Object metrics provider through JMX • Building Service Objects • airframe Dependency injection • airframe-surface Object type inspector • Misc: • airframe-control, airframe-jdbc, airframe-json, airframe-http, etc. 4
5. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Utilities for Debugging Applications 5
6. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Debugging Applications: Airframe Log • Airframe Log: A Modern Logging Library for Scala (Medium Blog) • ANSI color, source code location support 6
7. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Metrics • Human Readable Data Format (Duration, DataSize, etc.) • Handy Time Window String Support 7 “-1d” “-1w” “-7d”
8. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe JMX • Checking the internal states of remote JVM processes • JMX clients • jconsole has JMX metric monitor • Airframe JMX -> DataDog -> Dashboard 8
9. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Launcher • Using Class & Function Command Line Programs 9
10. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Dependency Injection with Airframe 10
11. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Today’s Goals • Lean How to Use Airframe DI (Dependency Injection) • Understand what can be simplified with DI • Learn 5 Airframe DI Design Patterns • That improve the thought processes in programming 11
12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. What is Dependency Injection (DI)? • Many Articles… • Inversion of Control Containers and the Dependency Injection pattern. Martin Fowler (2004) • StackOverflow, Wikipedia, … • Many Frameworks... • Spring, Google Guice, Scaldi, Macwire, Grafter, Weld, etc. • No framework approaches also exist (Pure-Scala DI) • Recent Definition: • Dependency Injection is the process of creating the static, stateless graph of service objects, where each service is parameterised by its dependencies. ▪ What is Dependency Injection? by Adam Warski (2018) • However, it’s still difficult to understand what is DI 12
13. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Simplifying DI with Airframe • Airframe Usage • import wvlet.airframe._ • Simple 3 Step DI • bind • design • build 13
14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Design Patterns In Airframe • Pattern 0 (basic): Building Service Objects with Auto-Wiring • Pattern 1: Configuring Modules • Pattern 2: Switching Bindings for Tests • Pattern 3: Managing Lifecycle in FILO order • Pattern 4: Bundling Service Traits • Flower-bundle pattern • Pattern 5: Binding Factory 14
15. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Constructor vs In-Trait Injections • Constructor Injection • In-Trait Injection 15
16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Building Service Objects • When coding A and B • You can focus on only direct dependencies • You can forget about indirect dependencies • Airframe DI builds A, B, and direct/indirect dependencies on your behalf. A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 16 You can forget this part
17. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Code Example • Focus on Logic 17 A DB Client Fluentd Logger B
18. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Hand Wiring vs Auto Wiring 18
19. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Configuring Modules • How to pass configuration objects to corresponding modules? • new A(new DBClient(new ConnectionPool(new DB(new DBConfig(...)), new ConnectionPoolConfig(...), new DBClientConfig(...))) • Things You Need to Remember • Argument orders (or argument names in Scala) of individual modules • How to instantiate modules A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 19 DB Config Connection Pool Config HttpClient Config
20. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 1: Adding Config • Put config to the closest place in the code 20
21. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing • In Airframe Design • You can replace DB and FluentdLogger to In-Memory Impl • How to build A and B differs, but the same code can be used A Memory DB Connection Pool DB Client DB Monitor Fluentd Logger In-memory Logger B 21 Overriding Design for Testing
22. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 2: Switching Bindings For Testing 22
23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 3: Managing Lifecycle • onStart, onShutdown hooks • JSR-250 annotations 23
24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Complex FILO Order Resource Management • FILO := First-In Last-Out • Airframe registers onStart and onShutdown lifecycle hooks when creating instances • When closing sessions, onShutdown will be called in the reverse order • Dependencies forms DAG • Dependencies will be generated when creating new service objects A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 1 3 4 5 6 7 2 8 Shared Resource 24
25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 4: Bundling Service Traits • Flower-Bundle Pattern • Create composable services 25
26. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Pattern 5: Bind Factory • Create a factory that change partial dependencies • e.g., creating differently configured instances of Databases 26
27. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. 3 Things You Can Forget With Airframe DI • 1. How to Build Service Objects • config, auto-wiring, flower bundle pattern • 2. How to Manage Resource Lifecycle • FILO order • 3. How to Use DI Itself (!!) • Only need to understand bind, design, and build. 27
28. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary: Reducing Code Complexity with Airframe DI • You can effectively forget about: • How to build service objects • How to manage resources in FILO order • How to use DI itself A DB Connection Pool DB Client DB Monitor Fluentd Logger HttpClient B 1 3 4 5 6 7 2 8 28 Implementation Details
29. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Internals 29
30. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Details • bind[X] • X becomes singleton • Usually there is no need to declare bind[X].toSingleton ▪ Unless you want to initialize X early with production mode • bindInstance[X] • When you need to call new X(d1, d2, …) every time • bindFactory[A1 => X] • For customizing A1 in X • Design • Immutable & Serializable • design1 + design2 + …. ▪ Overriding the previous design (adding order is important) • Session • withLazyMode/withProductionMode • noLifeCycleLogging 30
31. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Injecting Airframe Session with Scala Macros 31
32. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Airframe Surface: Object Shape Inspector • Reading Type Signatures From ScalaSig • Scala compiler embeds Scala Type Signatures (ScalaSig) to class files • Surface supports • type alias, tagged type • higher-kinded types class A (data:List[B]) class A data: List[java.lang.Object] class A data: List[java.lang.Object] ScalaSig: data:List[B] javac scalac Surface.of[A] data: List[B] scala.reflect.runtime. universe.TypeTag Type Erasure 32 ???
33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Future Work & Summary 33
34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Current State of Airframe • Version 0.73 (As of November 2018) • We already had 40+ releases in 2018 • Automated Release • Cross building libraries for Scala 2.11, 2.12, 2.13, and Scala.js • ‘sbt release’ command took 3 hours ▪ Sequential steps: ◦ compile -> test -> package -> upload x 18 modules x 4 Scala versions • Now a new version can be released in 10 minutes on Travis CI • Blog • 3 Tips for Maintaining Scala Projects • Future Work • Adding child session support • Support multiple constructor argument blocks 34
35. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Philosophy: Simplicity By Design • “Simplicity” by Philippe Dufour • A clock made by a legendary watchmaker in Switzerland • Every part of the clock is built by himself • Airframe • Provides simplicity for application developers 35
36. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Summary • To understand DI, think about what you can simplify • How to build objects • How to manage resources (FILO) • Learning DI framework itself • 5 Airframe Design Patterns • Pattern 1: Configuring Modules • Pattern 2: Overriding Bindings for Tests • Pattern 3: Managing Lifecycle in FILO order • Pattern 4: Bundling Service Traits ▪ Flower-bundle pattern • Pattern 5: Binding Factory Don’t Forget Adding GitHub Star! wvlet/airframe 36
37. Confidential © Arm 2017 Confidential © Arm 2017 Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 37