Unifying Frontend and Backend Development with Scala - Scala Con 2021

Scala can be used for developing both frontend (Scala.js) and backend (Scala JVM) applications. A missing piece has been bridging these two worlds using Scala. We built Airframe RPC, a framework that uses Scala traits as a unified RPC interface between servers and clients. With Airframe RPC, you can build HTTP/1 (Finagle) and HTTP/2 (gRPC) services just by defining Scala traits and case classes. It simplifies web application design as you only need to care about Scala interfaces without using existing web standards like REST, ProtocolBuffers, OpenAPI, etc. Scala.js support of Airframe also enables building interactive Web applications that can dynamically render DOM elements while talking with Scala-based RPC servers. With Airframe RPC, the value of Scala developers will be much higher both for frontend and backend areas.

Presentation Video at Scala Con 2021: https://www.youtube.com/watch?v=MliLBejnyy8

Transcript

1. © 2020 Treasure Data 1
   Taro L. Saito
   Treasure Data
   November 2nd, 2021
   ScalaCon 2021
   Unifying Frontend and Backend
   Development with Scala
   

   View Slide

2. Copyright 1995-2020 Treasure Data. All rights reserved.
   About Me: Taro L. Saito
   2
   ● Treasure Data
   ○ Principal software engineer
   ○ Building distributed query engine services
   ● OSS activities
   ○ Using Scala for 13+ years
   ■ Since Scala 2.7 (2008)
   ○ Airframe (This talk)
   ■ 20+ Scala utility modules
   ○ sbt-sonatype
   ■ Publishing to Maven central (Sonatype)
   ○ etc.
   Airframe
   

   View Slide

3. Copyright 1995-2020 Treasure Data. All rights reserved.
   Treasure Data: A Ready-to-Use Cloud Data Platform
   3
   Logs
   Device
   Data
   Batch
   Data
   PlazmaDB
   Table Schema
   Data Collection Cloud Storage Distributed Data Processing
   Jobs
   Job Management
   SQL Editor
   Scheduler
   Workflows
   Machine
   Learning
   Treasure Data OSS
   Third Party OSS
   Data
   ● Even if you are not an engineer, you can start collecting and analyzing your data
   

   View Slide

4. Copyright 1995-2020 Treasure Data. All rights reserved.
   Even Backend Engineers Need Frontend Skills
   ● Creating a better visualization of services
   ○ To monitor millions of queries running every day
   ○ Created a UI for service monitoring and cluster management with Scala.js
   4
   

   View Slide

5. Copyright 1995-2020 Treasure Data. All rights reserved.
   Scala.js Application: Notebook Query Editor
   ● Rich SQL editor
   ● Frontend
   ○ Scala.js
   ○ Monaco editor of Visual Studio
   Code
   ● Backend
   ○ Scala on JVM
   ○ RPC server
   ■ SQL runner
   ■ Task manager
   ● Written only in Scala
   5
   

   View Slide

6. Copyright 1995-2020 Treasure Data. All rights reserved.
   Too Many Frontend and Backend Frameworks
   ● How do we choose one of the frameworks to use?
   ● It’s almost impossible to find Scala engineers mastering all of these frontend and backend
   technologies.
   6
   Web Browsers
   Scala.js
   

   View Slide

7. Copyright 1995-2020 Treasure Data. All rights reserved.
   Airframe RPC
   ● Using plain Scala as an RPC interface between frontend and backend applications
   ● Low-learning cost
   ○ Requires only basic Scala knowledge, which can be learned with one Scala book
   ● Our experience so far after introducing Airframe RPC:
   ○ A Java engineer in Treasure Data can start writing a Scala RPC service interacting with a
   Scala.js UI in a week.
   7
   RPC Interface
   Scala.js Client RPC Web Server
   Generates
   Scala.js
   Web Application
   API Documentation
   RPC Call RPC Call
   Implements
   Airframe
   

   View Slide

8. Copyright 1995-2020 Treasure Data. All rights reserved.
   Airframe
   ● GitHub: wvlet/airframe
   ● 20+ useful modules written in pure Scala
   ○ Cross-built for Scala 2.12, 2.13, and Scala.js
   ○ Scala 3 support is in progress
   ■ 85%~ completed
   ■ WIP:
   ● Tasty reader & Scala reflection
   compatibility
   ● Core Modules (*: will be explained in this talk)
   ○ airframe-log*
   ○ airframe-codec*
   ■ Object serializer
   ○ airframe-http*
   ■ HTTP server/client with RPC support
   ○ airframe-rx*
   ■ Reactive DOM rendering
   ○ airframe-json/msgpack
   ■ JSON/MsgPack parser
   ○ airframe-di
   ■ Constructor injection helper
   ○ airframe-surface
   ■ Compile-time or runtime object inspector
   ○ AirSpec: Testing library integrated with Airframe DI
   ○ etc.
   8
   Airframe
   

   View Slide

9. Copyright 1995-2020 Treasure Data. All rights reserved.
   ● Various internal and third-party Scala/Java libraries were used
   ○ e.g., Many web frameworks, utilities, testing frameworks, etc.
   ● The cost of learning individual libraries and frameworks was high
   Before Airframe: 5 Years Ago (2016)
   9
   Knowledge
   Experiences
   Design Decisions
   Products
   24/7 Services
   Business Values
   Programming Various Libraries Outcome
   logger
   launcher
   object mapper
   JDBC reader
   json4s jackson
   ….
   

   View Slide

10. Copyright 1995-2020 Treasure Data. All rights reserved.
    Current (2021): Consolidating the Best Practices into Airframe OSS
    ● Since 2016, we started defining standards in Airframe
    ○ Engineers can focus on developing applications instead of learning frameworks
    10
    Knowledge
    Experiences
    Design Decisions
    Products
    24/7 Services
    Business Values
    Programming OSS Outcome
    Airframe
    

    View Slide

11. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-log: A Modern Logging Library for Scala and Scala.js
    ● There were too many choices just for generating human-readable logs
    ○ log4j, slf4j, scala-logging, java.util.logging, twitter/util, etc.
    ● airframe-log
    ○ Adding colorful logging methods to your class with LogSupport trait
    ○ Showing the source code location
    11
    

    View Slide

12. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-log usage in sbt-sonatype
    ● Shows detailed logs while publishing projects to Maven Central (Sonatype OSS Repository)
    ● News: sbt-sonatype update
    ○ Since 2019, sbt-sonatype introduced a new bundle upload process, which can upload thousands of
    artifact files (jars, source files) to Sonatype within a minute.
    12
    

    View Slide

13. Copyright 1995-2020 Treasure Data. All rights reserved.
    ● Scala.js compiles Scala code into JavaScript, which runs on web browsers
    ● Logging for the developer console of web browsers
    airframe-log: Debugging Scala.js Code
    13
    

    View Slide

14. Copyright 1995-2020 Treasure Data. All rights reserved.
    Frontend Development: Rendering DOM with Scala.js
    ● DOM element:
    ○ ….
    ■ tag, attribute, enclosed elements
    ○ scalajs-dom library has all necessary functions for manipulating DOM nodes
    ● A previous approach for DOM rendering:
    ○ Converting Scala’s XML literal into DOM tree nodes
    ■ OSS: monadic-html, Binding.scala
    ■ Similar to JSX in Facebook React (XML syntax embedded to JavaScript)
    ○ But, XML literal will be deprecated in Scala 3
    14
    

    View Slide

15. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-rx: Functional DOM Rendering
    ● The current best practice is defining
    functions for rendering HTML elements
    ○ table, tr, td, body, div, span, etc.
    ● Then, compose these functions
    ● Style 1: Separating DOM attributes and
    elements into different blocks
    ○ ScalaTags, Slinky
    ○ Some disadvantages
    ■ Too many parentheses!
    ■ Formatting with scalafmt is a bit
    challenging
    ● Style 2: Enclosing both DOM attributes
    and elements inside a single block
    ○ airframe-rx, scalajs-react
    ○ More Scala-friendly syntax
    ○ The begin/end of a tag are clear
    15
    ScalaTags
    airframe-rx
    

    View Slide

16. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-rx: Rendering Complex DOM
    ● RxElement interface:
    ○ def render(): RxElement
    ○ Support nesting with apply(...)
    ■ Embedding Scala functions and collections
    16
    

    View Slide

17. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-rx: Reactive Stream Interface Rx[A]
    ● Rx[A]
    ○ ReactiveX stream implementation
    ● Rewriting DOM interactively upon upstream data changes
    ○ e.g., mouse clicks, getting RPC responses (Future), interval
    timer events, etc.
    ● Reactive Operators
    ○ map, flatMap, filter, join, zip, etc.
    ○ The usage is almost the same with Scala collection APIs
    17
    Airframe
    

    View Slide

18. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-rx: Rx.variable
    ● Rx.variable
    ○ Fire an event if the variable is updated
    ● Interactive DOM Rendering
    ○ Subscribing upstream changes
    ■ Update DOMs for each change
    ○ Cancellation
    ■ Unsubscribe the event processing
    ■ Necessary for cleaning up event
    handlers added to the browser
    ● mouse click handler, timer, etc.
    18
    

    View Slide

19. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-rx: Gallery
    19
    

    View Slide

20. Copyright 1995-2020 Treasure Data. All rights reserved.
    RPC: Bridging Scala and Scala.js Applications
    Program Function
    Call
    Return
    Local Function Call
    (ideal)
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Remote Procedure Call
    (reality)
    Network
    Client
    Call
    Return
    Server
    Call
    Return
    20
    Scala.js
    Scala.js
    

    View Slide

21. Copyright 1995-2020 Treasure Data. All rights reserved.
    Traditional Approaches for Implementing RPC
    ● REST API
    ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
    ○ REST web frameworks for Scala
    ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
    ● gRPC
    ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
    ○ Generate server and client code stubs from proto files
    ○ Scala wrappers:
    ■ ScalaPB, muScala, akka-grpc, etc.
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    21
    

    View Slide

22. Copyright 1995-2020 Treasure Data. All rights reserved.
    New Approach: Defining RPC Interface with Scala
    ● Scala is a perfect fit for RPC
    ○ Scala functions = RPC methods
    ○ Scala objects (statically typed) = RPC request/response types
    ● Using the same interface and model classes between servers and clients
    ○ No need to think about REST nor ProtocolBuffers for implementing RPC
    22
    Airframe
    

    View Slide

23. Copyright 1995-2020 Treasure Data. All rights reserved.
    Necessary Building Blocks for Implementing RPC
    ● 1. Message serializer and deserializer
    ● 2. Network data format
    ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
    ● 3. RPC interface language
    ○ REST API, Protobuf, or Scala
    ● 4. HTTP client and server implementation
    ○ Code generator from RPC interface
    23
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    

    View Slide

24. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: MessagePack-Based Object Serialization
    ● MessagePack
    ○ A compact binary format compatible with
    JSON
    Object Object
    Pack Unpack
    Pack
    Unpack
    Server Side
    Client Side
    24
    Scala.js
    JSON
    

    View Slide

25. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: Predefined Codecs
    ● Primitive Codec
    ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
    ○ FloatCodec, DoubleCodec
    ○ StringCodec
    ○ BooleanCodec
    ○ TimeStampCodec
    ● Collection Codec
    ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec
    ○ OptionCodec, EitherCodec
    ● Java-specific Codec
    ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
    ○ JDBCResultSetCodec
    ● etc.
    25
    Airframe
    

    View Slide

26. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: Serializing Complex Objects
    Pack
    Unpack
    IntCodec
    StringCodec
    DoubleCodec
    MessageCodec.of[A]
    26
    Serialize
    Deserialize
    JSON
    ● airframe-surface
    ○ Inspecting object parameters using compile-time macros
    ○ Different implementations between Scala 2 and Scala 3
    

    View Slide

27. Copyright 1995-2020 Treasure Data. All rights reserved.
    Object Serialization for Multiple Data Formats
    ● Data to Object Mapping
    ○ We need many data readers as well as object mappers
    ● A lot of existing libraries for parsing JSON
    ○ circe, json4s, play-json, jackson, etc.
    YAML
    JDBC
    ResultSet
    YAML Parser +
    Object Mapper
    Config
    Object
    Table
    Object
    Object-Relation
    Mapper
    JSON
    JSON Parser +
    Object Mapper
    Object
    27
    

    View Slide

28. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: Using MessagePack As A Universal Data Format
    Object
    Unpack
    Pack
    JDBC
    ResultSet
    Pack/Unpack
    YAML
    JSON
    28
    ● Once your data are converted into MessagePack, you can use the same object
    serializer/deserializer (airframe-codec)
    ● Airframe RPC supports JSON format through MessagePack conversion
    Airframe
    airframe-codec
    

    View Slide

29. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe RPC: Serializing Function Call Request
    ● Serialize function call arguments as Map (arg_name -> arg_value)
    ● An example:
    ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
    ■ Serialize this Map into MessagePack with airframe-codec
    ● Mapping to HTTP requests
    ○ HTTP method: POST
    ○ Path: /(package name).(class name)/(method name)
    ■ e.g., /hello.api.v1.MyService/hello
    ○ Content body: serialized function call arguments
    29
    

    View Slide

30. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe RPC: Implementing RPC Servers
    ● Extending the RPC interface trait
    30
    

    View Slide

31. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe RPC: Backend Servers/Clients Are Pluggable
    RPC Interface RPC Web Server
    Generates API Documentation
    31
    Airframe
    RPC Call RPC Call
    Implements
    ● Finagle (HTTP/1)
    ● gRPC (HTTP/2)
    RPC Clients
    

    View Slide

32. Copyright 1995-2020 Treasure Data. All rights reserved.
    ● Finagle: Twitter’s HTTP server written in Scala
    ○ For HTTP/1 services accessible from web browsers running Scala.js code
    AIrframe RPC: Finagle Backend
    32
    Airframe
    Scala.js
    airframe-rx-html
    airframe-http-finagle
    

    View Slide

33. Copyright 1995-2020 Treasure Data. All rights reserved.
    ● gRPC
    ○ grpc-java (using Netty as backend)
    ● Airframe RPC + gRPC
    ○ No .proto file is required
    ○ Only Scala
    Airframe RPC: gRPC Backend
    33
    Airframe
    

    View Slide

34. Copyright 1995-2020 Treasure Data. All rights reserved.
    RPC Performance Comparison: Greeter Service
    ● Airframe RPC
    ○ serde: MessagePack (airframe-codec) -
    Scala case classes
    ○ Finagle (HTTP1) or gRPC (HTTP2)
    ● ScalaPB
    ○ serde: Protobuf - Scala case classes
    ○ gRPC (HTTP2)
    ● grpc-java
    ○ serde: Protobuf - Java classes
    ○ gRPC (HTTP2)
    ● Notes
    ○ Using Finagle with HTTP2 had almost no
    benefit over HTTP1
    ○ Multiplexing RPC requests into a
    single connection is the key
    performance factor
    ○ Overhead of ScalaPB over grpc-java
    ■ Scala Future (10% overhead)
    ■ Mapping Protobuf to Scala case
    classes (10% overhead)
    34
    HTTP/1 HTTP/2 (gRPC)
    

    View Slide

35. Copyright 1995-2020 Treasure Data. All rights reserved.
    [Advanced] Extending gRPC for Other Data Formats
    ● gRPC is data-format agonistic
    framework
    ○ You can use other than
    ProtocolBuffers
    ■ e.g., JSON, MessagePack
    ● There are two extension points:
    ● MethodDescriptor
    ○ Define gRPC endpoints
    corresponding to Scala functions
    ○ Register custom marshallers
    ● Request/ResponseMarshaller
    ○ Define how to encode/decode RPC
    request/response data
    ○ We are using airframe-codec here
    35
    

    View Slide

36. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating RPC Clients
    ● sbt-airframe plugin
    ○ Read RPC interface classes and generate HTTP client code for the target backend
    ■ Using different code generators for Scala, Scala.js, and gRPC
    sbt-airframe
    Code
    Generation
    RPC Client
    Scala.js
    36
    Scala.js Client
    HTTP/gRPC Client
    Open API Spec
    Cross-Language
    RPC Client
    

    View Slide

37. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating Open API Schema
    ● RPC Interface -> Open API schema YAML file
    ● Generating RPC Clients & Swagger Documentation
    37
    RPC Interface
    API Documentation
    Open API Spec
    (YAML)
    Cross-Language
    RPC Client
    Generate
    sbt-airframe
    API Documentation
    

    View Slide

38. Copyright 1995-2020 Treasure Data. All rights reserved.
    Making An RPC Call
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Response
    Server
    Call
    Response
    Airframe
    Scala.js
    38
    

    View Slide

39. Copyright 1995-2020 Treasure Data. All rights reserved.
    Summary: Enable Scala-Oriented Development with Airframe
    39
    RPC Interface
    Scala.js Client RPC Web Server
    Generates
    Scala.js
    Web Application
    API Documentation
    RPC Call RPC Call
    Implements
    Airframe
    ● Kick start frontend/backend application development with Scala
    ○ Only one language to learn
    ○ No need to think about REST, ProtocolBuffers, other frontend frameworks at the beginning.
    ● Find example code at https://github.com/wvlet/airframe/ (examples folder)
    

    View Slide