Unifying Frontend and Backend Development with Scala - Scala Con 2021

Scala can be used for developing both frontend (Scala.js) and backend (Scala JVM) applications. A missing piece has been bridging these two worlds using Scala. We built Airframe RPC, a framework that uses Scala traits as a unified RPC interface between servers and clients. With Airframe RPC, you can build HTTP/1 (Finagle) and HTTP/2 (gRPC) services just by defining Scala traits and case classes. It simplifies web application design as you only need to care about Scala interfaces without using existing web standards like REST, ProtocolBuffers, OpenAPI, etc. Scala.js support of Airframe also enables building interactive Web applications that can dynamically render DOM elements while talking with Scala-based RPC servers. With Airframe RPC, the value of Scala developers will be much higher both for frontend and backend areas.

Presentation Video at Scala Con 2021: https://www.youtube.com/watch?v=MliLBejnyy8

1. © 2020 Treasure Data 1 Taro L. Saito Treasure Data November 2nd, 2021 ScalaCon 2021 Unifying Frontend and Backend Development with Scala
2. Copyright 1995-2020 Treasure Data. All rights reserved. About Me: Taro L. Saito 2 • Treasure Data ◦ Principal software engineer ◦ Building distributed query engine services • OSS activities ◦ Using Scala for 13+ years ▪ Since Scala 2.7 (2008) ◦ Airframe (This talk) ▪ 20+ Scala utility modules ◦ sbt-sonatype ▪ Publishing to Maven central (Sonatype) ◦ etc. Airframe
3. Copyright 1995-2020 Treasure Data. All rights reserved. Treasure Data: A Ready-to-Use Cloud Data Platform 3 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data • Even if you are not an engineer, you can start collecting and analyzing your data
4. Copyright 1995-2020 Treasure Data. All rights reserved. Even Backend Engineers Need Frontend Skills • Creating a better visualization of services ◦ To monitor millions of queries running every day ◦ Created a UI for service monitoring and cluster management with Scala.js 4
5. Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Application: Notebook Query Editor • Rich SQL editor • Frontend ◦ Scala.js ◦ Monaco editor of Visual Studio Code • Backend ◦ Scala on JVM ◦ RPC server ▪ SQL runner ▪ Task manager • Written only in Scala 5
6. Copyright 1995-2020 Treasure Data. All rights reserved. Too Many Frontend and Backend Frameworks • How do we choose one of the frameworks to use? • It’s almost impossible to find Scala engineers mastering all of these frontend and backend technologies. 6 Web Browsers Scala.js
7. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC • Using plain Scala as an RPC interface between frontend and backend applications • Low-learning cost ◦ Requires only basic Scala knowledge, which can be learned with one Scala book • Our experience so far after introducing Airframe RPC: ◦ A Java engineer in Treasure Data can start writing a Scala RPC service interacting with a Scala.js UI in a week. 7 RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe
8. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe • GitHub: wvlet/airframe • 20+ useful modules written in pure Scala ◦ Cross-built for Scala 2.12, 2.13, and Scala.js ◦ Scala 3 support is in progress ▪ 85%~ completed ▪ WIP: • Tasty reader & Scala reflection compatibility • Core Modules (*: will be explained in this talk) ◦ airframe-log* ◦ airframe-codec* ▪ Object serializer ◦ airframe-http* ▪ HTTP server/client with RPC support ◦ airframe-rx* ▪ Reactive DOM rendering ◦ airframe-json/msgpack ▪ JSON/MsgPack parser ◦ airframe-di ▪ Constructor injection helper ◦ airframe-surface ▪ Compile-time or runtime object inspector ◦ AirSpec: Testing library integrated with Airframe DI ◦ etc. 8 Airframe
9. Copyright 1995-2020 Treasure Data. All rights reserved. • Various internal and third-party Scala/Java libraries were used ◦ e.g., Many web frameworks, utilities, testing frameworks, etc. • The cost of learning individual libraries and frameworks was high Before Airframe: 5 Years Ago (2016) 9 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming Various Libraries Outcome logger launcher object mapper JDBC reader json4s jackson ….
10. Copyright 1995-2020 Treasure Data. All rights reserved. Current (2021): Consolidating the Best Practices into Airframe OSS • Since 2016, we started defining standards in Airframe ◦ Engineers can focus on developing applications instead of learning frameworks 10 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe
11. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-log: A Modern Logging Library for Scala and Scala.js • There were too many choices just for generating human-readable logs ◦ log4j, slf4j, scala-logging, java.util.logging, twitter/util, etc. • airframe-log ◦ Adding colorful logging methods to your class with LogSupport trait ◦ Showing the source code location 11
12. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-log usage in sbt-sonatype • Shows detailed logs while publishing projects to Maven Central (Sonatype OSS Repository) • News: sbt-sonatype update ◦ Since 2019, sbt-sonatype introduced a new bundle upload process, which can upload thousands of artifact files (jars, source files) to Sonatype within a minute. 12
13. Copyright 1995-2020 Treasure Data. All rights reserved. • Scala.js compiles Scala code into JavaScript, which runs on web browsers • Logging for the developer console of web browsers airframe-log: Debugging Scala.js Code 13
14. Copyright 1995-2020 Treasure Data. All rights reserved. Frontend Development: Rendering DOM with Scala.js • DOM element: ◦ <div class=”container”> …. </div> ▪ tag, attribute, enclosed elements ◦ scalajs-dom library has all necessary functions for manipulating DOM nodes • A previous approach for DOM rendering: ◦ Converting Scala’s XML literal into DOM tree nodes ▪ OSS: monadic-html, Binding.scala ▪ Similar to JSX in Facebook React (XML syntax embedded to JavaScript) ◦ But, XML literal will be deprecated in Scala 3 14
15. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Functional DOM Rendering • The current best practice is defining functions for rendering HTML elements ◦ table, tr, td, body, div, span, etc. • Then, compose these functions • Style 1: Separating DOM attributes and elements into different blocks ◦ ScalaTags, Slinky ◦ Some disadvantages ▪ Too many parentheses! ▪ Formatting with scalafmt is a bit challenging • Style 2: Enclosing both DOM attributes and elements inside a single block ◦ airframe-rx, scalajs-react ◦ More Scala-friendly syntax ◦ The begin/end of a tag are clear 15 ScalaTags airframe-rx
16. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Rendering Complex DOM • RxElement interface: ◦ def render(): RxElement ◦ Support nesting with apply(...) ▪ Embedding Scala functions and collections 16
17. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Reactive Stream Interface Rx[A] • Rx[A] ◦ ReactiveX stream implementation • Rewriting DOM interactively upon upstream data changes ◦ e.g., mouse clicks, getting RPC responses (Future), interval timer events, etc. • Reactive Operators ◦ map, flatMap, filter, join, zip, etc. ◦ The usage is almost the same with Scala collection APIs 17 Airframe
18. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Rx.variable • Rx.variable ◦ Fire an event if the variable is updated • Interactive DOM Rendering ◦ Subscribing upstream changes ▪ Update DOMs for each change ◦ Cancellation ▪ Unsubscribe the event processing ▪ Necessary for cleaning up event handlers added to the browser • mouse click handler, timer, etc. 18
19. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Gallery 19
20. Copyright 1995-2020 Treasure Data. All rights reserved. RPC: Bridging Scala and Scala.js Applications Program Function Call Return Local Function Call (ideal) Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Remote Procedure Call (reality) Network Client Call Return Server Call Return 20 Scala.js Scala.js
21. Copyright 1995-2020 Treasure Data. All rights reserved. Traditional Approaches for Implementing RPC • REST API ◦ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.) ◦ REST web frameworks for Scala ▪ Play framework, akka-http, Finatra, Finch, skinny-framework, etc. • gRPC ◦ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language ◦ Generate server and client code stubs from proto files ◦ Scala wrappers: ▪ ScalaPB, muScala, akka-grpc, etc. Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return 21
22. Copyright 1995-2020 Treasure Data. All rights reserved. New Approach: Defining RPC Interface with Scala • Scala is a perfect fit for RPC ◦ Scala functions = RPC methods ◦ Scala objects (statically typed) = RPC request/response types • Using the same interface and model classes between servers and clients ◦ No need to think about REST nor ProtocolBuffers for implementing RPC 22 Airframe
23. Copyright 1995-2020 Treasure Data. All rights reserved. Necessary Building Blocks for Implementing RPC • 1. Message serializer and deserializer • 2. Network data format ◦ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC) • 3. RPC interface language ◦ REST API, Protobuf, or Scala • 4. HTTP client and server implementation ◦ Code generator from RPC interface 23 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return
24. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: MessagePack-Based Object Serialization • MessagePack ◦ A compact binary format compatible with JSON Object Object Pack Unpack Pack Unpack Server Side Client Side 24 Scala.js JSON
25. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Predefined Codecs • Primitive Codec ◦ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ◦ FloatCodec, DoubleCodec ◦ StringCodec ◦ BooleanCodec ◦ TimeStampCodec • Collection Codec ◦ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec ◦ OptionCodec, EitherCodec • Java-specific Codec ◦ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec ◦ JDBCResultSetCodec • etc. 25 Airframe
26. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Serializing Complex Objects Pack Unpack IntCodec StringCodec DoubleCodec MessageCodec.of[A] 26 Serialize Deserialize JSON • airframe-surface ◦ Inspecting object parameters using compile-time macros ◦ Different implementations between Scala 2 and Scala 3
27. Copyright 1995-2020 Treasure Data. All rights reserved. Object Serialization for Multiple Data Formats • Data to Object Mapping ◦ We need many data readers as well as object mappers • A lot of existing libraries for parsing JSON ◦ circe, json4s, play-json, jackson, etc. YAML JDBC ResultSet YAML Parser + Object Mapper Config Object Table Object Object-Relation Mapper JSON JSON Parser + Object Mapper Object 27
28. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Using MessagePack As A Universal Data Format Object Unpack Pack JDBC ResultSet Pack/Unpack YAML JSON 28 • Once your data are converted into MessagePack, you can use the same object serializer/deserializer (airframe-codec) • Airframe RPC supports JSON format through MessagePack conversion Airframe airframe-codec
29. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Serializing Function Call Request • Serialize function call arguments as Map (arg_name -> arg_value) • An example: ◦ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”) ▪ Serialize this Map into MessagePack with airframe-codec • Mapping to HTTP requests ◦ HTTP method: POST ◦ Path: /(package name).(class name)/(method name) ▪ e.g., /hello.api.v1.MyService/hello ◦ Content body: serialized function call arguments 29
30. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Implementing RPC Servers • Extending the RPC interface trait 30
31. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Backend Servers/Clients Are Pluggable RPC Interface RPC Web Server Generates API Documentation 31 Airframe RPC Call RPC Call Implements • Finagle (HTTP/1) • gRPC (HTTP/2) RPC Clients
32. Copyright 1995-2020 Treasure Data. All rights reserved. • Finagle: Twitter’s HTTP server written in Scala ◦ For HTTP/1 services accessible from web browsers running Scala.js code AIrframe RPC: Finagle Backend 32 Airframe Scala.js airframe-rx-html airframe-http-finagle
33. Copyright 1995-2020 Treasure Data. All rights reserved. • gRPC ◦ grpc-java (using Netty as backend) • Airframe RPC + gRPC ◦ No .proto file is required ◦ Only Scala Airframe RPC: gRPC Backend 33 Airframe
34. Copyright 1995-2020 Treasure Data. All rights reserved. RPC Performance Comparison: Greeter Service • Airframe RPC ◦ serde: MessagePack (airframe-codec) - Scala case classes ◦ Finagle (HTTP1) or gRPC (HTTP2) • ScalaPB ◦ serde: Protobuf - Scala case classes ◦ gRPC (HTTP2) • grpc-java ◦ serde: Protobuf - Java classes ◦ gRPC (HTTP2) • Notes ◦ Using Finagle with HTTP2 had almost no benefit over HTTP1 ◦ Multiplexing RPC requests into a single connection is the key performance factor ◦ Overhead of ScalaPB over grpc-java ▪ Scala Future (10% overhead) ▪ Mapping Protobuf to Scala case classes (10% overhead) 34 HTTP/1 HTTP/2 (gRPC)
35. Copyright 1995-2020 Treasure Data. All rights reserved. [Advanced] Extending gRPC for Other Data Formats • gRPC is data-format agonistic framework ◦ You can use other than ProtocolBuffers ▪ e.g., JSON, MessagePack • There are two extension points: • MethodDescriptor ◦ Define gRPC endpoints corresponding to Scala functions ◦ Register custom marshallers • Request/ResponseMarshaller ◦ Define how to encode/decode RPC request/response data ◦ We are using airframe-codec here 35
36. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating RPC Clients • sbt-airframe plugin ◦ Read RPC interface classes and generate HTTP client code for the target backend ▪ Using different code generators for Scala, Scala.js, and gRPC sbt-airframe Code Generation RPC Client Scala.js 36 Scala.js Client HTTP/gRPC Client Open API Spec Cross-Language RPC Client
37. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating Open API Schema • RPC Interface -> Open API schema YAML file • Generating RPC Clients & Swagger Documentation 37 RPC Interface API Documentation Open API Spec (YAML) Cross-Language RPC Client Generate sbt-airframe API Documentation
38. Copyright 1995-2020 Treasure Data. All rights reserved. Making An RPC Call Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Response Server Call Response Airframe Scala.js 38
39. Copyright 1995-2020 Treasure Data. All rights reserved. Summary: Enable Scala-Oriented Development with Airframe 39 RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe • Kick start frontend/backend application development with Scala ◦ Only one language to learn ◦ No need to think about REST, ProtocolBuffers, other frontend frameworks at the beginning. • Find example code at https://github.com/wvlet/airframe/ (examples folder)