1 Taro L. Saito Treasure Data October 17th, 2020 Scala Matsuri 2020 Scala For Everything From Frontend to Backend Applications Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ 


Copyright 1995-2020 Treasure Data. All rights reserved. About Me: Taro L. Saito 2 ● Treasure Data ○ Principal Software Engineer ○ Living in US for 5 years ● Ph.D. of Computer Science ○ Database systems, genome science. ● OSS: ○ Airframe, snappy-java (used in Parquet, Spark, etc.), sbt-sonatype, sbt-pack, etc. ● Book: 自己紹介


Copyright 1995-2020 Treasure Data. All rights reserved. Today’s Topic ● There are a lot of libraries and frameworks for backend and frontend applications ● Airframe ○ An OSS library collection that enables Scala-oriented development ● Goal: Forget everything other than Scala 3 Scala無双：Scala以外を全て駆逐する 
 Airframe Scala.js Web Browsers JSON

Copyright 1995-2020 Treasure Data. All rights reserved. Airframe: Core Scala Modules of Treasure Data ● Airframe (GitHub: wvlet/airframe) ○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019) ■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc. ○ Airframe RPC, Airframe Rx (Scala Matsuri 2020) ○ AirSpec ■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021) 4 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群 


Copyright 1995-2020 Treasure Data. All rights reserved. Treasure Data: A Ready-to-Use Cloud Data Platform 5 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data トレジャーデータはすぐに使えるクラウドデータプラットフォーム 


Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Application: Presto Service Monitor ● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions 6 Scala.jsで開発されたアプリケーションの例 


Copyright 1995-2020 Treasure Data. All rights reserved. Server APIs and Library for Spark Airframe DI DataFrame MPC1 airframe-codec airframe-msgpack Plazma Public API airframe-http airframe-finagle Airframe DI Airframe RPC airframe-fluentd Master Worker Design SparkContext TDSparkContext TDSparkService MPC1 Reader/Writer IO Manager Airframe DI airframe-http airframe-config airframe-launcher airframe-jmx airframe-metrics airframe-control airframe-metrics td-spark.jar airframe-log airframe-log airframe-codec airframe-json Airframe 7 Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている 
 ● MPC1: MessagePack Columnar Format ● See also td-spark internals (Spark Meetup Tokyo 2020)

Copyright 1995-2020 Treasure Data. All rights reserved. Scala is Versatile ● Backend ○ Can leverage JVM ecosystem (e.g., netty, gRPC, etc.) ● Frontend ○ Scala.js: ■ 1.0.0 is released in 2020 ● The current version is 1.3.0 ■ Compile Scala code into Javascripts ○ Out of scope of this talk ■ iOS/iPhone native applications ● How can we bridge server (Scala JVM) and client (Scala.js) applications? Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ？ 
 Scala.js Web Browsers Serve-Client Interaction 8

Copyright 1995-2020 Treasure Data. All rights reserved. Bridging Scala and Scala.js with RPC Program Function Call Return Local Function Call (ideal) Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる 
 Remote Procedure Call (reality) Network Client Call Return Server Call Return 9 Scala.js Scala.js

Copyright 1995-2020 Treasure Data. All rights reserved. Existing Approaches for Implementing RPC ● REST ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.) ○ REST web frameworks for Scala ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc. ● gRPC ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language ○ Generate server and client code stubs from proto files ○ Scala wrappers: ■ ScalaPB, muScala, akka-grpc, etc. RPCを実装するには、RESTやgRPCなどが使われている 
 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return 10

Copyright 1995-2020 Treasure Data. All rights reserved. API Design War https://cloud.google.com/blog/products/api-management/ understanding-grpc-openapi-and-rest-and-when-to-use-th em 11 ● Which API design should we choose? ● REST API ○ Designing good APIs over HTTP requires experiences and significant amount of time ● gRPC ○ Requires special software like code generators for using the ecosystem around ProtocolBuffers ● And also, there are too many choices for web frameworks しかし、APIデザイン、フレームワーク選びでの争いは付きない 


Copyright 1995-2020 Treasure Data. All rights reserved. Why Not Using Scala for RPC Interface? ● Scala is a functional and statically-typed object-oriented programming language ○ Scala functions = RPC methods ○ Scala objects (statically typed) = RPC request/response types ● Scala is a perfect fit for RPC! 12 関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している 


Copyright 1995-2020 Treasure Data. All rights reserved. Scala-First Approach: Airframe RPC RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation 13 Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法 
 RPC Call RPC Call Implements ● Forget about REST or ProtocolBuffers! ● Use Scala as a unified interface between server and clients Airframe

Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC Backend Is Pluggable RPC Interface RPC Web Server Generates API Documentation 14 Airframe RPCのServer, Clientは差し替え可能 
 Airframe RPC Call RPC Call Implements ● Backend server ○ Finagle (HTTP/1), gRPC (HTTP/2) ● Backend client ○ Finagle, gRPC, okhttp, Java URLConnection client etc. RPC Clients

Copyright 1995-2020 Treasure Data. All rights reserved. Building Blocks of RPC Frameworks ● 1. Message serializer and deserializer ● 2. Network data format ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC) ● 3. RPC interface language ○ REST API, Protobuf, or Scala ● 4. HTTP client and server implementation ○ Code generator 15 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return RPCフレームワークを作るのに必要な構成要素 


Copyright 1995-2020 Treasure Data. All rights reserved. MessagePack-Based Object Serialization ● MessagePack ○ A compact binary format compatible with JSON Object Object Pack Unpack Pack Unpack Server Side Client Side 16 Scala.js JSON Airframe MessagePackを使ってクライアントとサーバー間通信を行う 


Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Pack/Unpack Interface ● Pack ○ Converting Data to MessagePack ● Unpack ○ Converting MessagePack to Data Input Output Pack Unpack Pack Unpack 17 Airframe MessageCodec MessageCodec MessagePackを通したデータ変換インターフェースをScalaに適用 


Copyright 1995-2020 Treasure Data. All rights reserved. Pre-defined Codecs in airframe-codec ● Primitive Codecs ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ○ FloatCodec, DoubleCodec ○ StringCodec ○ BooleanCodec ○ TimeStampCodec ● Collection Codec ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc. ○ OptionCodec ● Java-specific Codec ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec ○ JDBCResultSetCodec ● etc. 18 Scalaで必要なほぼ全てのデータ型へのマッピングをサポート 
 Airframe

Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Complex Objects Pack Unpack IntCodec StringCodec DoubleCodec MessageCodec.of[A] 19 オブジェクトの型に合わせてCodecを合成できる 
 Serialize Deserialize JSON

Copyright 1995-2020 Treasure Data. All rights reserved. Defining RPC Interface with Scala ● Add ＠RPC annotation to a trait (= RPC interface) ● Define model classes (= data structures) using case classes ● Use the same interface and model classes between servers and clients RPC InterfaceをScalaで記述する 
 20

Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Function Call Request ● Function call arguments ○ A sequence of argument name -> value, …. ● Example: ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”) ○ Serialize this Map into MessagePack with airframe-codec ● HTTP Mapping ○ HTTP method: POST ○ Path: /(package name)/hello ■ e.g., /hello.api.v1/hello ○ Content body: serialized function call argument data 21 関数呼び出しの引数もシリアライズする 


Copyright 1995-2020 Treasure Data. All rights reserved. Implementing RPC Servers ● Extending the RPC interface trait Serverの実装はRPC traitの継承 
 22

Copyright 1995-2020 Treasure Data. All rights reserved. ● Finagle: Twitter’s HTTP server written in Scala ○ HTTP/1 based. Accessible from web browsers running Scala.js code airframe-http: Starting A Finagle Backend RPC Server 23 Airframe Scala.js airframe-rx-html airframe-http: FinagleバックエンドのHTTPサーバーを起動する 
 airframe-http-finagle

Copyright 1995-2020 Treasure Data. All rights reserved. airframe-http: gRPC Backend ● Suited to internal microservices ● HTTP/2 based protocol ○ Multiplexing multiple RPC requests within a single connection ● No .proto file is required 24 Airframe airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない 


Copyright 1995-2020 Treasure Data. All rights reserved. Advanced Topic: Extending gRPC ● gRPC is data-format agonistic framework ○ To support Scala interface and MessagePack, we have two extension points in grpc-java. ● MethodDescriptor ○ Define gRPC endpoints corresponding to Scala functions ○ Register marshallers ● Request/ResponseMarshaller ○ Define how to encode/decode RPC request/response data 25 Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている 


Copyright 1995-2020 Treasure Data. All rights reserved. RPC Performance Comparison (Greeter Service) ● Airframe RPC ○ serde: MessagePack (airframe-codec) <-> Scala case classes ○ Finagle (HTTP1) or gRPC (HTTP2) ● ScalaPB ○ serde: Protobuf <-> Scala case classes ○ gRPC (HTTP2) ● grpc-java ○ serde: Protobuf <-> Java classes ○ gRPC (HTTP2) ● Notes ○ Using Finagle with HTTP2 had almost no benefit over HTTP1 ■ Multiplexing RPC requests over HTTP2 is the key performance factor ○ Overhead of ScalaPB ■ Scala Future (10% overhead) ■ Mapping Protobuf to Scala case classes (10% overhead) 26 gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない 
 HTTP/1 HTTP/2 (gRPC)

Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating RPC Clients ● sbt-airframe plugin ○ Read RPC interface classes ○ Generate HTTP client code for accessing the RPC server ○ OpenAPI schema generation support sbt-airframe Code Generation RPC Client Scala.js 27 Scala.js Client HTTP/gRPC Client Open API Spec Cross-Language RPC Client RPCクライアントを生成するsbt-airframeプラグイン 


Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating Open API Schema ● RPC Interface -> Open API schema YAML file ● Generating RPC Clients & Swagger Documentation 28 ScalaのインターフェースからOpenAPIのYAMLを生成する 
 RPC Interface API Documentation Open API Spec (YAML) Cross-Language RPC Client Generate sbt-airframe API Documentation

Copyright 1995-2020 Treasure Data. All rights reserved. Making An RPC Call RPCを実行する。サーバー、クライアント共にScalaで動いている 
 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Response Server Call Response Airframe Scala.js 29

Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Programming Tips ● Scala.js: Compiling Scala -> JavaScript ● Use Pure Scala Code ○ [NG] Java code ■ Dependent libraries also need to be pure Scala ● Airframe is carefully designed to minimize non-Scala dependencies ○ [NG] Annotation, runtime-reflection ○ [OK] Scala Macros (compile-time meta-programming) ● RPC ○ Asynchronous calls using Scala Future ○ Await(Future) is not supported ■ Because JS is a single-threaded model. ■ Need to properly chain Future operators with map, flatMap, rescue, etc. 30 Scala.jsでのプログラミングのコツ 
 Scala.js

Copyright 1995-2020 Treasure Data. All rights reserved. Debugging Scala.js Code with airframe-log ● Showing logs at the developer console of the browser 31 airframe-logとブラウザのコンソールでScala.jsコードのデバッグ 


Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM with Scala.js ● DOM element: ○ ○ scalajs-dom library has functions necessary for generating DOMs ● Previous approach: Converting XML literal into DOM elements ○ monadic-html, Binding.scala ○ But XML literal will be deprecated in Dotty (Scala 3.0) 32 Scala.jsでのDOMレンダリング手法には様々な手法がある 


Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM ● Current Best Practice ○ Using Scala functions for building DOM elements ● Option 1: Separation between DOM elements and attributes ○ ScalaTags, Slinky ○ Too many parentheses! ● Option 2: No separation between DOM elements and attributes ○ scalajs-react, Airframe Rx ○ More Scala-friendly syntax 33 DOMのelement, attributeを区別しない方が関数型に適している 
 ScalaTags Airframe Rx

Copyright 1995-2020 Treasure Data. All rights reserved. Complex DOM Rendering ● Airframe Rx ● RxElement interface ○ def render(): RxElement ○ Support nesting with apply(...) ■ Embedding Scala functions and collections 34 airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする 


Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Rx ● Rx.variable ○ Re-render DOM if the variable is updated ● Gathering best practices from existing libraries ○ Type-safe DOM elements and attributes ■ ScalaTags, scalajs-react ○ Reactive streaming ■ Monix ■ Scala Collection ■ Scala.rx ○ Cancelable Interface ■ Cleanup DOM and event handlers when updating DOM ■ Monix, monadic-html 35 既存技術の良いところ取りをしたAirframe Rx 


Copyright 1995-2020 Treasure Data. All rights reserved. Reactive Stream Interface: Rx[X] ● Rx[X] ○ ReactiveX stream implementation ○ e.g., Monix, Scala.Rx ● Reactive DOM Rendering ○ Rewriting DOM interactively upon upstream data change ● Reactive Operators ○ map, flatMap, filter, join, etc. ○ Almost the same with Scala collection APIs 36 Reactive stream interface Rx[X]でインタラクティブなDOMを作成 


Copyright 1995-2020 Treasure Data. All rights reserved. gRPC Streaming with Airframe Rx ● Interactive communication between server and clients with reactive stream interface: Rx[A] 37 gRPC StreamingもRxインターフェースでサポートできる 


Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Enables Scala-Oriented Development 38 RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe AirframeでScala中心のアプリケーション開発を可能に 
 ● Only need to learn how to write Scala for kick-starting client/server application development

Copyright 1995-2020 Treasure Data. All rights reserved. Summary ● Scala is a powerful language that can consolidate existing technologies ● Minimizing the learning cost of application development with Scala-First approach ○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose 39 まとめ：既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe 
 Airframe Scala.js Web Browsers JSON