Scala For Everything: From Frontend to Backend Applications - ScalaMatsuri 2020

ScalaMatsuri 2020 presentation slides.
- Japanese Title: Scala無双: フロントエンドもバックエンドも全てScalaでなぎ倒せ

Scala is a powerful language; You can build frontend applications with Scala.js, and efficient backend application servers for JVM. In this session, we will learn how to build everything with Scala by using Airframe OSS framework.

Airframe is a library designed for maximizing the advantages of Scala as a hybrid of object-oriented and functional programming language. In this session, we will learn how to use Airframe to build REST APIs and RPC services (Finagle, gRPC backend), and how to create frontend applications in Scala.js that interact with the servers using functional interfaces for dynamically updating web pages.

Transcript

1. 1
   Taro L. Saito
   Treasure Data
   October 17th, 2020
   Scala Matsuri 2020
   Scala For Everything
   From Frontend to Backend Applications
   Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ
   

   

   View Slide

2. Copyright 1995-2020 Treasure Data. All rights reserved.
   About Me: Taro L. Saito
   2
   ● Treasure Data
   ○ Principal Software Engineer
   ○ Living in US for 5 years
   ● Ph.D. of Computer Science
   ○ Database systems, genome science.
   ● OSS:
   ○ Airframe, snappy-java (used in
   Parquet, Spark, etc.), sbt-sonatype,
   sbt-pack, etc.
   ● Book:
   自己紹介

   

   View Slide

3. Copyright 1995-2020 Treasure Data. All rights reserved.
   Today’s Topic
   ● There are a lot of libraries and frameworks for backend and frontend applications
   ● Airframe
   ○ An OSS library collection that enables Scala-oriented development
   ● Goal: Forget everything other than Scala
   3
   Scala無双：Scala以外を全て駆逐する
   

   Airframe
   Scala.js Web Browsers
   JSON
   

   View Slide

4. Copyright 1995-2020 Treasure Data. All rights reserved.
   Airframe: Core Scala Modules of Treasure Data
   ● Airframe (GitHub: wvlet/airframe)
   ○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019)
   ■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc.
   ○ Airframe RPC, Airframe Rx (Scala Matsuri 2020)
   ○ AirSpec
   ■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021)
   4
   Knowledge
   Experiences
   Design Decisions
   Products
   24/7 Services
   Business Values
   Programming OSS Outcome
   Airframe
   AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群
   

   

   View Slide

5. Copyright 1995-2020 Treasure Data. All rights reserved.
   Treasure Data: A Ready-to-Use Cloud Data Platform
   5
   Logs
   Device
   Data
   Batch
   Data
   PlazmaDB
   Table Schema
   Data Collection Cloud Storage Distributed Data Processing
   Jobs
   Job Management
   SQL Editor
   Scheduler
   Workflows
   Machine
   Learning
   Treasure Data OSS
   Third Party OSS
   Data
   トレジャーデータはすぐに使えるクラウドデータプラットフォーム
   

   

   View Slide

6. Copyright 1995-2020 Treasure Data. All rights reserved.
   Scala.js Application: Presto Service Monitor
   ● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions
   6
   Scala.jsで開発されたアプリケーションの例
   

   

   View Slide

7. Copyright 1995-2020 Treasure Data. All rights reserved.
   Server APIs and Library for Spark
   Airframe DI
   DataFrame MPC1
   airframe-codec
   airframe-msgpack
   Plazma Public API
   airframe-http
   airframe-finagle
   Airframe DI
   Airframe RPC
   airframe-fluentd
   Master Worker
   Design
   SparkContext
   TDSparkContext TDSparkService
   MPC1 Reader/Writer IO Manager
   Airframe DI
   airframe-http
   airframe-config
   airframe-launcher
   airframe-jmx
   airframe-metrics
   airframe-control
   airframe-metrics
   td-spark.jar
   airframe-log
   airframe-log
   airframe-codec
   airframe-json
   Airframe
   7
   Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている
   

   ● MPC1: MessagePack Columnar Format
   ● See also td-spark internals (Spark Meetup Tokyo
   2020)
   

   View Slide

8. Copyright 1995-2020 Treasure Data. All rights reserved.
   Scala is Versatile
   ● Backend
   ○ Can leverage JVM ecosystem (e.g.,
   netty, gRPC, etc.)
   ● Frontend
   ○ Scala.js:
   ■ 1.0.0 is released in 2020
   ● The current version is 1.3.0
   ■ Compile Scala code into
   Javascripts
   ○ Out of scope of this talk
   ■ iOS/iPhone native applications
   ● How can we bridge server (Scala JVM)
   and client (Scala.js) applications?
   Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ？
   

   Scala.js
   Web Browsers
   Serve-Client
   Interaction
   8
   

   View Slide

9. Copyright 1995-2020 Treasure Data. All rights reserved.
   Bridging Scala and Scala.js with RPC
   Program Function
   Call
   Return
   Local Function Call
   (ideal)
   Program Function
   Serialize
   Deserialize
   Request Data
   Deserialize
   Response Data
   Serialize
   ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる
   

   Remote Procedure Call
   (reality)
   Network
   Client
   Call
   Return
   Server
   Call
   Return
   9
   Scala.js
   Scala.js
   

   View Slide

10. Copyright 1995-2020 Treasure Data. All rights reserved.
    Existing Approaches for Implementing RPC
    ● REST
    ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
    ○ REST web frameworks for Scala
    ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
    ● gRPC
    ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
    ○ Generate server and client code stubs from proto files
    ○ Scala wrappers:
    ■ ScalaPB, muScala, akka-grpc, etc.
    RPCを実装するには、RESTやgRPCなどが使われている
    

    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    10
    

    View Slide

11. Copyright 1995-2020 Treasure Data. All rights reserved.
    API Design War
    https://cloud.google.com/blog/products/api-management/
    understanding-grpc-openapi-and-rest-and-when-to-use-th
    em
    11
    ● Which API design should we choose?
    ● REST API
    ○ Designing good APIs over HTTP
    requires experiences and significant
    amount of time
    ● gRPC
    ○ Requires special software like code
    generators for using the ecosystem
    around ProtocolBuffers
    ● And also, there are too many choices for
    web frameworks
    しかし、APIデザイン、フレームワーク選びでの争いは付きない
    

    

    View Slide

12. Copyright 1995-2020 Treasure Data. All rights reserved.
    Why Not Using Scala for RPC Interface?
    ● Scala is a functional and statically-typed object-oriented programming language
    ○ Scala functions = RPC methods
    ○ Scala objects (statically typed) = RPC request/response types
    ● Scala is a perfect fit for RPC!
    12
    関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している
    

    

    View Slide

13. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala-First Approach: Airframe RPC
    RPC Interface
    Scala.js Client RPC Web Server
    Generates
    Scala.js
    Web Application
    API Documentation
    13
    Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法
    

    RPC Call RPC Call
    Implements
    ● Forget about REST or ProtocolBuffers!
    ● Use Scala as a unified interface between server and clients Airframe
    

    View Slide

14. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe RPC Backend Is Pluggable
    RPC Interface RPC Web Server
    Generates API Documentation
    14
    Airframe RPCのServer, Clientは差し替え可能
    

    Airframe
    RPC Call RPC Call
    Implements
    ● Backend server
    ○ Finagle (HTTP/1), gRPC (HTTP/2)
    ● Backend client
    ○ Finagle, gRPC, okhttp, Java URLConnection client etc.
    RPC Clients
    

    View Slide

15. Copyright 1995-2020 Treasure Data. All rights reserved.
    Building Blocks of RPC Frameworks
    ● 1. Message serializer and deserializer
    ● 2. Network data format
    ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
    ● 3. RPC interface language
    ○ REST API, Protobuf, or Scala
    ● 4. HTTP client and server implementation
    ○ Code generator
    15
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    RPCフレームワークを作るのに必要な構成要素
    

    

    View Slide

16. Copyright 1995-2020 Treasure Data. All rights reserved.
    MessagePack-Based Object Serialization
    ● MessagePack
    ○ A compact binary format compatible
    with JSON
    Object Object
    Pack Unpack
    Pack
    Unpack
    Server Side
    Client Side
    16
    Scala.js
    JSON
    Airframe
    MessagePackを使ってクライアントとサーバー間通信を行う
    

    

    View Slide

17. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: Pack/Unpack Interface
    ● Pack
    ○ Converting Data to MessagePack
    ● Unpack
    ○ Converting MessagePack to Data
    Input Output
    Pack Unpack
    Pack
    Unpack
    17
    Airframe
    MessageCodec MessageCodec
    MessagePackを通したデータ変換インターフェースをScalaに適用
    

    

    View Slide

18. Copyright 1995-2020 Treasure Data. All rights reserved.
    Pre-defined Codecs in airframe-codec
    ● Primitive Codecs
    ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
    ○ FloatCodec, DoubleCodec
    ○ StringCodec
    ○ BooleanCodec
    ○ TimeStampCodec
    ● Collection Codec
    ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
    ○ OptionCodec
    ● Java-specific Codec
    ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
    ○ JDBCResultSetCodec
    ● etc.
    18
    Scalaで必要なほぼ全てのデータ型へのマッピングをサポート
    

    Airframe
    

    View Slide

19. Copyright 1995-2020 Treasure Data. All rights reserved.
    Serializing Complex Objects
    Pack
    Unpack
    IntCodec
    StringCodec
    DoubleCodec
    MessageCodec.of[A]
    19
    オブジェクトの型に合わせてCodecを合成できる
    

    Serialize
    Deserialize
    JSON
    

    View Slide

20. Copyright 1995-2020 Treasure Data. All rights reserved.
    Defining RPC Interface with Scala
    ● Add ＠RPC annotation to a trait (= RPC interface)
    ● Define model classes (= data structures) using case classes
    ● Use the same interface and model classes between servers and clients
    RPC InterfaceをScalaで記述する
    

    20
    

    View Slide

21. Copyright 1995-2020 Treasure Data. All rights reserved.
    Serializing Function Call Request
    ● Function call arguments
    ○ A sequence of argument name -> value, ….
    ● Example:
    ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
    ○ Serialize this Map into MessagePack with airframe-codec
    ● HTTP Mapping
    ○ HTTP method: POST
    ○ Path: /(package name)/hello
    ■ e.g., /hello.api.v1/hello
    ○ Content body: serialized function call argument data
    21
    関数呼び出しの引数もシリアライズする
    

    

    View Slide

22. Copyright 1995-2020 Treasure Data. All rights reserved.
    Implementing RPC Servers
    ● Extending the RPC interface trait
    Serverの実装はRPC traitの継承
    

    22
    

    View Slide

23. Copyright 1995-2020 Treasure Data. All rights reserved.
    ● Finagle: Twitter’s HTTP server written in Scala
    ○ HTTP/1 based. Accessible from web browsers running Scala.js code
    airframe-http: Starting A Finagle Backend RPC Server
    23
    Airframe
    Scala.js
    airframe-rx-html
    airframe-http: FinagleバックエンドのHTTPサーバーを起動する
    

    airframe-http-finagle
    

    View Slide

24. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-http: gRPC Backend
    ● Suited to internal microservices
    ● HTTP/2 based protocol
    ○ Multiplexing multiple RPC
    requests within a single
    connection
    ● No .proto file is required
    24
    Airframe
    airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない
    

    

    View Slide

25. Copyright 1995-2020 Treasure Data. All rights reserved.
    Advanced Topic: Extending gRPC
    ● gRPC is data-format agonistic
    framework
    ○ To support Scala interface and
    MessagePack, we have two
    extension points in grpc-java.
    ● MethodDescriptor
    ○ Define gRPC endpoints
    corresponding to Scala
    functions
    ○ Register marshallers
    ● Request/ResponseMarshaller
    ○ Define how to encode/decode
    RPC request/response data
    25
    Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている
    

    

    View Slide

26. Copyright 1995-2020 Treasure Data. All rights reserved.
    RPC Performance Comparison (Greeter Service)
    ● Airframe RPC
    ○ serde: MessagePack (airframe-codec) <->
    Scala case classes
    ○ Finagle (HTTP1) or gRPC (HTTP2)
    ● ScalaPB
    ○ serde: Protobuf <-> Scala case classes
    ○ gRPC (HTTP2)
    ● grpc-java
    ○ serde: Protobuf <-> Java classes
    ○ gRPC (HTTP2)
    ● Notes
    ○ Using Finagle with HTTP2 had almost no
    benefit over HTTP1
    ■ Multiplexing RPC requests over
    HTTP2 is the key performance factor
    ○ Overhead of ScalaPB
    ■ Scala Future (10% overhead)
    ■ Mapping Protobuf to Scala case
    classes (10% overhead)
    26
    gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない
    

    HTTP/1 HTTP/2 (gRPC)
    

    View Slide

27. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating RPC Clients
    ● sbt-airframe plugin
    ○ Read RPC interface classes
    ○ Generate HTTP client code for accessing the RPC server
    ○ OpenAPI schema generation support
    sbt-airframe
    Code
    Generation
    RPC Client
    Scala.js
    27
    Scala.js Client
    HTTP/gRPC Client
    Open API Spec
    Cross-Language
    RPC Client
    RPCクライアントを生成するsbt-airframeプラグイン
    

    

    View Slide

28. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating Open API Schema
    ● RPC Interface -> Open API schema YAML file
    ● Generating RPC Clients & Swagger Documentation
    28
    ScalaのインターフェースからOpenAPIのYAMLを生成する
    

    RPC Interface
    API Documentation
    Open API Spec
    (YAML)
    Cross-Language
    RPC Client
    Generate
    sbt-airframe
    API Documentation
    

    View Slide

29. Copyright 1995-2020 Treasure Data. All rights reserved.
    Making An RPC Call
    RPCを実行する。サーバー、クライアント共にScalaで動いている
    

    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Response
    Server
    Call
    Response
    Airframe
    Scala.js
    29
    

    View Slide

30. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala.js Programming Tips
    ● Scala.js: Compiling Scala -> JavaScript
    ● Use Pure Scala Code
    ○ [NG] Java code
    ■ Dependent libraries also need to be pure Scala
    ● Airframe is carefully designed to minimize non-Scala dependencies
    ○ [NG] Annotation, runtime-reflection
    ○ [OK] Scala Macros (compile-time meta-programming)
    ● RPC
    ○ Asynchronous calls using Scala Future
    ○ Await(Future) is not supported
    ■ Because JS is a single-threaded model.
    ■ Need to properly chain Future operators with map, flatMap, rescue, etc.
    30
    Scala.jsでのプログラミングのコツ
    

    Scala.js
    

    View Slide

31. Copyright 1995-2020 Treasure Data. All rights reserved.
    Debugging Scala.js Code with airframe-log
    ● Showing logs at the developer console of the browser
    31
    airframe-logとブラウザのコンソールでScala.jsコードのデバッグ
    

    

    View Slide

32. Copyright 1995-2020 Treasure Data. All rights reserved.
    Rendering DOM with Scala.js
    ● DOM element:
    ○ ….
    ○ scalajs-dom library has functions necessary for generating DOMs
    ● Previous approach: Converting XML literal into DOM elements
    ○ monadic-html, Binding.scala
    ○ But XML literal will be deprecated in Dotty (Scala 3.0)
    32
    Scala.jsでのDOMレンダリング手法には様々な手法がある
    

    

    View Slide

33. Copyright 1995-2020 Treasure Data. All rights reserved.
    Rendering DOM
    ● Current Best Practice
    ○ Using Scala functions for
    building DOM elements
    ● Option 1: Separation between
    DOM elements and attributes
    ○ ScalaTags, Slinky
    ○ Too many parentheses!
    ● Option 2: No separation
    between DOM elements and
    attributes
    ○ scalajs-react, Airframe Rx
    ○ More Scala-friendly syntax
    33
    DOMのelement, attributeを区別しない方が関数型に適している
    

    ScalaTags
    Airframe Rx
    

    View Slide

34. Copyright 1995-2020 Treasure Data. All rights reserved.
    Complex DOM Rendering
    ● Airframe Rx
    ● RxElement interface
    ○ def render(): RxElement
    ○ Support nesting with apply(...)
    ■ Embedding Scala functions
    and collections
    34
    airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする
    

    

    View Slide

35. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe Rx
    ● Rx.variable
    ○ Re-render DOM if the variable is updated
    ● Gathering best practices from existing libraries
    ○ Type-safe DOM elements and attributes
    ■ ScalaTags, scalajs-react
    ○ Reactive streaming
    ■ Monix
    ■ Scala Collection
    ■ Scala.rx
    ○ Cancelable Interface
    ■ Cleanup DOM and event handlers when
    updating DOM
    ■ Monix, monadic-html
    35
    既存技術の良いところ取りをしたAirframe Rx
    

    

    View Slide

36. Copyright 1995-2020 Treasure Data. All rights reserved.
    Reactive Stream Interface: Rx[X]
    ● Rx[X]
    ○ ReactiveX stream implementation
    ○ e.g., Monix, Scala.Rx
    ● Reactive DOM Rendering
    ○ Rewriting DOM interactively upon upstream data
    change
    ● Reactive Operators
    ○ map, flatMap, filter, join, etc.
    ○ Almost the same with Scala collection APIs
    36
    Reactive stream interface Rx[X]でインタラクティブなDOMを作成
    

    

    View Slide

37. Copyright 1995-2020 Treasure Data. All rights reserved.
    gRPC Streaming with Airframe Rx
    ● Interactive communication between server and clients with reactive stream
    interface: Rx[A]
    37
    gRPC StreamingもRxインターフェースでサポートできる
    

    

    View Slide

38. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe Enables Scala-Oriented Development
    38
    RPC Interface
    Scala.js Client RPC Web Server
    Generates
    Scala.js
    Web Application
    API Documentation
    RPC Call RPC Call
    Implements
    Airframe
    AirframeでScala中心のアプリケーション開発を可能に
    

    ● Only need to learn how to write Scala for kick-starting client/server application
    development
    

    View Slide

39. Copyright 1995-2020 Treasure Data. All rights reserved.
    Summary
    ● Scala is a powerful language that can consolidate existing technologies
    ● Minimizing the learning cost of application development with Scala-First
    approach
    ○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose
    39
    まとめ：既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe
    

    Airframe
    Scala.js Web Browsers
    JSON
    

    View Slide