$30 off During Our Annual Pro Sale. View Details »

Scala For Everything: From Frontend to Backend Applications - ScalaMatsuri 2020

Scala For Everything: From Frontend to Backend Applications - ScalaMatsuri 2020

ScalaMatsuri 2020 presentation slides.
- Japanese Title: Scala無双: フロントエンドもバックエンドも全てScalaでなぎ倒せ

Scala is a powerful language; You can build frontend applications with Scala.js, and efficient backend application servers for JVM. In this session, we will learn how to build everything with Scala by using Airframe OSS framework.

Airframe is a library designed for maximizing the advantages of Scala as a hybrid of object-oriented and functional programming language. In this session, we will learn how to use Airframe to build REST APIs and RPC services (Finagle, gRPC backend), and how to create frontend applications in Scala.js that interact with the servers using functional interfaces for dynamically updating web pages.

Taro L. Saito

October 16, 2020
Tweet

More Decks by Taro L. Saito

Other Decks in Programming

Transcript

  1. 1
    Taro L. Saito
    Treasure Data
    October 17th, 2020
    Scala Matsuri 2020
    Scala For Everything
    From Frontend to Backend Applications
    Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ

    View Slide

  2. Copyright 1995-2020 Treasure Data. All rights reserved.
    About Me: Taro L. Saito
    2
    ● Treasure Data
    ○ Principal Software Engineer
    ○ Living in US for 5 years
    ● Ph.D. of Computer Science
    ○ Database systems, genome science.
    ● OSS:
    ○ Airframe, snappy-java (used in
    Parquet, Spark, etc.), sbt-sonatype,
    sbt-pack, etc.
    ● Book:
    自己紹介


    View Slide

  3. Copyright 1995-2020 Treasure Data. All rights reserved.
    Today’s Topic
    ● There are a lot of libraries and frameworks for backend and frontend applications
    ● Airframe
    ○ An OSS library collection that enables Scala-oriented development
    ● Goal: Forget everything other than Scala
    3
    Scala無双:Scala以外を全て駆逐する

    Airframe
    Scala.js Web Browsers
    JSON

    View Slide

  4. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe: Core Scala Modules of Treasure Data
    ● Airframe (GitHub: wvlet/airframe)
    ○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019)
    ■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc.
    ○ Airframe RPC, Airframe Rx (Scala Matsuri 2020)
    ○ AirSpec
    ■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021)
    4
    Knowledge
    Experiences
    Design Decisions
    Products
    24/7 Services
    Business Values
    Programming OSS Outcome
    Airframe
    AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群

    View Slide

  5. Copyright 1995-2020 Treasure Data. All rights reserved.
    Treasure Data: A Ready-to-Use Cloud Data Platform
    5
    Logs
    Device
    Data
    Batch
    Data
    PlazmaDB
    Table Schema
    Data Collection Cloud Storage Distributed Data Processing
    Jobs
    Job Management
    SQL Editor
    Scheduler
    Workflows
    Machine
    Learning
    Treasure Data OSS
    Third Party OSS
    Data
    トレジャーデータはすぐに使えるクラウドデータプラットフォーム

    View Slide

  6. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala.js Application: Presto Service Monitor
    ● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions
    6
    Scala.jsで開発されたアプリケーションの例

    View Slide

  7. Copyright 1995-2020 Treasure Data. All rights reserved.
    Server APIs and Library for Spark
    Airframe DI
    DataFrame MPC1
    airframe-codec
    airframe-msgpack
    Plazma Public API
    airframe-http
    airframe-finagle
    Airframe DI
    Airframe RPC
    airframe-fluentd
    Master Worker
    Design
    SparkContext
    TDSparkContext TDSparkService
    MPC1 Reader/Writer IO Manager
    Airframe DI
    airframe-http
    airframe-config
    airframe-launcher
    airframe-jmx
    airframe-metrics
    airframe-control
    airframe-metrics
    td-spark.jar
    airframe-log
    airframe-log
    airframe-codec
    airframe-json
    Airframe
    7
    Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている

    ● MPC1: MessagePack Columnar Format
    ● See also td-spark internals (Spark Meetup Tokyo
    2020)

    View Slide

  8. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala is Versatile
    ● Backend
    ○ Can leverage JVM ecosystem (e.g.,
    netty, gRPC, etc.)
    ● Frontend
    ○ Scala.js:
    ■ 1.0.0 is released in 2020
    ● The current version is 1.3.0
    ■ Compile Scala code into
    Javascripts
    ○ Out of scope of this talk
    ■ iOS/iPhone native applications
    ● How can we bridge server (Scala JVM)
    and client (Scala.js) applications?
    Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ?

    Scala.js
    Web Browsers
    Serve-Client
    Interaction
    8

    View Slide

  9. Copyright 1995-2020 Treasure Data. All rights reserved.
    Bridging Scala and Scala.js with RPC
    Program Function
    Call
    Return
    Local Function Call
    (ideal)
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる

    Remote Procedure Call
    (reality)
    Network
    Client
    Call
    Return
    Server
    Call
    Return
    9
    Scala.js
    Scala.js

    View Slide

  10. Copyright 1995-2020 Treasure Data. All rights reserved.
    Existing Approaches for Implementing RPC
    ● REST
    ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
    ○ REST web frameworks for Scala
    ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
    ● gRPC
    ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
    ○ Generate server and client code stubs from proto files
    ○ Scala wrappers:
    ■ ScalaPB, muScala, akka-grpc, etc.
    RPCを実装するには、RESTやgRPCなどが使われている

    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    10

    View Slide

  11. Copyright 1995-2020 Treasure Data. All rights reserved.
    API Design War
    https://cloud.google.com/blog/products/api-management/
    understanding-grpc-openapi-and-rest-and-when-to-use-th
    em
    11
    ● Which API design should we choose?
    ● REST API
    ○ Designing good APIs over HTTP
    requires experiences and significant
    amount of time
    ● gRPC
    ○ Requires special software like code
    generators for using the ecosystem
    around ProtocolBuffers
    ● And also, there are too many choices for
    web frameworks
    しかし、APIデザイン、フレームワーク選びでの争いは付きない

    View Slide

  12. Copyright 1995-2020 Treasure Data. All rights reserved.
    Why Not Using Scala for RPC Interface?
    ● Scala is a functional and statically-typed object-oriented programming language
    ○ Scala functions = RPC methods
    ○ Scala objects (statically typed) = RPC request/response types
    ● Scala is a perfect fit for RPC!
    12
    関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している

    View Slide

  13. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala-First Approach: Airframe RPC
    RPC Interface
    Scala.js Client RPC Web Server
    Generates
    Scala.js
    Web Application
    API Documentation
    13
    Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法

    RPC Call RPC Call
    Implements
    ● Forget about REST or ProtocolBuffers!
    ● Use Scala as a unified interface between server and clients Airframe

    View Slide

  14. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe RPC Backend Is Pluggable
    RPC Interface RPC Web Server
    Generates API Documentation
    14
    Airframe RPCのServer, Clientは差し替え可能

    Airframe
    RPC Call RPC Call
    Implements
    ● Backend server
    ○ Finagle (HTTP/1), gRPC (HTTP/2)
    ● Backend client
    ○ Finagle, gRPC, okhttp, Java URLConnection client etc.
    RPC Clients

    View Slide

  15. Copyright 1995-2020 Treasure Data. All rights reserved.
    Building Blocks of RPC Frameworks
    ● 1. Message serializer and deserializer
    ● 2. Network data format
    ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
    ● 3. RPC interface language
    ○ REST API, Protobuf, or Scala
    ● 4. HTTP client and server implementation
    ○ Code generator
    15
    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Return
    Server
    Call
    Return
    RPCフレームワークを作るのに必要な構成要素

    View Slide

  16. Copyright 1995-2020 Treasure Data. All rights reserved.
    MessagePack-Based Object Serialization
    ● MessagePack
    ○ A compact binary format compatible
    with JSON
    Object Object
    Pack Unpack
    Pack
    Unpack
    Server Side
    Client Side
    16
    Scala.js
    JSON
    Airframe
    MessagePackを使ってクライアントとサーバー間通信を行う

    View Slide

  17. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-codec: Pack/Unpack Interface
    ● Pack
    ○ Converting Data to MessagePack
    ● Unpack
    ○ Converting MessagePack to Data
    Input Output
    Pack Unpack
    Pack
    Unpack
    17
    Airframe
    MessageCodec MessageCodec
    MessagePackを通したデータ変換インターフェースをScalaに適用

    View Slide

  18. Copyright 1995-2020 Treasure Data. All rights reserved.
    Pre-defined Codecs in airframe-codec
    ● Primitive Codecs
    ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
    ○ FloatCodec, DoubleCodec
    ○ StringCodec
    ○ BooleanCodec
    ○ TimeStampCodec
    ● Collection Codec
    ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
    ○ OptionCodec
    ● Java-specific Codec
    ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
    ○ JDBCResultSetCodec
    ● etc.
    18
    Scalaで必要なほぼ全てのデータ型へのマッピングをサポート

    Airframe

    View Slide

  19. Copyright 1995-2020 Treasure Data. All rights reserved.
    Serializing Complex Objects
    Pack
    Unpack
    IntCodec
    StringCodec
    DoubleCodec
    MessageCodec.of[A]
    19
    オブジェクトの型に合わせてCodecを合成できる

    Serialize
    Deserialize
    JSON

    View Slide

  20. Copyright 1995-2020 Treasure Data. All rights reserved.
    Defining RPC Interface with Scala
    ● Add @RPC annotation to a trait (= RPC interface)
    ● Define model classes (= data structures) using case classes
    ● Use the same interface and model classes between servers and clients
    RPC InterfaceをScalaで記述する

    20

    View Slide

  21. Copyright 1995-2020 Treasure Data. All rights reserved.
    Serializing Function Call Request
    ● Function call arguments
    ○ A sequence of argument name -> value, ….
    ● Example:
    ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
    ○ Serialize this Map into MessagePack with airframe-codec
    ● HTTP Mapping
    ○ HTTP method: POST
    ○ Path: /(package name)/hello
    ■ e.g., /hello.api.v1/hello
    ○ Content body: serialized function call argument data
    21
    関数呼び出しの引数もシリアライズする

    View Slide

  22. Copyright 1995-2020 Treasure Data. All rights reserved.
    Implementing RPC Servers
    ● Extending the RPC interface trait
    Serverの実装はRPC traitの継承

    22

    View Slide

  23. Copyright 1995-2020 Treasure Data. All rights reserved.
    ● Finagle: Twitter’s HTTP server written in Scala
    ○ HTTP/1 based. Accessible from web browsers running Scala.js code
    airframe-http: Starting A Finagle Backend RPC Server
    23
    Airframe
    Scala.js
    airframe-rx-html
    airframe-http: FinagleバックエンドのHTTPサーバーを起動する

    airframe-http-finagle

    View Slide

  24. Copyright 1995-2020 Treasure Data. All rights reserved.
    airframe-http: gRPC Backend
    ● Suited to internal microservices
    ● HTTP/2 based protocol
    ○ Multiplexing multiple RPC
    requests within a single
    connection
    ● No .proto file is required
    24
    Airframe
    airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない

    View Slide

  25. Copyright 1995-2020 Treasure Data. All rights reserved.
    Advanced Topic: Extending gRPC
    ● gRPC is data-format agonistic
    framework
    ○ To support Scala interface and
    MessagePack, we have two
    extension points in grpc-java.
    ● MethodDescriptor
    ○ Define gRPC endpoints
    corresponding to Scala
    functions
    ○ Register marshallers
    ● Request/ResponseMarshaller
    ○ Define how to encode/decode
    RPC request/response data
    25
    Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている

    View Slide

  26. Copyright 1995-2020 Treasure Data. All rights reserved.
    RPC Performance Comparison (Greeter Service)
    ● Airframe RPC
    ○ serde: MessagePack (airframe-codec) <->
    Scala case classes
    ○ Finagle (HTTP1) or gRPC (HTTP2)
    ● ScalaPB
    ○ serde: Protobuf <-> Scala case classes
    ○ gRPC (HTTP2)
    ● grpc-java
    ○ serde: Protobuf <-> Java classes
    ○ gRPC (HTTP2)
    ● Notes
    ○ Using Finagle with HTTP2 had almost no
    benefit over HTTP1
    ■ Multiplexing RPC requests over
    HTTP2 is the key performance factor
    ○ Overhead of ScalaPB
    ■ Scala Future (10% overhead)
    ■ Mapping Protobuf to Scala case
    classes (10% overhead)
    26
    gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない

    HTTP/1 HTTP/2 (gRPC)

    View Slide

  27. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating RPC Clients
    ● sbt-airframe plugin
    ○ Read RPC interface classes
    ○ Generate HTTP client code for accessing the RPC server
    ○ OpenAPI schema generation support
    sbt-airframe
    Code
    Generation
    RPC Client
    Scala.js
    27
    Scala.js Client
    HTTP/gRPC Client
    Open API Spec
    Cross-Language
    RPC Client
    RPCクライアントを生成するsbt-airframeプラグイン

    View Slide

  28. Copyright 1995-2020 Treasure Data. All rights reserved.
    sbt-airframe: Generating Open API Schema
    ● RPC Interface -> Open API schema YAML file
    ● Generating RPC Clients & Swagger Documentation
    28
    ScalaのインターフェースからOpenAPIのYAMLを生成する

    RPC Interface
    API Documentation
    Open API Spec
    (YAML)
    Cross-Language
    RPC Client
    Generate
    sbt-airframe
    API Documentation

    View Slide

  29. Copyright 1995-2020 Treasure Data. All rights reserved.
    Making An RPC Call
    RPCを実行する。サーバー、クライアント共にScalaで動いている

    Program Function
    Serialize
    Deserialize
    Request Data
    Deserialize
    Response Data
    Serialize
    Client
    Call
    Response
    Server
    Call
    Response
    Airframe
    Scala.js
    29

    View Slide

  30. Copyright 1995-2020 Treasure Data. All rights reserved.
    Scala.js Programming Tips
    ● Scala.js: Compiling Scala -> JavaScript
    ● Use Pure Scala Code
    ○ [NG] Java code
    ■ Dependent libraries also need to be pure Scala
    ● Airframe is carefully designed to minimize non-Scala dependencies
    ○ [NG] Annotation, runtime-reflection
    ○ [OK] Scala Macros (compile-time meta-programming)
    ● RPC
    ○ Asynchronous calls using Scala Future
    ○ Await(Future) is not supported
    ■ Because JS is a single-threaded model.
    ■ Need to properly chain Future operators with map, flatMap, rescue, etc.
    30
    Scala.jsでのプログラミングのコツ

    Scala.js

    View Slide

  31. Copyright 1995-2020 Treasure Data. All rights reserved.
    Debugging Scala.js Code with airframe-log
    ● Showing logs at the developer console of the browser
    31
    airframe-logとブラウザのコンソールでScala.jsコードのデバッグ

    View Slide

  32. Copyright 1995-2020 Treasure Data. All rights reserved.
    Rendering DOM with Scala.js
    ● DOM element:
    ○ ….
    ○ scalajs-dom library has functions necessary for generating DOMs
    ● Previous approach: Converting XML literal into DOM elements
    ○ monadic-html, Binding.scala
    ○ But XML literal will be deprecated in Dotty (Scala 3.0)
    32
    Scala.jsでのDOMレンダリング手法には様々な手法がある

    View Slide

  33. Copyright 1995-2020 Treasure Data. All rights reserved.
    Rendering DOM
    ● Current Best Practice
    ○ Using Scala functions for
    building DOM elements
    ● Option 1: Separation between
    DOM elements and attributes
    ○ ScalaTags, Slinky
    ○ Too many parentheses!
    ● Option 2: No separation
    between DOM elements and
    attributes
    ○ scalajs-react, Airframe Rx
    ○ More Scala-friendly syntax
    33
    DOMのelement, attributeを区別しない方が関数型に適している

    ScalaTags
    Airframe Rx

    View Slide

  34. Copyright 1995-2020 Treasure Data. All rights reserved.
    Complex DOM Rendering
    ● Airframe Rx
    ● RxElement interface
    ○ def render(): RxElement
    ○ Support nesting with apply(...)
    ■ Embedding Scala functions
    and collections
    34
    airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする

    View Slide

  35. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe Rx
    ● Rx.variable
    ○ Re-render DOM if the variable is updated
    ● Gathering best practices from existing libraries
    ○ Type-safe DOM elements and attributes
    ■ ScalaTags, scalajs-react
    ○ Reactive streaming
    ■ Monix
    ■ Scala Collection
    ■ Scala.rx
    ○ Cancelable Interface
    ■ Cleanup DOM and event handlers when
    updating DOM
    ■ Monix, monadic-html
    35
    既存技術の良いところ取りをしたAirframe Rx

    View Slide

  36. Copyright 1995-2020 Treasure Data. All rights reserved.
    Reactive Stream Interface: Rx[X]
    ● Rx[X]
    ○ ReactiveX stream implementation
    ○ e.g., Monix, Scala.Rx
    ● Reactive DOM Rendering
    ○ Rewriting DOM interactively upon upstream data
    change
    ● Reactive Operators
    ○ map, flatMap, filter, join, etc.
    ○ Almost the same with Scala collection APIs
    36
    Reactive stream interface Rx[X]でインタラクティブなDOMを作成

    View Slide

  37. Copyright 1995-2020 Treasure Data. All rights reserved.
    gRPC Streaming with Airframe Rx
    ● Interactive communication between server and clients with reactive stream
    interface: Rx[A]
    37
    gRPC StreamingもRxインターフェースでサポートできる

    View Slide

  38. Copyright 1995-2020 Treasure Data. All rights reserved.
    Airframe Enables Scala-Oriented Development
    38
    RPC Interface
    Scala.js Client RPC Web Server
    Generates
    Scala.js
    Web Application
    API Documentation
    RPC Call RPC Call
    Implements
    Airframe
    AirframeでScala中心のアプリケーション開発を可能に

    ● Only need to learn how to write Scala for kick-starting client/server application
    development

    View Slide

  39. Copyright 1995-2020 Treasure Data. All rights reserved.
    Summary
    ● Scala is a powerful language that can consolidate existing technologies
    ● Minimizing the learning cost of application development with Scala-First
    approach
    ○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose
    39
    まとめ:既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe

    Airframe
    Scala.js Web Browsers
    JSON

    View Slide