Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalaライブラリを作る前に知っておきたいメンテナンスのこと

 Scalaライブラリを作る前に知っておきたいメンテナンスのこと

in ScalaMatsuri 2019

Mitsuhiro Shibuya

June 28, 2019
Tweet

Other Decks in Programming

Transcript

  1. Things you'd better know
    before creating your Scala
    library
    Mitsuhiro Shibuya(@m4buya) Opt, Inc.
    for ScalaMatsuri 2019
    Scalaライブラリを作る前に知っておきたい

    メンテナンスのこと


    View full-size slide

  2. About me
    Mitsuhiro Shibuya @m4buya
    ● The Leader of SRE team in Opt, an advertising agency
    ● Loves Ruby, Scala and cloud infrastructure
    ● Thinks “Scala is a typed Ruby”
    ● https://github.com/mshibuya
    ● RailsAdmin/CarrierWave committer

    View full-size slide

  3. Agenda ● About this talk
    ● Supporting multiple JVM
    and Scala versions
    ● Interoperability with Java
    ● Keeping compatibility
    across releases
    ● Versioning
    ● Wrapping up

    View full-size slide

  4. About this talk

    View full-size slide

  5. Assumptions
    ● We value the act of creating libraries
    ● Libraries are reusable solutions for specific problems
    ● Writing reusable code takes some cost
    ● Libraries used by many people are worth the cost
    ● If a library cannot gain enough users, its creation effort will
    not be rewarded (creating one-off solution is cheaper)
    ライブラリとは再利用可能な解決策であって、 多くの人に使っても
    らえないとそのためにかかったコストがペイしない


    View full-size slide

  6. Libraries which aren’t used so much...
    ● are not popular enough
    ● do not solve the problem which we expect it to
    solve
    ● are hard to use
    ● have support for limited runtimes
    ● break easiliy
    使われづらいライブラリの性質のうち「特定の環境でしか動かない」
    「すぐ壊れる」について何ができるかを考えていきます

    In this talk we’ll focus on these two topics and
    explore what maintenance effort we can do,
    so we are not afraid of what happens after
    releasing

    View full-size slide

  7. bigquery-fake-client
    The content of this talk is largely based on my experience in
    development and publication of this OSS project.
    Please take a look if you’re interested, and give us a star if you
    like!
    この発表の多くの部分はbigquery-fake-clientを作った時の経験が
    ベースになっています。Starお待ちしてます★

    https://github.com/opt-tech/bigquery-fake-client

    View full-size slide

  8. Supporting multiple JVM and
    Scala versions
    複数JVM/Scalaバージョンへの対応


    View full-size slide

  9. Scala applications depend on
    They are evolving with time, so people are using various
    versions of them at a time. In a context of library, limiting
    support to a specific version will reduce chance of the library
    to be chosen.
    Scalaアプリが依存するScalaコンパイラとJVMには複数のバージョ
    ンがあり、ライブラリはなるべく多くに対応したい


    View full-size slide

  10. Supporting multiple JVM versions
    JVM versions are expected to be backward-compatible, so if
    you are to support multiple versions of JVMs, using the
    minimum version of JVM to support is enough.
    ● e.g. If you decide that your library supports JDK 8 and 11,
    use JDK 8 for compiling and building artifacts.
    JVMは後方互換なので、複数のバージョンに対応したい時は一番
    前のバージョンでコンパイル・ビルドすればOK


    View full-size slide

  11. Supporting multiple Scala versions
    Scala versions are expected to be backward-compatible only
    within the same first two digits of version (e.g. if you compile
    your Scala library with Scala 2.12.3, resulting artifact only
    works with Scala Runtime of 2.12.x). You have to build
    artifacts for each Scala version you want to support.
    sbt has a built-in feature for cross building:
    Scalaは末尾を除くバージョンが同一のものでしか互換でないので、
    複数に対応するにはその分だけビルドを行う必要がある

    // build.sbt
    val scala211 = "2.11.12"
    scalaVersion := scala211
    crossScalaVersions := Seq(scala211, "2.12.8")
    # in sbt shell
    sbt> +compile => kicks code compilation for all scala versions

    View full-size slide

  12. Make sure to test with every version you want to support!
    Softwares break easily, especially for things you are not
    focusing on.
    Your CI tool should have a cross-building feature against
    different version of dependencies. For TravisCI:
    意図せず壊れないよう、対応したい全てのバージョンについてちゃ
    んとテストしよう!TravisCIではこんな感じにできる

    language: scala
    scala:
    - 2.11.12
    - 2.12.8
    jdk:
    - openjdk8
    - openjdk11
    script:
    - sbt ++$TRAVIS_SCALA_VERSION test

    View full-size slide

  13. Interoperability with Java
    Javaとの相互運用性


    View full-size slide

  14. Having interoperability also promotes use
    ● What we discuss here is making a Scala library which is
    ready to be called by Java applications/libraries.
    ● But it can be tricky, because Scala has language features
    which Java doesn’t have
    ● Some of Scala features are specially encoded to JVM
    bytecode, so you must aware of their encoding scheme
    ● Some of these schemes might look unnatural to Java
    users
    ○ e.g. To use Scala singleton object Person, Java code
    looks like Person$.MODULE$
    ScalaライブラリをJavaから使えるように作れれば利用の幅が広が
    るが、特別な呼び出し方が必要になることも


    View full-size slide

  15. To make Scala code Java-friendly
    ● Avoid use of default parameters
    ○ Builder pattern can achieve almost the same effect for
    constructors
    ● Avoid use of singleton objects (companion objects are OK)
    ● Avoid exposing Scala-specific types, like collections
    ○ scala-java8-compat will help make Java-ish API
    ● Writing wrapper-class in Java might be an option
    ● It is important to actually write Java code to ensure your
    Scala API has the expected interface
    Javaから呼びやすいScalaコードのためには。Scala特有の機能を使
    わないようにする。Javaでwrapperを書く手も

    new Foo.Builder().setBar("abc").setBaz(1).build();

    View full-size slide

  16. Running Java test code inside your Scala library
    By including Java test code in your library using junit-interface,
    you can make sure that your library works just the same as
    being called from Scala.
    Scalaライブラリを呼び出すテストコードをJavaで書こう。
    junit-interfaceでsbtから簡単に実行できる

    https://github.com/sbt/junit-interface
    // build.sbt
    libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test"
    package jp.ne.opt.bigqueryfake;
    import org.junit.Test;
    public class FakeBigQueryTest {
    @Test
    public void testInstantiation() {
    new FakeBigQuery(FakeBigQueryOptions.newBuilder().build());
    }
    }

    View full-size slide

  17. Keeping compatibility across
    releases
    リリースバージョン間での互換性の維持


    View full-size slide

  18. Compatibility matters
    Who wants to use a library which changes APIs so often that
    each upgrade needs fixing so many things in their
    application?
    Compatibility of Scala libraries boils down to two part:
    互換性は大事。Scalaライブラリの互換性はソース互換性とバイナリ
    互換性の2つに分けられる

    Source compatibility Binary compatibility
    Application /
    Library
    Source Code
    Depend on
    Library
    1.0 Library
    2.0
    Depend on
    Library
    1.0 Library
    2.0
    Application /
    Library
    Binary

    View full-size slide

  19. Source Compatibility
    If upgrading a library causes compile error, they are source
    incompatible.
    ● Many kind of changes falls into this category, like
    changing a class or method name
    ● Pretty straightforward
    ● There is a kind of change that breaks source compatibility
    but does not break any binary compatibility
    バージョンアップしてコンパイルが通らなければソース非互換。バイ
    ナリ互換だがソース非互換というケースもある


    View full-size slide

  20. Binary Compatibility
    When a newly compiled library binary fails to work with
    existing binary, they’re binary incompatible.
    ● Easy to overlook
    ● Leads to the situation called ‘dependency hell’
    間接依存するライブラリにバイナリ互換性がないと、直接依存する
    ライブラリをアップデートできなくなってしまう

    Images from: https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html

    View full-size slide

  21. When does binary incompatibility arise?
    ● Adding method parameters with default values
    ● Inferred type being changed on modifying implementation
    ● And more...
    バイナリ互換性が壊れるパターン:デフォルト値付きで引数追加、
    型注釈がないために実装の変更によって型が変わる等

    def method(str: String): Unit
    // changed to ↓
    def method(str: String, bool: Boolean = true): Unit
    def method() = Seq.empty
    // changed to ↓
    def method() = Vector.empty

    View full-size slide

  22. Use MiMa to ensure binary compatibility
    Migration Manager(MiMa) is the standard and easiest way for
    checking binary compatibility.
    Migration Manager(MiMa)を利用することで手軽にバイナリ非互換性
    をチェックできる。CIタスクにも追加しよう

    // plugins.sbt
    addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")
    // build.sbt
    mimaPreviousArtifacts := Set("jp.ne.opt" % s"bigquery-fake-client_2.12" % "0.1.0")
    # in sbt shell
    sbt> mimaReportBinaryIssues

    [error] * method
    setConnection(java.sql.Connection)jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder
    in class jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder does not have a
    correspondent in current version
    ...
    https://github.com/lightbend/migration-manager

    View full-size slide

  23. Versioning
    バージョン番号の扱い方


    View full-size slide

  24. Nevertheless, incompatibility is unavoidable
    If you keep maintaining libraries, someday you’ll be forced to
    make incompatible changes.
    ● API is poorly designed and can’t stand future evolution
    ● Design is forced to be changed by external events, like a
    breaking change in dependent library
    This is where versioning comes. It’s a methodology for
    handling incompatibility softly.
    There are two frequently-used conventions in the Scala world.
    とはいえ非互換な変更を余儀なくされる時は来る。バージョン番号
    をうまく使えば非互換性をよりソフトに扱えるようになる


    View full-size slide

  25. Semantic Versioning (SemVer)
    セマンティック・バージョニング(SemVer)でのバージョンの上げ方。

    1.4.3
    major:
    bumped when
    incompatible
    API changes
    are made
    minor:
    bumped when
    backward-
    compatible
    functionalities
    are added
    patch:
    bumped when
    backward-
    compatible
    bug fix are
    made
    Widely used convention for many languages.

    View full-size slide

  26. Java/Scala Style Versioning
    Java/Scalaスタイルのバージョンの上げ方。

    1.4.3
    epoch:
    bumped on
    epoch-making
    changes
    major:
    bumped when
    incompatible
    API changes
    are made
    minor:
    bumped when
    backward-
    compatible
    functionalities
    are added
    Originated from Java, this style is used by many Java and
    Scala projects, including Scala compiler/runtime.

    View full-size slide

  27. Then, which to use?
    ● SemVer is the primary choice for new libraries
    ● Whichever you choose, it is important to state your
    libraries’ versioning policy explicitly
    ● Versioning policies of popular libraries:
    ○ Play
    ○ Cats
    ○ ScalikeJDBC
    これから作るならSemVerが第一選択。どちらを選ぶにせよ、バー
    ジョニングポリシーとして明示しておくことが大事

    Since Play 2.0.0, Play is versioned as epoch.major.minor. Play currently releases a
    new major version about every year. Major versions can break APIs, but we try to
    make sure most existing code will compile with deprecation.
    After 1.0.0 release, we decided to use MAJOR.MINOR.PATCH Semantic
    Versioning 2.0.0 going forward, which is different from the EPOCH.MAJOR.MINOR
    scheme common among Java and Scala libraries (including the Scala lang).
    ● major: Change only when the supported Scala major version changes:
    ○ Version 1 supports Scala 2.9 & 2.10
    ○ Version 2 supports Scala 2.10, Scala 2.11 and Scala 2.12
    ○ Version 3 supports Scala 2.10, Scala 2.11 and Scala 2.12
    ● minor: Change to indicate functionality and API compatibility changes, the
    same minor version must provide the same functionality and APIs
    ● fix: For releasing smaller improvements, bug fixes and new features

    View full-size slide

  28. Persistent Versioning
    For the libraries which are depended heavily by other libraries,
    it might be worth to consider using a convention called
    Persistent Versioning when making incompatible upgrade.
    他のライブラリから依存されているようなライブラリに非互換な変更
    を加える際はPersistent Versioningを検討しよう

    “com.example.library” %
    “library” % “1.0.0”
    package com.example.library
    “com.example.library” %
    “library2” % “2.0.0”
    package com.example.library2
    http://eed3si9n.com/persistent-versioning

    View full-size slide

  29. Let users know about changes you made!
    It’s always a good practice to have a brief summary of
    changes applied to the library, especially for breaking one.
    There’s a good guideline for it.
    ユーザーにどんな変更(とりわけ破壊的な)があったか伝わるよう変
    更履歴を書こう。keepachangelogのスタイルがお勧め

    ## [0.0.7] - 2015-02-16
    ### Added
    - Link, and make it obvious that date format is ISO 8601.
    ### Changed
    - Clarified the section on "Is there a standard change log format?".
    ### Fixed
    - Fix Markdown links to tag comparison URL with footnote-style links.
    https://keepachangelog.com/

    View full-size slide

  30. Wrapping up
    まとめ


    View full-size slide

  31. ライブラリは使われてはじめてペイする。バイナリ互換性に注意。
    バージョニングで非互換性をソフトに

    ● Library is a reusable solution, it is important to be used
    widely
    ● Supporting multiple JVM and Scala versions and keeping
    interoperability with Java increases the chance of being
    used
    ● Keeping compatibility is important. In particular, binary
    compatibility requires special attention
    ● Deal with incompatibility using versioning scheme

    View full-size slide

  32. You’ve now learned how to
    maintain a library stably. Let’s roll
    your own and help other
    developers!
    これで安定してライブラリをメンテする準備はととのったはず。あな
    たもライブラリを公開して他の人を助けよう!


    View full-size slide

  33. オプトではScalaエンジニアをはじめエンジニアを積極採用中です!
    ご興味があればブースか弊社社員までお声がけを!!

    Our company Opt is eagerly looking for…
    ● Scala Engineer
    ● Data Science Engineer
    ● Engineering Manager
    Come visit us at the booth if you’re interested!!
    PR

    View full-size slide