Scalaライブラリを作る前に知っておきたいメンテナンスのこと

in ScalaMatsuri 2019

Transcript

1. Things you'd better know
   before creating your Scala
   library
   Mitsuhiro Shibuya(@m4buya) Opt, Inc.
   for ScalaMatsuri 2019
   Scalaライブラリを作る前に知っておきたい

   メンテナンスのこと

   

   View Slide

2. About me
   Mitsuhiro Shibuya @m4buya
   ● The Leader of SRE team in Opt, an advertising agency
   ● Loves Ruby, Scala and cloud infrastructure
   ● Thinks “Scala is a typed Ruby”
   ● https://github.com/mshibuya
   ● RailsAdmin/CarrierWave committer
   

   View Slide

3. Agenda ● About this talk
   ● Supporting multiple JVM
   and Scala versions
   ● Interoperability with Java
   ● Keeping compatibility
   across releases
   ● Versioning
   ● Wrapping up
   

   View Slide

4. About this talk
   

   View Slide

5. Libraries
   

   View Slide

6. Assumptions
   ● We value the act of creating libraries
   ● Libraries are reusable solutions for specific problems
   ● Writing reusable code takes some cost
   ● Libraries used by many people are worth the cost
   ● If a library cannot gain enough users, its creation effort will
   not be rewarded (creating one-off solution is cheaper)
   ライブラリとは再利用可能な解決策であって、 多くの人に使っても
   らえないとそのためにかかったコストがペイしない

   

   View Slide

7. Libraries which aren’t used so much...
   ● are not popular enough
   ● do not solve the problem which we expect it to
   solve
   ● are hard to use
   ● have support for limited runtimes
   ● break easiliy
   使われづらいライブラリの性質のうち「特定の環境でしか動かない」
   「すぐ壊れる」について何ができるかを考えていきます

   In this talk we’ll focus on these two topics and
   explore what maintenance effort we can do,
   so we are not afraid of what happens after
   releasing
   

   View Slide

8. bigquery-fake-client
   The content of this talk is largely based on my experience in
   development and publication of this OSS project.
   Please take a look if you’re interested, and give us a star if you
   like!
   この発表の多くの部分はbigquery-fake-clientを作った時の経験が
   ベースになっています。Starお待ちしてます★

   https://github.com/opt-tech/bigquery-fake-client
   

   View Slide

9. Supporting multiple JVM and
   Scala versions
   複数JVM/Scalaバージョンへの対応

   

   View Slide

10. Scala applications depend on
    They are evolving with time, so people are using various
    versions of them at a time. In a context of library, limiting
    support to a specific version will reduce chance of the library
    to be chosen.
    Scalaアプリが依存するScalaコンパイラとJVMには複数のバージョ
    ンがあり、ライブラリはなるべく多くに対応したい

    

    View Slide

11. Supporting multiple JVM versions
    JVM versions are expected to be backward-compatible, so if
    you are to support multiple versions of JVMs, using the
    minimum version of JVM to support is enough.
    ● e.g. If you decide that your library supports JDK 8 and 11,
    use JDK 8 for compiling and building artifacts.
    JVMは後方互換なので、複数のバージョンに対応したい時は一番
    前のバージョンでコンパイル・ビルドすればOK

    

    View Slide

12. Supporting multiple Scala versions
    Scala versions are expected to be backward-compatible only
    within the same first two digits of version (e.g. if you compile
    your Scala library with Scala 2.12.3, resulting artifact only
    works with Scala Runtime of 2.12.x). You have to build
    artifacts for each Scala version you want to support.
    sbt has a built-in feature for cross building:
    Scalaは末尾を除くバージョンが同一のものでしか互換でないので、
    複数に対応するにはその分だけビルドを行う必要がある

    // build.sbt
    val scala211 = "2.11.12"
    scalaVersion := scala211
    crossScalaVersions := Seq(scala211, "2.12.8")
    # in sbt shell
    sbt> +compile => kicks code compilation for all scala versions
    

    View Slide

13. Make sure to test with every version you want to support!
    Softwares break easily, especially for things you are not
    focusing on.
    Your CI tool should have a cross-building feature against
    different version of dependencies. For TravisCI:
    意図せず壊れないよう、対応したい全てのバージョンについてちゃ
    んとテストしよう！TravisCIではこんな感じにできる

    language: scala
    scala:
    - 2.11.12
    - 2.12.8
    jdk:
    - openjdk8
    - openjdk11
    script:
    - sbt ++$TRAVIS_SCALA_VERSION test
    

    View Slide

14. Interoperability with Java
    Javaとの相互運用性

    

    View Slide

15. Having interoperability also promotes use
    ● What we discuss here is making a Scala library which is
    ready to be called by Java applications/libraries.
    ● But it can be tricky, because Scala has language features
    which Java doesn’t have
    ● Some of Scala features are specially encoded to JVM
    bytecode, so you must aware of their encoding scheme
    ● Some of these schemes might look unnatural to Java
    users
    ○ e.g. To use Scala singleton object Person, Java code
    looks like Person$.MODULE$
    ScalaライブラリをJavaから使えるように作れれば利用の幅が広が
    るが、特別な呼び出し方が必要になることも

    

    View Slide

16. To make Scala code Java-friendly
    ● Avoid use of default parameters
    ○ Builder pattern can achieve almost the same effect for
    constructors
    ● Avoid use of singleton objects (companion objects are OK)
    ● Avoid exposing Scala-specific types, like collections
    ○ scala-java8-compat will help make Java-ish API
    ● Writing wrapper-class in Java might be an option
    ● It is important to actually write Java code to ensure your
    Scala API has the expected interface
    Javaから呼びやすいScalaコードのためには。Scala特有の機能を使
    わないようにする。Javaでwrapperを書く手も

    new Foo.Builder().setBar("abc").setBaz(1).build();
    

    View Slide

17. Running Java test code inside your Scala library
    By including Java test code in your library using junit-interface,
    you can make sure that your library works just the same as
    being called from Scala.
    Scalaライブラリを呼び出すテストコードをJavaで書こう。
    junit-interfaceでsbtから簡単に実行できる

    https://github.com/sbt/junit-interface
    // build.sbt
    libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test"
    package jp.ne.opt.bigqueryfake;
    import org.junit.Test;
    public class FakeBigQueryTest {
    @Test
    public void testInstantiation() {
    new FakeBigQuery(FakeBigQueryOptions.newBuilder().build());
    }
    }
    

    View Slide

18. Keeping compatibility across
    releases
    リリースバージョン間での互換性の維持

    

    View Slide

19. Compatibility matters
    Who wants to use a library which changes APIs so often that
    each upgrade needs fixing so many things in their
    application?
    Compatibility of Scala libraries boils down to two part:
    互換性は大事。Scalaライブラリの互換性はソース互換性とバイナリ
    互換性の２つに分けられる

    Source compatibility Binary compatibility
    Application /
    Library
    Source Code
    Depend on
    Library
    1.0 Library
    2.0
    Depend on
    Library
    1.0 Library
    2.0
    Application /
    Library
    Binary
    

    View Slide

20. Source Compatibility
    If upgrading a library causes compile error, they are source
    incompatible.
    ● Many kind of changes falls into this category, like
    changing a class or method name
    ● Pretty straightforward
    ● There is a kind of change that breaks source compatibility
    but does not break any binary compatibility
    バージョンアップしてコンパイルが通らなければソース非互換。バイ
    ナリ互換だがソース非互換というケースもある

    

    View Slide

21. Binary Compatibility
    When a newly compiled library binary fails to work with
    existing binary, they’re binary incompatible.
    ● Easy to overlook
    ● Leads to the situation called ‘dependency hell’
    間接依存するライブラリにバイナリ互換性がないと、直接依存する
    ライブラリをアップデートできなくなってしまう

    Images from: https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html
    

    View Slide

22. When does binary incompatibility arise?
    ● Adding method parameters with default values
    ● Inferred type being changed on modifying implementation
    ● And more...
    バイナリ互換性が壊れるパターン：デフォルト値付きで引数追加、
    型注釈がないために実装の変更によって型が変わる等

    def method(str: String): Unit
    // changed to ↓
    def method(str: String, bool: Boolean = true): Unit
    def method() = Seq.empty
    // changed to ↓
    def method() = Vector.empty
    

    View Slide

23. Use MiMa to ensure binary compatibility
    Migration Manager(MiMa) is the standard and easiest way for
    checking binary compatibility.
    Migration Manager(MiMa)を利用することで手軽にバイナリ非互換性
    をチェックできる。CIタスクにも追加しよう

    // plugins.sbt
    addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")
    // build.sbt
    mimaPreviousArtifacts := Set("jp.ne.opt" % s"bigquery-fake-client_2.12" % "0.1.0")
    # in sbt shell
    sbt> mimaReportBinaryIssues
    …
    [error] * method
    setConnection(java.sql.Connection)jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder
    in class jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder does not have a
    correspondent in current version
    ...
    https://github.com/lightbend/migration-manager
    

    View Slide

24. Versioning
    バージョン番号の扱い方

    

    View Slide

25. Nevertheless, incompatibility is unavoidable
    If you keep maintaining libraries, someday you’ll be forced to
    make incompatible changes.
    ● API is poorly designed and can’t stand future evolution
    ● Design is forced to be changed by external events, like a
    breaking change in dependent library
    This is where versioning comes. It’s a methodology for
    handling incompatibility softly.
    There are two frequently-used conventions in the Scala world.
    とはいえ非互換な変更を余儀なくされる時は来る。バージョン番号
    をうまく使えば非互換性をよりソフトに扱えるようになる

    

    View Slide

26. Semantic Versioning (SemVer)
    セマンティック・バージョニング(SemVer)でのバージョンの上げ方。

    1.4.3
    major:
    bumped when
    incompatible
    API changes
    are made
    minor:
    bumped when
    backward-
    compatible
    functionalities
    are added
    patch:
    bumped when
    backward-
    compatible
    bug fix are
    made
    Widely used convention for many languages.
    

    View Slide

27. Java/Scala Style Versioning
    Java/Scalaスタイルのバージョンの上げ方。

    1.4.3
    epoch:
    bumped on
    epoch-making
    changes
    major:
    bumped when
    incompatible
    API changes
    are made
    minor:
    bumped when
    backward-
    compatible
    functionalities
    are added
    Originated from Java, this style is used by many Java and
    Scala projects, including Scala compiler/runtime.
    

    View Slide

28. Then, which to use?
    ● SemVer is the primary choice for new libraries
    ● Whichever you choose, it is important to state your
    libraries’ versioning policy explicitly
    ● Versioning policies of popular libraries:
    ○ Play
    ○ Cats
    ○ ScalikeJDBC
    これから作るならSemVerが第一選択。どちらを選ぶにせよ、バー
    ジョニングポリシーとして明示しておくことが大事

    Since Play 2.0.0, Play is versioned as epoch.major.minor. Play currently releases a
    new major version about every year. Major versions can break APIs, but we try to
    make sure most existing code will compile with deprecation.
    After 1.0.0 release, we decided to use MAJOR.MINOR.PATCH Semantic
    Versioning 2.0.0 going forward, which is different from the EPOCH.MAJOR.MINOR
    scheme common among Java and Scala libraries (including the Scala lang).
    ● major: Change only when the supported Scala major version changes:
    ○ Version 1 supports Scala 2.9 & 2.10
    ○ Version 2 supports Scala 2.10, Scala 2.11 and Scala 2.12
    ○ Version 3 supports Scala 2.10, Scala 2.11 and Scala 2.12
    ● minor: Change to indicate functionality and API compatibility changes, the
    same minor version must provide the same functionality and APIs
    ● fix: For releasing smaller improvements, bug fixes and new features
    

    View Slide

29. Persistent Versioning
    For the libraries which are depended heavily by other libraries,
    it might be worth to consider using a convention called
    Persistent Versioning when making incompatible upgrade.
    他のライブラリから依存されているようなライブラリに非互換な変更
    を加える際はPersistent Versioningを検討しよう

    “com.example.library” %
    “library” % “1.0.0”
    package com.example.library
    “com.example.library” %
    “library2” % “2.0.0”
    package com.example.library2
    http://eed3si9n.com/persistent-versioning
    

    View Slide

30. Let users know about changes you made!
    It’s always a good practice to have a brief summary of
    changes applied to the library, especially for breaking one.
    There’s a good guideline for it.
    ユーザーにどんな変更（とりわけ破壊的な）があったか伝わるよう変
    更履歴を書こう。keepachangelogのスタイルがお勧め

    ## [0.0.7] - 2015-02-16
    ### Added
    - Link, and make it obvious that date format is ISO 8601.
    ### Changed
    - Clarified the section on "Is there a standard change log format?".
    ### Fixed
    - Fix Markdown links to tag comparison URL with footnote-style links.
    https://keepachangelog.com/
    

    View Slide

31. Wrapping up
    まとめ

    

    View Slide

32. ライブラリは使われてはじめてペイする。バイナリ互換性に注意。
    バージョニングで非互換性をソフトに

    ● Library is a reusable solution, it is important to be used
    widely
    ● Supporting multiple JVM and Scala versions and keeping
    interoperability with Java increases the chance of being
    used
    ● Keeping compatibility is important. In particular, binary
    compatibility requires special attention
    ● Deal with incompatibility using versioning scheme
    

    View Slide

33. You’ve now learned how to
    maintain a library stably. Let’s roll
    your own and help other
    developers!
    これで安定してライブラリをメンテする準備はととのったはず。あな
    たもライブラリを公開して他の人を助けよう！

    

    View Slide

34. オプトではScalaエンジニアをはじめエンジニアを積極採用中です！
    ご興味があればブースか弊社社員までお声がけを!！

    Our company Opt is eagerly looking for…
    ● Scala Engineer
    ● Data Science Engineer
    ● Engineering Manager
    Come visit us at the booth if you’re interested!!
    PR
    

    View Slide

35. Questions?
    

    View Slide