Scalaライブラリを作る前に知っておきたいメンテナンスのこと

in ScalaMatsuri 2019

Transcript

1. Things you'd better know before creating your Scala library Mitsuhiro Shibuya(@m4buya) Opt, Inc. for ScalaMatsuri 2019 Scalaライブラリを作る前に知っておきたい
 メンテナンスのこと

2. About me Mitsuhiro Shibuya @m4buya • The Leader of SRE team in Opt, an advertising agency • Loves Ruby, Scala and cloud infrastructure • Thinks “Scala is a typed Ruby” • https://github.com/mshibuya • RailsAdmin/CarrierWave committer
3. Agenda • About this talk • Supporting multiple JVM and Scala versions • Interoperability with Java • Keeping compatibility across releases • Versioning • Wrapping up
4. About this talk
5. Libraries
6. Assumptions • We value the act of creating libraries • Libraries are reusable solutions for specific problems • Writing reusable code takes some cost • Libraries used by many people are worth the cost • If a library cannot gain enough users, its creation effort will not be rewarded (creating one-off solution is cheaper) ライブラリとは再利用可能な解決策であって、 多くの人に使っても らえないとそのためにかかったコストがペイしない

7. Libraries which aren’t used so much... • are not popular enough • do not solve the problem which we expect it to solve • are hard to use • have support for limited runtimes • break easiliy 使われづらいライブラリの性質のうち「特定の環境でしか動かない」 「すぐ壊れる」について何ができるかを考えていきます
 In this talk we’ll focus on these two topics and explore what maintenance effort we can do, so we are not afraid of what happens after releasing
8. bigquery-fake-client The content of this talk is largely based on my experience in development and publication of this OSS project. Please take a look if you’re interested, and give us a star if you like! この発表の多くの部分はbigquery-fake-clientを作った時の経験が ベースになっています。Starお待ちしてます★
 https://github.com/opt-tech/bigquery-fake-client
9. Supporting multiple JVM and Scala versions 複数JVM/Scalaバージョンへの対応

10. Scala applications depend on They are evolving with time, so people are using various versions of them at a time. In a context of library, limiting support to a specific version will reduce chance of the library to be chosen. Scalaアプリが依存するScalaコンパイラとJVMには複数のバージョ ンがあり、ライブラリはなるべく多くに対応したい

11. Supporting multiple JVM versions JVM versions are expected to be backward-compatible, so if you are to support multiple versions of JVMs, using the minimum version of JVM to support is enough. • e.g. If you decide that your library supports JDK 8 and 11, use JDK 8 for compiling and building artifacts. JVMは後方互換なので、複数のバージョンに対応したい時は一番 前のバージョンでコンパイル・ビルドすればOK

12. Supporting multiple Scala versions Scala versions are expected to be backward-compatible only within the same first two digits of version (e.g. if you compile your Scala library with Scala 2.12.3, resulting artifact only works with Scala Runtime of 2.12.x). You have to build artifacts for each Scala version you want to support. sbt has a built-in feature for cross building: Scalaは末尾を除くバージョンが同一のものでしか互換でないので、 複数に対応するにはその分だけビルドを行う必要がある
 // build.sbt val scala211 = "2.11.12" scalaVersion := scala211 crossScalaVersions := Seq(scala211, "2.12.8") # in sbt shell sbt> +compile => kicks code compilation for all scala versions
13. Make sure to test with every version you want to support! Softwares break easily, especially for things you are not focusing on. Your CI tool should have a cross-building feature against different version of dependencies. For TravisCI: 意図せず壊れないよう、対応したい全てのバージョンについてちゃ んとテストしよう！TravisCIではこんな感じにできる
 language: scala scala: - 2.11.12 - 2.12.8 jdk: - openjdk8 - openjdk11 script: - sbt ++$TRAVIS_SCALA_VERSION test
14. Interoperability with Java Javaとの相互運用性

15. Having interoperability also promotes use • What we discuss here is making a Scala library which is ready to be called by Java applications/libraries. • But it can be tricky, because Scala has language features which Java doesn’t have • Some of Scala features are specially encoded to JVM bytecode, so you must aware of their encoding scheme • Some of these schemes might look unnatural to Java users ◦ e.g. To use Scala singleton object Person, Java code looks like Person$.MODULE$ ScalaライブラリをJavaから使えるように作れれば利用の幅が広が るが、特別な呼び出し方が必要になることも

16. To make Scala code Java-friendly • Avoid use of default parameters ◦ Builder pattern can achieve almost the same effect for constructors • Avoid use of singleton objects (companion objects are OK) • Avoid exposing Scala-specific types, like collections ◦ scala-java8-compat will help make Java-ish API • Writing wrapper-class in Java might be an option • It is important to actually write Java code to ensure your Scala API has the expected interface Javaから呼びやすいScalaコードのためには。Scala特有の機能を使 わないようにする。Javaでwrapperを書く手も
 new Foo.Builder().setBar("abc").setBaz(1).build();
17. Running Java test code inside your Scala library By including Java test code in your library using junit-interface, you can make sure that your library works just the same as being called from Scala. Scalaライブラリを呼び出すテストコードをJavaで書こう。 junit-interfaceでsbtから簡単に実行できる
 https://github.com/sbt/junit-interface // build.sbt libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test" package jp.ne.opt.bigqueryfake; import org.junit.Test; public class FakeBigQueryTest { @Test public void testInstantiation() { new FakeBigQuery(FakeBigQueryOptions.newBuilder().build()); } }
18. Keeping compatibility across releases リリースバージョン間での互換性の維持

19. Compatibility matters Who wants to use a library which changes APIs so often that each upgrade needs fixing so many things in their application? Compatibility of Scala libraries boils down to two part: 互換性は大事。Scalaライブラリの互換性はソース互換性とバイナリ 互換性の２つに分けられる
 Source compatibility Binary compatibility Application / Library Source Code Depend on Library 1.0 Library 2.0 Depend on Library 1.0 Library 2.0 Application / Library Binary
20. Source Compatibility If upgrading a library causes compile error, they are source incompatible. • Many kind of changes falls into this category, like changing a class or method name • Pretty straightforward • There is a kind of change that breaks source compatibility but does not break any binary compatibility バージョンアップしてコンパイルが通らなければソース非互換。バイ ナリ互換だがソース非互換というケースもある

21. Binary Compatibility When a newly compiled library binary fails to work with existing binary, they’re binary incompatible. • Easy to overlook • Leads to the situation called ‘dependency hell’ 間接依存するライブラリにバイナリ互換性がないと、直接依存する ライブラリをアップデートできなくなってしまう
 Images from: https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html
22. When does binary incompatibility arise? • Adding method parameters with default values • Inferred type being changed on modifying implementation • And more... バイナリ互換性が壊れるパターン：デフォルト値付きで引数追加、 型注釈がないために実装の変更によって型が変わる等
 def method(str: String): Unit // changed to ↓ def method(str: String, bool: Boolean = true): Unit def method() = Seq.empty // changed to ↓ def method() = Vector.empty
23. Use MiMa to ensure binary compatibility Migration Manager(MiMa) is the standard and easiest way for checking binary compatibility. Migration Manager(MiMa)を利用することで手軽にバイナリ非互換性 をチェックできる。CIタスクにも追加しよう
 // plugins.sbt addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0") // build.sbt mimaPreviousArtifacts := Set("jp.ne.opt" % s"bigquery-fake-client_2.12" % "0.1.0") # in sbt shell sbt> mimaReportBinaryIssues … [error] * method setConnection(java.sql.Connection)jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder in class jp.ne.opt.bigqueryfake.FakeBigQueryOptions#Builder does not have a correspondent in current version ... https://github.com/lightbend/migration-manager
24. Versioning バージョン番号の扱い方

25. Nevertheless, incompatibility is unavoidable If you keep maintaining libraries, someday you’ll be forced to make incompatible changes. • API is poorly designed and can’t stand future evolution • Design is forced to be changed by external events, like a breaking change in dependent library This is where versioning comes. It’s a methodology for handling incompatibility softly. There are two frequently-used conventions in the Scala world. とはいえ非互換な変更を余儀なくされる時は来る。バージョン番号 をうまく使えば非互換性をよりソフトに扱えるようになる

26. Semantic Versioning (SemVer) セマンティック・バージョニング(SemVer)でのバージョンの上げ方。
 1.4.3 major: bumped when incompatible API changes are made minor: bumped when backward- compatible functionalities are added patch: bumped when backward- compatible bug fix are made Widely used convention for many languages.
27. Java/Scala Style Versioning Java/Scalaスタイルのバージョンの上げ方。
 1.4.3 epoch: bumped on epoch-making changes major: bumped when incompatible API changes are made minor: bumped when backward- compatible functionalities are added Originated from Java, this style is used by many Java and Scala projects, including Scala compiler/runtime.
28. Then, which to use? • SemVer is the primary choice for new libraries • Whichever you choose, it is important to state your libraries’ versioning policy explicitly • Versioning policies of popular libraries: ◦ Play ◦ Cats ◦ ScalikeJDBC これから作るならSemVerが第一選択。どちらを選ぶにせよ、バー ジョニングポリシーとして明示しておくことが大事
 Since Play 2.0.0, Play is versioned as epoch.major.minor. Play currently releases a new major version about every year. Major versions can break APIs, but we try to make sure most existing code will compile with deprecation. After 1.0.0 release, we decided to use MAJOR.MINOR.PATCH Semantic Versioning 2.0.0 going forward, which is different from the EPOCH.MAJOR.MINOR scheme common among Java and Scala libraries (including the Scala lang). • major: Change only when the supported Scala major version changes: ◦ Version 1 supports Scala 2.9 & 2.10 ◦ Version 2 supports Scala 2.10, Scala 2.11 and Scala 2.12 ◦ Version 3 supports Scala 2.10, Scala 2.11 and Scala 2.12 • minor: Change to indicate functionality and API compatibility changes, the same minor version must provide the same functionality and APIs • fix: For releasing smaller improvements, bug fixes and new features
29. Persistent Versioning For the libraries which are depended heavily by other libraries, it might be worth to consider using a convention called Persistent Versioning when making incompatible upgrade. 他のライブラリから依存されているようなライブラリに非互換な変更 を加える際はPersistent Versioningを検討しよう
 “com.example.library” % “library” % “1.0.0” package com.example.library “com.example.library” % “library2” % “2.0.0” package com.example.library2 http://eed3si9n.com/persistent-versioning
30. Let users know about changes you made! It’s always a good practice to have a brief summary of changes applied to the library, especially for breaking one. There’s a good guideline for it. ユーザーにどんな変更（とりわけ破壊的な）があったか伝わるよう変 更履歴を書こう。keepachangelogのスタイルがお勧め
 ## [0.0.7] - 2015-02-16 ### Added - Link, and make it obvious that date format is ISO 8601. ### Changed - Clarified the section on "Is there a standard change log format?". ### Fixed - Fix Markdown links to tag comparison URL with footnote-style links. https://keepachangelog.com/
31. Wrapping up まとめ

32. ライブラリは使われてはじめてペイする。バイナリ互換性に注意。 バージョニングで非互換性をソフトに
 • Library is a reusable solution, it is important to be used widely • Supporting multiple JVM and Scala versions and keeping interoperability with Java increases the chance of being used • Keeping compatibility is important. In particular, binary compatibility requires special attention • Deal with incompatibility using versioning scheme
33. You’ve now learned how to maintain a library stably. Let’s roll your own and help other developers! これで安定してライブラリをメンテする準備はととのったはず。あな たもライブラリを公開して他の人を助けよう！

34. オプトではScalaエンジニアをはじめエンジニアを積極採用中です！ ご興味があればブースか弊社社員までお声がけを!！
 Our company Opt is eagerly looking for… • Scala Engineer • Data Science Engineer • Engineering Manager Come visit us at the booth if you’re interested!! PR
35. Questions?