5 tips to build long-lasting Scala OSS

https://scalebythebay2018.sched.com/event/ImvE/5-tips-to-build-long-lasting-scala-oss

Seven years ago, I started ScalikeJDBC, one of the popular database libraries in Scala. After years, I am still working on it. In the meantime, the Scala community has been growing sharply. Despite the growth of the community, we haven't seen the increase in long-lasting OSS projects yet. In this talk, I will share pieces of my knowledge learned through my experiences with OSS.

The 5 tips I will share in this talk are

1) Find your lifework project
2) Be careful about adding Scala dependencies
3) Stick with binary compatibility
4) Provide cross builds
5) Have effective CI builds

If you'd like to know more about the tips, this talk is for you.

* sbt project template including all tips: https://github.com/seratch/long-lasting-scala.g8

Transcript

1. 5 tips to build long-lasting Scala OSS Kazuhiro Sera @seratch Scale By The Bay 2018
2. @seratch • My name is Kaz (Kazuhiro Sera) • Happy with Scala for 8 years • Came from Tokyo, Japan ! • Come and join ScalaMatsuri in June next year! • My 2nd talk at Scal(ae) By The Bay • Love open source software projects • Started ScalikeJDBC, Skinny Framework, AWScala, etc • Took over the maintenance of json4s, Scalate
3. What is ScalikeJDBC? • My most long-lasting OSS (7 year history) • A relational database client library in Scala • Provide a “Scala-like” (Scala idiomatic) way • Practical: easy-to-use APIs + less dependencies • Brief history: • Started as a simple lib similar to Twitter’s querulous • Has more functionalities now • Scala 2.10: String Interpolation, Macros • SQL objects, one-to-x extraction, type-safe DSL
4. Basic usage of ScalikeJDBC • Only 3 steps to issue a database query: • 1) Build a SQL object • 2) Define a function to extract values from ResultSet • 3) Perform SQL execution with an implicitly passed database session (which can be transaction-wired) 1)> val q: SQL[Nothing, NoExtractor] = sql"select id, name from members limit 2" 2)> val toTuple: (WrappedResultSet) => (Int, Option[String]) = | (rs) => (rs.get[Int]("id"), rs.get[Option[String]]("name")) 3)> val rows: Seq[(Int, Option[String])] = q.map(toTuple).list.apply() [SQL Execution] select id, name from members limit 2; (0 ms) rows: Seq[(Int, Option[String])] = List((1,Some(Alice)), (2,Some(Bob)))
5. Other libraries? • According to the number of stars, • 1) Slick 2,124 • 2) Quill 1,311 • 3) Doobie 1,283 • 4) ScalikeJDBC 948 • ScalikeJDBC is ranked fourth (Not bad!) • Try in-memory database example on scalikejdbc.org • Any feedback? Reach out to me today
6. ScalikeJDBC turned 7 yrs old!
7. Happened during the time • Scala binary versions: 5 • 2.9.1 ~ 2.9.2 ~ 2.10 ~ 2.11 ~ 2.12 ~ (2.13.0-M5 ~) • (Prior to 2.10, every single patch version was bin-incompatible) • JDK major versions: 5 • 7 ~ 8(LTS) ~ 9 ~ 10 ~ 11(LTS) ~ • JDBC API spec minor updates: 3 • 4.1 (JDK7) ~ 4.2 (JDK8) ~ 4.3 (JDK9) • Added 4.2 APIs when dropping JDK7 support • sbt migration: 0.13 to 1 • For the sbt plugin generating source code from database
8. 5 tips to build long-lasting OSS • 1) Find your lifework project • 2) Be careful to add Scala dependencies • 3) Stick with binary compatibility • 4) Provide cross builds • 5) Have effective CI builds
9. 1) Find your lifework project • Not specific to Scala OSS • It’s all about the sustainability of voluntary works • Two things to know • Focus on lifework projects • Many projects terminated due to author’s social events • e.g. Changing job, having a family, school graduation • Can continue it regardless of those big changes? • Difficulty to find successors • Inconvenient truth; even if you have many users… • Ideally, continue working on your project somehow
10. 2) Be careful to add Scala deps • It’s unfortunate that I have to say this but it’s critical • Two things to know • "You’re in the same boat with many others” • Major upgrade: have to wait for all releases; it causes significant delay of your release • Cross builds: can’t do if just one of them doesn’t • Suggestion: Choose Java over Scala inside • Not cool… but I have to say it’s a realistic trade-off • Pros: Can make upgrade/cross builds much easier • Cons: Handle null values inside
11. 2’) Work with null in Scala <<< Working with java.util.List >>> scala> import scala.collection.JavaConverters._ import scala.collection.JavaConverters._ scala> val j = java.util.Arrays.asList("A", "B", "C") j: java.util.List[String] = [A, B, C] scala> val s = j.asScala s: scala.collection.mutable.Buffer[String] = Buffer(A, B, C) scala> null.asInstanceOf[java.util.List[String]].asScala // use Option! res0: scala.collection.mutable.Buffer[String] = null <<< Working with primitives >>> scala> val j: java.lang.Integer = null j: Integer = null scala> val s: Int = j // auto boxing s: Int = 0 scala> val s = Option(j).map(Int.unbox) s: Option[Int] = None java.util.List can be null. null.asScala returns null. A null value can be unexpectedly converted to the type’s default value. (0 : scala.Int’s default value)
12. 3) Stick with bin-compatibility • Bin-compatible? • Normal situation; not see any runtime linkage errors • If you break bin-compatibility: • Painful runtime errors (NoSuchMethodError etc) • Scala compiler cannot detect runtime issues beforehand • Two things to do • Define policies about bin-compatibility guarantee • Once you released the first stable version • Surely guarantee binary-compatibility • Solution: Use MiMa
13. 3’) Use MiMa to ensure that • Use MiMa (Migration Manager for Scala) • Can detect syntactic incompatibilities • Developed and maintained by Lightbend devs // project/plugins.sbt addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0") // build.sbt project.settings( MimaPlugin.mimaDefaultSettings ++ Seq( mimaPreviousArtifacts := { Set("0.1.0", "0.1.1").map { v => organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v } }, test in Test := { mimaReportBinaryIssues.value (test in Test).value } ) )
14. 4) Provide cross builds • Cross builds = publish against multiple Scala bin versions • Reality: We cannot always use the latest Scala • Apache Spark took 2 years to support Scala 2.12 • Remember the tip 2) “be careful to add Scala deps” • Two things to do • Support latest stable major & its previous major • To be specific, Scala 2.12 and 2.11 at this moment • Don’t have many files under src/main/scala-X.Y • Testing a plenty of code there is hard+time-consuming
15. 5) Have effective CI builds • Why do we need it? • To reduce review cost of pull requests • To merge pull requests with confidence • You need reliable evidences (regressions, bin-combat, code style, etc) • Two things to do • Enable it anyway! • You can use TravisCI/CircleCI for free • Automate everything apart from code reviews • Check bin-compatibility, consistency of code style, cross builds, multiple JDKs, etc • Configure matrix builds to cover all patterns
16. 5’) TravisCI Build Example language: scala sudo: false scala: - 2.11.12 - 2.12.7 - 2.13.0-M5 jdk: - oraclejdk8 - openjdk11 matrix: include: - scala: 2.12.7 jdk: oraclejdk9 script: sbt ++$TRAVIS_SCALA_VERSION scalafmtCheck test # code formatter check + test • Scala 2.11 + JDK 8, 11 = 2 builds • Scala 2.12 + JDK 8, 9, 11 = 3 builds • Scala 2.13.0-M5 + JDK 8, 11 = 2 builds Additional 1 matrix build 3 Scala versions * 2 JDKs = 6 patterns 7 patterns in total
17. Start with the sbt template! • sbt project template (aka g8 template) • Including all techniques in this talk • Should be a good starting point for you mkdir my-awesome-library cd my-awesome-library sbt new seratch/long-lasting-scala.g8 sbt test
18. Recap • You found out about ScalikeJDBC • Contact me at any time during the conf • If you like it, increment the stars (1K soon!) • 5 tips to build long-lasting Scala OSS • 1) Find what interests you for years • 2) Be careful about adding Scala dependencies • 3) Stick with binary compatibility • 4) Provide cross builds • 5) Have effective CI builds