5 tips to build long-lasting Scala OSS (cont’d)

A presentation at Tokyo Scala Developers on Dec 6, 2018.
https://www.meetup.com/Tokyo-Scala-Developers/events/256619365/

Transcript

1. 5 tips to build long-lasting Scala OSS (cont’d) Kazuhiro Sera @seratch Tokyo Scala Developers (Dec 6, 2018)
2. @seratch • My name is Kaz (Kazuhiro Sera) • Has been happy with Scala for 8 years • Love open source software projects • Started ScalikeJDBC, Skinny Framework, AWScala, etc • Took over the maintenance of json4s, Scalate • Gave this talk at Scale By The Bay 2018 https://www.youtube.com/ watch?v=nN7yMChrpMM
3. What is ScalikeJDBC? • My most long-lasting OSS (7 year history) • A relational database client library in Scala • Provide a “Scala-like” (Scala idiomatic) way • Practical: easy-to-use APIs + less dependencies • Brief history: • Started as a simple lib similar to Twitter’s querulous • Has more functionalities now • Scala 2.10: String Interpolation, Macros • SQL objects, one-to-x extraction, type-safe DSL
4. Basic usage of ScalikeJDBC • Only 3 steps to issue a database query: • 1) Build a SQL object • 2) Define a function to extract values from ResultSet • 3) Perform SQL execution with an implicitly passed database session (which can be transaction-wired) 1)> val q: SQL[Nothing, NoExtractor] = sql"select id, name from members limit 2" 2)> val toTuple: (WrappedResultSet) => (Int, Option[String]) = | (rs) => (rs.get[Int]("id"), rs.get[Option[String]]("name")) 3)> val rows: Seq[(Int, Option[String])] = q.map(toTuple).list.apply() [SQL Execution] select id, name from members limit 2; (0 ms) rows: Seq[(Int, Option[String])] = List((1,Some(Alice)), (2,Some(Bob)))
5. Other options in the same field • According to the number of stars, • 1) Slick 2,133 • 2) Quill 1,324 • 3) Doobie 1,298 • 4) ScalikeJDBC 953 • ScalikeJDBC is ranked fourth (Not bad!) • Try in-memory database example on scalikejdbc.org • Any feedback? Reach out to me today
6. ScalikeJDBC turned 7 yrs old!
7. Upgrades during the time • Scala binary versions: 5 • 2.9.1 ~ 2.9.2 ~ 2.10 ~ 2.11 ~ 2.12 ~ (2.13.0-M5 ~) • (Prior to 2.10, every single patch version was bin-incompatible) • JDK major versions: 5 • 7 ~ 8(LTS) ~ 9 ~ 10 ~ 11(LTS) ~ • JDBC API spec minor updates: 3 • 4.1 (JDK7) ~ 4.2 (JDK8) ~ 4.3 (JDK9) • Added 4.2 APIs when dropping JDK7 support • sbt migration: 0.13 to 1 • For the sbt plugin generating source code from database
8. 5 tips to build long-lasting OSS • 1) Find & focus your lifework project • 2) Be deliberate on Scala dependencies • 3) Stick with binary compatibility • 4) Provide cross builds • 5) Have effective CI builds Let’s get down to the nitty-gritty of this talk!
9. 1) Lifework project • Not technical tip, not specific to Scala OSS • It’s all about the sustainability of voluntary works • Two things to know • Find & Focus on lifework projects • Many projects terminated due to author’s social events • e.g. Changing job, having a family, school graduation • If you really want to work on it for a long time.. • Difficult to find successors • Inconvenient truth; even if you have many users… • Ideally, continue working on your project somehow
10. 2) Be deliberate on Scala deps • It’s unfortunate that I have to say this, but it’s critical • Two things to know • "You’re in the same boat with many others” • Major upgrade: have to wait for all releases; causing significant delay of your release • Cross builds: the greatest common divisor • Suggestion: Choose Java over Scala inside • An unpleasant decision but a realistic trade-off • Pros: Can make upgrade/cross builds much easier • Cons: Requires dirty works like handling null values
11. 2’) Live with Java in Scala • Utilize scala.collection.JavaConveters._ • Allow easily converting Java objects to Scala objects • JavaConverters._: enrich my library pattern • JavaConversions._ (implicit conversions) are deprecated • Be always aware of null values • Like a professional Java developer • Obsessively use Option everywhere • java.util.List can be null = javaList.asScala may return null value for scala.collection.Seq • Primitive wrappers can be null + auto boxing
12. 2’) Example Code <<< Working with java.util.List >>> scala> import scala.collection.JavaConverters._ import scala.collection.JavaConverters._ scala> val j = java.util.Arrays.asList("A", "B", "C") j: java.util.List[String] = [A, B, C] scala> val s = j.asScala s: scala.collection.mutable.Buffer[String] = Buffer(A, B, C) scala> null.asInstanceOf[java.util.List[String]].asScala // use Option! res0: scala.collection.mutable.Buffer[String] = null <<< Working with primitives >>> scala> val j: java.lang.Integer = null j: Integer = null scala> val s: Int = j // auto boxing s: Int = 0 scala> val s = Option(j).map(Int.unbox) s: Option[Int] = None java.util.List can be null. null.asScala returns null. A null value can be unexpectedly converted to the type’s default value. (0 : scala.Int’s default value)
13. 3) Stick with bin-compatibility • Bin-compatible? • Normal situation; no runtime linkage errors • If you break bin-compatibility: • Painful runtime errors (NoSuchMethodError etc) • Scala compiler cannot detect runtime issues… • Two things to do • Have clear policies about bin-compatibility guarantee • After the first stable version release • Guaranteeing among patch versions • Use MiMa to surely guarantee binary-compatibility
14. 3’) Use MiMa to ensure that • MiMa: Migration Manager for Scala • Detect syntactic incompatibilities • Compare the byte code from current source code with jar files in past releases • Developed and has been maintained by Lightbend // project/plugins.sbt addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0") // build.sbt project.settings(MimaPlugin.mimaDefaultSettings ++ Seq( mimaPreviousArtifacts := { Set("0.1.0", "0.1.1").map { v => organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v } }, test in Test := { mimaReportBinaryIssues.value (test in Test).value } ))
15. 3’) Java bin-compatibility? • Are Java libraries always compatible with past releases? • No, they aren’t. Not 100% safe. • Jackson minor versions are intentionally incompatible • Also, binary compatibility can be broken by mistake • I can say it’s undoubtedly less frequent than Scala • Validating all dependency upgrades is unrealistic • It’s same for both Java and Scala • Carefully choose trusted libraries • Have unit tests to detect runtime issues
16. 4) Provide cross builds • Cross builds = publish against multiple Scala bin versions • Why? To acquire more users in different situations • Reality: We cannot always use the latest Scala • Apache Spark took 2 years to support Scala 2.12 • Two things to do • Support latest stable major & its previous • To be specific, Scala 2.12 and 2.11 at this moment • Don’t have too many files under src/main/scala-X.Y • Having multiple implementations for same API • Testing plenty of such code is hard+time-consuming
17. 5) Have effective CI builds • Why do we need it? • To reduce review cost of pull requests • To merge pull requests with confidence • Reliable evidences needed (regressions, bin-combat, code style, etc) • Two things to do • Enable it anyway if not yet done! • You can use TravisCI/CircleCI for free • Try automating everything apart from reviews • Check bin-compatibility, code style consistency, cross builds, multiple JDKs, etc • Configure matrix builds to cover all patterns
18. 5’) TravisCI Build Example language: scala sudo: false scala: - 2.11.12 - 2.12.7 - 2.13.0-M5 jdk: - oraclejdk8 - openjdk11 matrix: include: - scala: 2.12.7 jdk: oraclejdk9 script: sbt ++$TRAVIS_SCALA_VERSION scalafmtCheck test # code formatter check + test • Scala 2.11 + JDK 8, 11 = 2 builds • Scala 2.12 + JDK 8, 9, 11 = 3 builds • Scala 2.13.0-M5 + JDK 8, 11 = 2 builds Additional 1 matrix build 3 Scala versions * 2 JDKs = 6 patterns 7 patterns in total 1) Make sure if code formatting is done 2) Do compilation 3) Run tests
19. Start with the sbt template! • sbt project template (aka g8 template) • Including all techniques in this talk • Should be a good starting point for you mkdir my-awesome-library cd my-awesome-library sbt new seratch/long-lasting-scala.g8 sbt test
20. Demo $ sbt new seratch/long-lasting-scala.g8 organizationName [com.example]: com.github.seratch projectName [my-long-lasting-project]: my-awesome-library githubAccountName [(Your GitHub account ID)]: seratch yourFullName [(Your full name)]: Kazuhiro Sera Template applied in /Users/ksera/tmp/my-awesome-library/.
21. Demo - build.sbt lazy val root = (project in file(".")) .settings( organization := "com.github.seratch", name := "my-awesome-library", version := "0.1.0-SNAPSHOT", scalaVersion := "2.12.7", crossScalaVersions := Seq("2.12.7", "2.11.12"), scalacOptions ++= Seq("-deprecation", "-unchecked", "-feature", "-Xfuture"), libraryDependencies ++= Seq( "org.scalatest" %% "scalatest" % "3.0.5" % Test ), // FYI: https://www.scala-sbt.org/1.0/docs/Using-Sonatype.html publishTo := { /* omitted */ }, pomExtra := { /* omitted */ } ) .settings(mimaSettings) val mimaSettings = MimaPlugin.mimaDefaultSettings ++ Seq( mimaPreviousArtifacts := { val previousVersions: Set[String] = Set.empty // e.g. Set("0.1.0", "0.1.1") previousVersions.map { v => organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v } }, test in Test := { mimaReportBinaryIssues.value (test in Test).value } )
22. Demo - project/plugins.sbt scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature") // MiMa ensures bin-compatibility among patch releases // https://github.com/lightbend/migration-manager addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0") // Code formatter for Scala // https://scalameta.org/scalafmt/ addSbtPlugin("com.geirsson" % "sbt-scalafmt" % "1.5.1") // Publishing this library to Sonatype repositories + Maven Central // https://www.scala-sbt.org/1.0/docs/Using-Sonatype.html addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.2") addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3")
23. Demo - .travis.yml language: scala sudo: false scala: - 2.11.12 - 2.12.7 - 2.13.0-M5 jdk: - oraclejdk8 - openjdk11 matrix: include: - scala: 2.12.7 jdk: oraclejdk9 script: sbt ++$TRAVIS_SCALA_VERSION scalafmtSbtCheck scalafmtCheck test
24. Recap • You found out about ScalikeJDBC • Contact me at any time during the conf • If you like it, increment the stars (1K soon!) • 5 tips to build long-lasting Scala OSS • 1) Find & focus your lifework project • 2) Be deliberate on Scala dependencies • 3) Stick with binary compatibility • 4) Provide cross builds • 5) Have effective CI builds