5 tips to build long-lasting Scala OSS

https://scalebythebay2018.sched.com/event/ImvE/5-tips-to-build-long-lasting-scala-oss

Seven years ago, I started ScalikeJDBC, one of the popular database libraries in Scala. After years, I am still working on it. In the meantime, the Scala community has been growing sharply. Despite the growth of the community, we haven't seen the increase in long-lasting OSS projects yet. In this talk, I will share pieces of my knowledge learned through my experiences with OSS.

The 5 tips I will share in this talk are

1) Find your lifework project
2) Be careful about adding Scala dependencies
3) Stick with binary compatibility
4) Provide cross builds
5) Have effective CI builds

If you'd like to know more about the tips, this talk is for you.

* sbt project template including all tips: https://github.com/seratch/long-lasting-scala.g8

Transcript

1. 5 tips to build
   long-lasting
   Scala OSS
   Kazuhiro Sera
   @seratch
   Scale By The Bay 2018
   

   View Slide

2. @seratch
   • My name is Kaz (Kazuhiro Sera)
   • Happy with Scala for 8 years
   • Came from Tokyo, Japan !
   • Come and join ScalaMatsuri in June next year!
   • My 2nd talk at Scal(ae) By The Bay
   • Love open source software projects
   • Started ScalikeJDBC, Skinny Framework, AWScala, etc
   • Took over the maintenance of json4s, Scalate
   
   

   View Slide

3. What is ScalikeJDBC?
   • My most long-lasting OSS (7 year history)
   • A relational database client library in Scala
   • Provide a “Scala-like” (Scala idiomatic) way
   • Practical: easy-to-use APIs + less dependencies
   • Brief history:
   • Started as a simple lib similar to Twitter’s querulous
   • Has more functionalities now
   • Scala 2.10: String Interpolation, Macros
   • SQL objects, one-to-x extraction, type-safe DSL
   

   View Slide

4. Basic usage of ScalikeJDBC
   • Only 3 steps to issue a database query:
   • 1) Build a SQL object
   • 2) Define a function to extract values from ResultSet
   • 3) Perform SQL execution with an implicitly passed
   database session (which can be transaction-wired)
   1)> val q: SQL[Nothing, NoExtractor] = sql"select id, name from members limit 2"
   2)> val toTuple: (WrappedResultSet) => (Int, Option[String]) =
   | (rs) => (rs.get[Int]("id"), rs.get[Option[String]]("name"))
   3)> val rows: Seq[(Int, Option[String])] = q.map(toTuple).list.apply()
   [SQL Execution] select id, name from members limit 2; (0 ms)
   rows: Seq[(Int, Option[String])] = List((1,Some(Alice)), (2,Some(Bob)))
   

   View Slide

5. Other libraries?
   • According to the number of stars,
   • 1) Slick 2,124
   • 2) Quill 1,311
   • 3) Doobie 1,283
   • 4) ScalikeJDBC 948
   • ScalikeJDBC is ranked fourth (Not bad!)
   • Try in-memory database example on scalikejdbc.org
   • Any feedback? Reach out to me today
   

   View Slide

6. ScalikeJDBC turned 7 yrs old!
   

   View Slide

7. Happened during the time
   • Scala binary versions: 5
   • 2.9.1 ~ 2.9.2 ~ 2.10 ~ 2.11 ~ 2.12 ~ (2.13.0-M5 ~)
   • (Prior to 2.10, every single patch version was bin-incompatible)
   • JDK major versions: 5
   • 7 ~ 8(LTS) ~ 9 ~ 10 ~ 11(LTS) ~
   • JDBC API spec minor updates: 3
   • 4.1 (JDK7) ~ 4.2 (JDK8) ~ 4.3 (JDK9)
   • Added 4.2 APIs when dropping JDK7 support
   • sbt migration: 0.13 to 1
   • For the sbt plugin generating source code from database
   

   View Slide

8. 5 tips to build long-lasting OSS
   • 1) Find your lifework project
   • 2) Be careful to add Scala dependencies
   • 3) Stick with binary compatibility
   • 4) Provide cross builds
   • 5) Have effective CI builds
   

   View Slide

9. 1) Find your lifework project
   • Not specific to Scala OSS
   • It’s all about the sustainability of voluntary works
   • Two things to know
   • Focus on lifework projects
   • Many projects terminated due to author’s social events
   • e.g. Changing job, having a family, school graduation
   • Can continue it regardless of those big changes?
   • Difficulty to find successors
   • Inconvenient truth; even if you have many users…
   • Ideally, continue working on your project somehow
   

   View Slide

10. 2) Be careful to add Scala deps
    • It’s unfortunate that I have to say this but it’s critical
    • Two things to know
    • "You’re in the same boat with many others”
    •
    Major upgrade: have to wait for all releases; it causes
    significant delay of your release
    •
    Cross builds: can’t do if just one of them doesn’t
    • Suggestion: Choose Java over Scala inside
    • Not cool… but I have to say it’s a realistic trade-off
    •
    Pros: Can make upgrade/cross builds much easier
    •
    Cons: Handle null values inside
    

    View Slide

11. 2’) Work with null in Scala
    <<< Working with java.util.List >>>
    scala> import scala.collection.JavaConverters._
    import scala.collection.JavaConverters._
    scala> val j = java.util.Arrays.asList("A", "B", "C")
    j: java.util.List[String] = [A, B, C]
    scala> val s = j.asScala
    s: scala.collection.mutable.Buffer[String] = Buffer(A, B, C)
    scala> null.asInstanceOf[java.util.List[String]].asScala // use Option!
    res0: scala.collection.mutable.Buffer[String] = null
    <<< Working with primitives >>>
    scala> val j: java.lang.Integer = null
    j: Integer = null
    scala> val s: Int = j // auto boxing
    s: Int = 0
    scala> val s = Option(j).map(Int.unbox)
    s: Option[Int] = None
    java.util.List can be null.
    null.asScala returns null.
    A null value can be unexpectedly
    converted to the type’s default value.
    (0 : scala.Int’s default value)
    

    View Slide

12. 3) Stick with bin-compatibility
    • Bin-compatible?
    • Normal situation; not see any runtime linkage errors
    • If you break bin-compatibility:
    • Painful runtime errors (NoSuchMethodError etc)
    • Scala compiler cannot detect runtime issues beforehand
    • Two things to do
    • Define policies about bin-compatibility guarantee
    • Once you released the first stable version
    • Surely guarantee binary-compatibility
    • Solution: Use MiMa
    

    View Slide

13. 3’) Use MiMa to ensure that
    • Use MiMa (Migration Manager for Scala)
    • Can detect syntactic incompatibilities
    • Developed and maintained by Lightbend devs
    // project/plugins.sbt
    addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")
    // build.sbt
    project.settings(
    MimaPlugin.mimaDefaultSettings ++ Seq(
    mimaPreviousArtifacts := {
    Set("0.1.0", "0.1.1").map { v =>
    organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v
    }
    },
    test in Test := {
    mimaReportBinaryIssues.value
    (test in Test).value
    }
    )
    )
    

    View Slide

14. 4) Provide cross builds
    • Cross builds = publish against multiple Scala bin versions
    • Reality: We cannot always use the latest Scala
    • Apache Spark took 2 years to support Scala 2.12
    • Remember the tip 2) “be careful to add Scala deps”
    • Two things to do
    • Support latest stable major & its previous major
    • To be specific, Scala 2.12 and 2.11 at this moment
    • Don’t have many files under src/main/scala-X.Y
    • Testing a plenty of code there is hard+time-consuming
    

    View Slide

15. 5) Have effective CI builds
    • Why do we need it?
    • To reduce review cost of pull requests
    • To merge pull requests with confidence
    • You need reliable evidences (regressions, bin-combat, code style, etc)
    • Two things to do
    • Enable it anyway!
    • You can use TravisCI/CircleCI for free
    • Automate everything apart from code reviews
    • Check bin-compatibility, consistency of code style, cross
    builds, multiple JDKs, etc
    • Configure matrix builds to cover all patterns
    

    View Slide

16. 5’) TravisCI Build Example
    language: scala
    sudo: false
    scala:
    - 2.11.12
    - 2.12.7
    - 2.13.0-M5
    jdk:
    - oraclejdk8
    - openjdk11
    matrix:
    include:
    - scala: 2.12.7
    jdk: oraclejdk9
    script:
    sbt ++$TRAVIS_SCALA_VERSION scalafmtCheck test # code formatter check + test
    • Scala 2.11 + JDK 8, 11 = 2 builds
    • Scala 2.12 + JDK 8, 9, 11 = 3 builds
    • Scala 2.13.0-M5 + JDK 8, 11 = 2 builds
    Additional 1 matrix build
    3 Scala versions * 2 JDKs
    = 6 patterns
    7 patterns in total
    

    View Slide

17. Start with the sbt template!
    • sbt project template (aka g8 template)
    • Including all techniques in this talk
    • Should be a good starting point for you
    mkdir my-awesome-library
    cd my-awesome-library
    sbt new seratch/long-lasting-scala.g8
    sbt test
    

    View Slide

18. Recap
    • You found out about ScalikeJDBC
    • Contact me at any time during the conf
    • If you like it, increment the stars (1K soon!)
    • 5 tips to build long-lasting Scala OSS
    • 1) Find what interests you for years
    • 2) Be careful about adding Scala dependencies
    • 3) Stick with binary compatibility
    • 4) Provide cross builds
    • 5) Have effective CI builds
    

    View Slide