Upgrade to Pro — share decks privately, control downloads, hide ads and more …

5 tips to build long-lasting Scala OSS (cont’d)

5 tips to build long-lasting Scala OSS (cont’d)

A presentation at Tokyo Scala Developers on Dec 6, 2018.
https://www.meetup.com/Tokyo-Scala-Developers/events/256619365/

Kazuhiro Sera

December 06, 2018
Tweet

More Decks by Kazuhiro Sera

Other Decks in Programming

Transcript

  1. 5 tips to build
    long-lasting
    Scala OSS (cont’d)
    Kazuhiro Sera
    @seratch
    Tokyo Scala Developers
    (Dec 6, 2018)

    View full-size slide

  2. @seratch
    • My name is Kaz (Kazuhiro Sera)
    • Has been happy with Scala for 8 years
    • Love open source software projects
    • Started ScalikeJDBC, Skinny Framework, AWScala, etc
    • Took over the maintenance of json4s, Scalate
    • Gave this talk at Scale By The Bay 2018
    https://www.youtube.com/
    watch?v=nN7yMChrpMM

    View full-size slide

  3. What is ScalikeJDBC?
    • My most long-lasting OSS (7 year history)
    • A relational database client library in Scala
    • Provide a “Scala-like” (Scala idiomatic) way
    • Practical: easy-to-use APIs + less dependencies
    • Brief history:
    • Started as a simple lib similar to Twitter’s querulous
    • Has more functionalities now
    • Scala 2.10: String Interpolation, Macros
    • SQL objects, one-to-x extraction, type-safe DSL

    View full-size slide

  4. Basic usage of ScalikeJDBC
    • Only 3 steps to issue a database query:
    • 1) Build a SQL object
    • 2) Define a function to extract values from ResultSet
    • 3) Perform SQL execution with an implicitly passed
    database session (which can be transaction-wired)
    1)> val q: SQL[Nothing, NoExtractor] = sql"select id, name from members limit 2"
    2)> val toTuple: (WrappedResultSet) => (Int, Option[String]) =
    | (rs) => (rs.get[Int]("id"), rs.get[Option[String]]("name"))
    3)> val rows: Seq[(Int, Option[String])] = q.map(toTuple).list.apply()
    [SQL Execution] select id, name from members limit 2; (0 ms)
    rows: Seq[(Int, Option[String])] = List((1,Some(Alice)), (2,Some(Bob)))

    View full-size slide

  5. Other options in the same field
    • According to the number of stars,
    • 1) Slick 2,133
    • 2) Quill 1,324
    • 3) Doobie 1,298
    • 4) ScalikeJDBC 953
    • ScalikeJDBC is ranked fourth (Not bad!)
    • Try in-memory database example on scalikejdbc.org
    • Any feedback? Reach out to me today

    View full-size slide

  6. ScalikeJDBC turned 7 yrs old!

    View full-size slide

  7. Upgrades during the time
    • Scala binary versions: 5
    • 2.9.1 ~ 2.9.2 ~ 2.10 ~ 2.11 ~ 2.12 ~ (2.13.0-M5 ~)
    • (Prior to 2.10, every single patch version was bin-incompatible)
    • JDK major versions: 5
    • 7 ~ 8(LTS) ~ 9 ~ 10 ~ 11(LTS) ~
    • JDBC API spec minor updates: 3
    • 4.1 (JDK7) ~ 4.2 (JDK8) ~ 4.3 (JDK9)
    • Added 4.2 APIs when dropping JDK7 support
    • sbt migration: 0.13 to 1
    • For the sbt plugin generating source code from database

    View full-size slide

  8. 5 tips to build long-lasting OSS
    • 1) Find & focus your lifework project
    • 2) Be deliberate on Scala dependencies
    • 3) Stick with binary compatibility
    • 4) Provide cross builds
    • 5) Have effective CI builds
    Let’s get down to
    the nitty-gritty of this talk!

    View full-size slide

  9. 1) Lifework project
    • Not technical tip, not specific to Scala OSS
    • It’s all about the sustainability of voluntary works
    • Two things to know
    • Find & Focus on lifework projects
    • Many projects terminated due to author’s social events
    • e.g. Changing job, having a family, school graduation
    • If you really want to work on it for a long time..
    • Difficult to find successors
    • Inconvenient truth; even if you have many users…
    • Ideally, continue working on your project somehow

    View full-size slide

  10. 2) Be deliberate on Scala deps
    • It’s unfortunate that I have to say this, but it’s critical
    • Two things to know
    • "You’re in the same boat with many others”

    Major upgrade: have to wait for all releases; causing
    significant delay of your release

    Cross builds: the greatest common divisor
    • Suggestion: Choose Java over Scala inside
    • An unpleasant decision but a realistic trade-off

    Pros: Can make upgrade/cross builds much easier

    Cons: Requires dirty works like handling null
    values

    View full-size slide

  11. 2’) Live with Java in Scala
    • Utilize scala.collection.JavaConveters._
    • Allow easily converting Java objects to Scala objects
    • JavaConverters._: enrich my library pattern
    • JavaConversions._ (implicit conversions) are deprecated
    • Be always aware of null values
    • Like a professional Java developer
    • Obsessively use Option everywhere
    • java.util.List can be null = javaList.asScala may return
    null value for scala.collection.Seq
    • Primitive wrappers can be null + auto boxing

    View full-size slide

  12. 2’) Example Code
    <<< Working with java.util.List >>>
    scala> import scala.collection.JavaConverters._
    import scala.collection.JavaConverters._
    scala> val j = java.util.Arrays.asList("A", "B", "C")
    j: java.util.List[String] = [A, B, C]
    scala> val s = j.asScala
    s: scala.collection.mutable.Buffer[String] = Buffer(A, B, C)
    scala> null.asInstanceOf[java.util.List[String]].asScala // use Option!
    res0: scala.collection.mutable.Buffer[String] = null
    <<< Working with primitives >>>
    scala> val j: java.lang.Integer = null
    j: Integer = null
    scala> val s: Int = j // auto boxing
    s: Int = 0
    scala> val s = Option(j).map(Int.unbox)
    s: Option[Int] = None
    java.util.List can be null.
    null.asScala returns null.
    A null value can be unexpectedly
    converted to the type’s default value.
    (0 : scala.Int’s default value)

    View full-size slide

  13. 3) Stick with bin-compatibility
    • Bin-compatible?
    • Normal situation; no runtime linkage errors
    • If you break bin-compatibility:
    • Painful runtime errors (NoSuchMethodError etc)
    • Scala compiler cannot detect runtime issues…
    • Two things to do
    • Have clear policies about bin-compatibility guarantee
    • After the first stable version release
    • Guaranteeing among patch versions
    • Use MiMa to surely guarantee binary-compatibility

    View full-size slide

  14. 3’) Use MiMa to ensure that
    • MiMa: Migration Manager for Scala
    • Detect syntactic incompatibilities
    • Compare the byte code from current source code with jar
    files in past releases
    • Developed and has been maintained by Lightbend
    // project/plugins.sbt
    addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")
    // build.sbt
    project.settings(MimaPlugin.mimaDefaultSettings ++
    Seq(
    mimaPreviousArtifacts := {
    Set("0.1.0", "0.1.1").map { v =>
    organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v
    }
    },
    test in Test := {
    mimaReportBinaryIssues.value
    (test in Test).value
    }
    ))

    View full-size slide

  15. 3’) Java bin-compatibility?
    • Are Java libraries always compatible with past releases?
    • No, they aren’t. Not 100% safe.
    • Jackson minor versions are intentionally incompatible
    • Also, binary compatibility can be broken by mistake
    • I can say it’s undoubtedly less frequent than Scala
    • Validating all dependency upgrades is unrealistic
    • It’s same for both Java and Scala
    • Carefully choose trusted libraries
    • Have unit tests to detect runtime issues

    View full-size slide

  16. 4) Provide cross builds
    • Cross builds = publish against multiple Scala bin versions
    • Why? To acquire more users in different situations
    • Reality: We cannot always use the latest Scala
    • Apache Spark took 2 years to support Scala 2.12
    • Two things to do
    • Support latest stable major & its previous
    • To be specific, Scala 2.12 and 2.11 at this moment
    • Don’t have too many files under src/main/scala-X.Y
    • Having multiple implementations for same API
    • Testing plenty of such code is hard+time-consuming

    View full-size slide

  17. 5) Have effective CI builds
    • Why do we need it?
    • To reduce review cost of pull requests
    • To merge pull requests with confidence
    • Reliable evidences needed (regressions, bin-combat, code style, etc)
    • Two things to do
    • Enable it anyway if not yet done!
    • You can use TravisCI/CircleCI for free
    • Try automating everything apart from reviews
    • Check bin-compatibility, code style consistency, cross
    builds, multiple JDKs, etc
    • Configure matrix builds to cover all patterns

    View full-size slide

  18. 5’) TravisCI Build Example
    language: scala
    sudo: false
    scala:
    - 2.11.12
    - 2.12.7
    - 2.13.0-M5
    jdk:
    - oraclejdk8
    - openjdk11
    matrix:
    include:
    - scala: 2.12.7
    jdk: oraclejdk9
    script:
    sbt ++$TRAVIS_SCALA_VERSION scalafmtCheck test # code formatter check + test
    • Scala 2.11 + JDK 8, 11 = 2 builds
    • Scala 2.12 + JDK 8, 9, 11 = 3 builds
    • Scala 2.13.0-M5 + JDK 8, 11 = 2 builds
    Additional 1 matrix build
    3 Scala versions * 2 JDKs
    = 6 patterns
    7 patterns in total
    1) Make sure if code
    formatting is done
    2) Do compilation
    3) Run tests

    View full-size slide

  19. Start with the sbt template!
    • sbt project template (aka g8 template)
    • Including all techniques in this talk
    • Should be a good starting point for you
    mkdir my-awesome-library
    cd my-awesome-library
    sbt new seratch/long-lasting-scala.g8
    sbt test

    View full-size slide

  20. Demo
    $ sbt new seratch/long-lasting-scala.g8
    organizationName [com.example]: com.github.seratch
    projectName [my-long-lasting-project]: my-awesome-library
    githubAccountName [(Your GitHub account ID)]: seratch
    yourFullName [(Your full name)]: Kazuhiro Sera
    Template applied in /Users/ksera/tmp/my-awesome-library/.

    View full-size slide

  21. Demo - build.sbt
    lazy val root = (project in file("."))
    .settings(
    organization := "com.github.seratch",
    name := "my-awesome-library",
    version := "0.1.0-SNAPSHOT",
    scalaVersion := "2.12.7",
    crossScalaVersions := Seq("2.12.7", "2.11.12"),
    scalacOptions ++= Seq("-deprecation", "-unchecked", "-feature", "-Xfuture"),
    libraryDependencies ++= Seq(
    "org.scalatest" %% "scalatest" % "3.0.5" % Test
    ),
    // FYI: https://www.scala-sbt.org/1.0/docs/Using-Sonatype.html
    publishTo := { /* omitted */ },
    pomExtra := { /* omitted */ }
    )
    .settings(mimaSettings)
    val mimaSettings = MimaPlugin.mimaDefaultSettings ++ Seq(
    mimaPreviousArtifacts := {
    val previousVersions: Set[String] = Set.empty // e.g. Set("0.1.0", "0.1.1")
    previousVersions.map { v =>
    organization.value % s"${name.value}_${scalaBinaryVersion.value}" % v
    }
    },
    test in Test := {
    mimaReportBinaryIssues.value
    (test in Test).value
    }
    )

    View full-size slide

  22. Demo - project/plugins.sbt
    scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature")
    // MiMa ensures bin-compatibility among patch releases
    // https://github.com/lightbend/migration-manager
    addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")
    // Code formatter for Scala
    // https://scalameta.org/scalafmt/
    addSbtPlugin("com.geirsson" % "sbt-scalafmt" % "1.5.1")
    // Publishing this library to Sonatype repositories + Maven Central
    // https://www.scala-sbt.org/1.0/docs/Using-Sonatype.html
    addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.2")
    addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.3")

    View full-size slide

  23. Demo - .travis.yml
    language: scala
    sudo: false
    scala:
    - 2.11.12
    - 2.12.7
    - 2.13.0-M5
    jdk:
    - oraclejdk8
    - openjdk11
    matrix:
    include:
    - scala: 2.12.7
    jdk: oraclejdk9
    script:
    sbt ++$TRAVIS_SCALA_VERSION scalafmtSbtCheck scalafmtCheck test

    View full-size slide

  24. Recap
    • You found out about ScalikeJDBC
    • Contact me at any time during the conf
    • If you like it, increment the stars (1K soon!)
    • 5 tips to build long-lasting Scala OSS
    • 1) Find & focus your lifework project
    • 2) Be deliberate on Scala dependencies
    • 3) Stick with binary compatibility
    • 4) Provide cross builds
    • 5) Have effective CI builds

    View full-size slide