Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

All I learned while working on a Scala OSS pro...

All I learned while working on a Scala OSS project for over six years #ScalaMatsuri

* http://2018.scalamatsuri.org/index_en.html
* https://www.youtube.com/watch?v=y7BxvT-Jm6w

In 2011, I started a Scala open source project named ScalikeJDBC. Thanks to many contributors, I am still working on the project after six years. In the meantime, the Scala community has been growing sharply, and the trends have been continuously changing. Despite the fact that the number of developers who work on OSS projects has increased, there are not many developers who have been working on a Scala project for several years. In this talk, I will share my experiences and pieces of knowledge through maintaining the OSS project for more than six years.

Kazuhiro Sera

March 17, 2018
Tweet

More Decks by Kazuhiro Sera

Other Decks in Programming

Transcript

  1. All I learned while working on a Scala OSS project

    for over six years Kazuhiro Sera @seratch 1 4DBMB044QSPKFDUΛ೥Ҏ্ଓ͚ΔͨΊʹඞཁͳ͜ͱ ͱ͍͏ςʔϚͰɺ੉ྑ࿨߂!TFSBUDI@KB͕͓࿩͠͠·͢ɻ
  2. In 2011, I started a Scala open source project named

    ScalikeJDBC. Thanks to many contributors, I am still working on the project after six years. In the meantime, the Scala community has been growing sharply, and the trends have been continuously changing. Despite the fact that the number of developers who work on OSS projects has increased, there are not many developers who have been working on a single Scala project for several years. In this talk, I will share my experiences and pieces of knowledge through maintaining a single OSS project for more than six years. ೥͔ΒऔΓ૊ΜͰ͖ͨ4DBMJLF+%#$Λ͸͡Ίͱ͢Δ 4DBMB044։ൃ͔Βಘͨ஌ݟͳͲΛ͓࿩͠͠·͢ɻ 2
  3. Agenda Who Am I? My “Hello World” in Scala (2009)

    ScalikeJDBC, My Long-lived Scala Project Tips for Continuation Avoiding Depending on Scala Libraries Cross-building Tips Ensuring binary-compatibility (MiMa) Building a Community of Friends ͜ͷΑ͏ͳ಺༰ʹ͍͓ͭͯ࿩͠͠·͢ɻ্ڃऀͷํʹ͸ͪΐͬ ͱ෺଍Γͳ͍಺༰͔΋͠Ε·ͤΜ͕ɺ͝༰ࣻΛɻ 3
  4. Kazuhiro Sera @seratch Software Engineer, VPoE at SmartNews, Inc. We’re

    a samurai sponsor Server-side software engineer (Scala, Java, Ruby) Love coding, open-source community A Scala enthusiast for 7+ years Creator of ScalikeJDBC, Skinny Framework One of the maintainers of Scalate, json4s, Scalatra 4NBSU/FXTͰಇ͍͍ͯ·͢ɻαʔόαΠυΤϯδχΞɺ 4DBMJLF+%#$ɺ4LJOOZͳͲͷ4DBMB044Λ΍͍ͬͯ·͢ɻ5
  5. My “Hello World” in Scala In 2009, I was a

    Java developer at the time Looking for a new one / unsatisfied with Groovy At the beginning, I tried with Scala 2.7.3 My first impression of Scala “IDE integration was still very poor. It’s a bit early to seriously use it at work.” Excited with hybrid paradigms, actors, concise syntax I started learning it seriously in the following year ॳΊͯ࢖ͬͨͷ͸Ͱ౰࣌͸+BWBΤϯδχΞͰͨ͠ɻ ࢓ࣄʹ͸·ͩ࢖͑ͳ͍͚Ͳ໘ന͍ݴޠͩͳͱࢥ͍·ͨ͠ɻ 9
  6. Let’s try Scala 2.7.3 4DBMBΛࢼͯ͠Έ·͠ΐ͏ɻͱɺ+BWBͰಈ͖·ͤ ΜͶɻTCU͔Β3&1-ͷىಈ͕Ͱ͖·ͤΜɻ $ java -version java

    version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) $ sbt ++2.7.3 console [info] 'compiler-interface' not yet compiled for Scala 2.7.3.final. Compiling... error: error while loading Consumer, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/function/Consumer.class)' is broken (bad constant pool tag 18 at byte 15) one error found 
 On JDK 8+, it fails to run the REPL… 10
  7. Let’s try 2.8.0.. 4DBMBͳΒͲ͏͔Τϥʔ಺༰ҧ͍·͕͢ ͪ͜Β΋࢒೦͔Βಈ࡞͠ͳ͍Α͏Ͱ͢ɻ $ sbt ++2.8.0 console [info]

    'compiler-interface' not yet compiled for Scala 2.8.0.final. Compiling... error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken (bad constant pool tag 18 at byte 10) error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken (bad constant pool tag 18 at byte 76) error: error while loading Arrays, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Arrays.class)' is broken (bad constant pool tag 18 at byte 765) error: error while loading Comparator, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Comparator.class)' is broken (bad constant pool tag 18 at byte 20) /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_6e129fa2/xsbt/ExtractAPI.scala:549: error: java.util.Comparator does not take type parameters private[this] val sortClasses = new Comparator[Symbol] { ^ 5 errors found 11 On JDK 8+, it fails to run the REPL…
  8. Let’s try 2.9.0.. ΋ಈ͔ͳ͍ͩͱ $ sbt ++2.9.0 console [info] 'compiler-interface'

    not yet compiled for Scala 2.9.0.final. Compiling... error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken (bad constant pool tag 18 at byte 10) error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken (bad constant pool tag 18 at byte 76) error: error while loading Arrays, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Arrays.class)' is broken (bad constant pool tag 18 at byte 765) error: error while loading Comparator, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Comparator.class)' is broken (bad constant pool tag 18 at byte 20) /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_a0025195/xsbt/ExtractAPI.scala:549: error: java.util.Comparator does not take type parameters private[this] val sortClasses = new Comparator[Symbol] { ^ 5 errors found 12 On JDK 8+, it fails to run the REPL…
  9. 2.10.2+ runs on JDK 8 ☺ ͔Β+BWBͰಈ͘Α͏ʹͳΓɺͦΕ·Ͱͷόʔδϣ ϯ͸ಈ͖·ͤΜͰͨ͠ɻࠓޙ΋ࣅͨΑ͏ͳ͜ͱ͸ى͖Δ͔΋ɻ $ sbt

    ++2.10.2 console [info] 'compiler-interface' not yet compiled for Scala 2.10.2. Compiling... [info] Compilation completed in 7.522 s [info] Starting scala interpreter... [info] Welcome to Scala version 2.10.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151). Type in expressions to have them evaluated. Type :help for more information. scala> 13 Scala 2.10.2+ works on Java 8 ☺
  10. 2.7.3 on JDK7.. ͱ͍͏͜ͱͰΛࢼ͚ͨ͠Ε͹+BWBΛ࢖Θͳ͍ͱ ͓΍ɺ͜Ε΋ಈ͔ͳ͍ͷͰ͠ΐ͏͔ʁʁ $ export JAVA_HOME=`/usr/libexec/java_home -v 1.7`

    $ java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) $ sbt ++2.7.3 console [info] 'compiler-interface' not yet compiled for Scala 2.7.3.final. Compiling... /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_7f1250db/ CompilerInterface.scala:9: error: value interactive is not a member of package scala.tools.nsc import scala.tools.nsc.interactive.RangePositions ^ (omitted) 42 errors found [error] (run-main) Error compiling sbt component 'compiler-interface' 
 14 Somehow, it fails to boot on Java7 ..
  11. 2.7.3 without sbt ☺ ࣮͸ݱߦTCUͰ౰࣌ͷ4DBMBΛಈ͔͢͜ͱ͕Ͱ͖·ͤΜɻ TCUΛ࢖Θͣʹىಈ͢Ε͹+BWBͰ౰વಈ࡞͠·͢ɻ $ export JAVA_HOME=`/usr/libexec/java_home -v

    1.7` $ java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) $ bin/scala Welcome to Scala version 2.7.3.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80). Type in expressions to have them evaluated. Type :help for more information. 15 “sbt console” supports only Scala 2.9.0+. You have to use the scala script instead.
  12. Scala 2.7 and beyond ೥ܦͬͯ4DBMB͸ͲΕ͘Β͍มΘͬͨͷͰ͠ΐ͏͔  ͜Ε͸Ҏ߱ͷϝδϟʔόʔδϣϯϦϦʔεͷ೥දͰ͢ɻ Scala 2.8 2010-07-15

    Scala 2.9 2011-05-12 Scala 2.10 2012-12-20 Scala 2.11 2014-05-20 Scala 2.12 2016-10-29 Scala 2.13.0-M3 2018-01-31 Scala 3.0 (with Dotty) ??? 16
  13. Missing parts (2.8, 2.9) Named and default arguments (2.8+) Package

    objects (2.8+) The App trait (2.9+) JavaConverters/JavaConversions (2.9+) ͱͰ͸͜ΕΒͷػೳ͕௥Ճ͞Ε·ͨ͠ɻ ౰࣌͸YͱYʹޓ׵ੑ͕͋Γ·ͤΜͰͨ͠ɻ 17
  14. Missing parts (2.10) 2.x binary compatibility (2.10+) Value classes (2.10+)

    Implicit classes (2.10+) String Interpolation (2.10+) Futures and Promises (2.10+) Dynamic (2.10+) Macros (2.10+) ͸λʔχϯάϙΠϯτͱͳΔϦϦʔεͰͨ͠ɻ όΠφϦޓ׵ੑͷվળ ଟ͘ͷػೳ͕௥Ճ͞Ε·ͨ͠ɻ 18
  15. Missing parts (2.11, 2.12) Blackbox/whitebox macros (2.11+) Quasiquotes (2.11+) Java

    8 SAM types compatibility (2.12.+) Using invokeDynamic for lambda (2.12+)  ͸ϚΫϩͷվળͱ಺෦తͳ࠷దԽ͕ओͰͨ͠ɻ ݴޠͷར༻ऀͱͯ͠͸৽͍͠ػೳ͸͋·Γ͋Γ·ͤΜɻ 19
  16. Moderate changes recently ͜ͷΑ͏ʹҎ߱ͷݴޠͷมߋ͸͋·Γଟ͋͘Γ·ͤΜɻ 044Λ࢝ΊΔλΠϛϯάͱͯ͠͸Α͍͔΋͠Ε·ͤΜͶɻ Scala 2.8 2010-07-15 Scala 2.9

    2011-05-12 Scala 2.10 2012-12-20 Scala 2.11 2014-05-20 Scala 2.12 2016-10-29 Scala 2.13.0-M3 2018-01-31 Scala 3.0 (with Dotty) ??? 20
  17. Why I Started My Project I needed a practical DB

    library in 2011 I was going to start using Scala for real projects Twitter’s querulous looked reasonable, but it supported MySQL only ScalaQuery looked hard to master for me For Fun / To learn new things At the time, I was new to both Scala and OSS I wanted to try something exciting by myself ೥౰࣌ɺ࣮༻తͳ%#MJCSBSZΛඞཁͱ͍ͯͨ͠ͷͱɺ ৽͍͜͠ͱΛ΍ͬͯΈ͍ͨͱ͍͏ؾ͔࣋ͪΒ࢝Ί·ͨ͠ɻ 22
  18. Non-practical Libraries In return for fancy APIs Not designed to

    solve real-world business problems New team members need long time to catch up Troubleshooting can be bothersome Insufficient information for debugging Frequently breaking API compatibilities Author suddenly stopped maintaining the library.. Χοί͍͍"1*͕ͩݱ࣮ͷ໰୊ʹରԠͰ͖ͳ͍ɾֶशίετ ߴ͍ɻσόοά͠ʹ͍͘ɻޓ׵ੑΛΑ่͘͢ɻϝϯς͕ಥવࢭ ·Δɻ͜ΕΒʹ֘౰͠ͳ͍͜ͱ͕ࢲͷʮ࣮༻తʯͷج४Ͱ͢ɻ 24
  19. Philosophy of ScalikeJDBC “Just write SQL and get things done!”

    Easy-to-understand Requires only Scala basics + JDBC basics No steep learning curve! (except for Scala ) Well-considered flexibility to deal with edge cases No breaking API changes as much as possible 4DBMJLF+%#$͕େࣄʹ͍ͯ͠Δ͜ͱɿ͙͢ʹཧղ͠࢖͑Δɺ ಛघέʔεʹରԠͰ͖Δɺޓ׵ੑΛۃྗ่͞ͳ͍ 25
  20. Tips for Continuation KISS (Keep It Simple, Stupid) Principle Keep

    it Contributors Friendly Avoiding Depending on Scala Libraries Knowledge on Important Factors Cross-building Binary-compatibilities Building a Community of Friends You’re not alone! ܧଓͰେࣄͳ͜ͱɿγϯϓϧʹอͭ͜ͱɺॏཁͳཁૉʢΫϩε ϏϧυόΠφϦޓ׵ʣͷཧղɺίϛϡχςΟͷܗ੒ 28
  21. Common reasons to do that Await their releases every Scala

    major upgrade If you depend on short-lived libraries.. When becoming unmaintained, you’ll have to fork and maintain them by yourself May bring binary compatibility issues 4DBMBNBKPSVQHSBEFͷ౓ʹSFMFBTF଴ͪɺ୹໋MJCSBSZΛ ࠷ѱGPSL͢ΔӋ໨ɺόΠφϦޓ׵ੑͷ໰୊ͳͲ͋Γ͑·͢ɻ 30
  22. Fetters for Cross-building So difficult to do cross-building while depending

    on other Scala libraries All depending Scala libraries must do the exactly same cross-building (unrealistic..) If you give cross-building high priority, you should consider to depend on not Scala libraries but Java libraries (or no dependencies..) ΫϩεϏϧυͰଞͷ4DBMBMJCSBSZ΁ͷґଘ͸૬౰ͳ଍͔ͤͰ ͢ɻશͯͷґଘ͕ΫϩεϏϧυΛ͠ͳ͚Ε͹੒ཱ͠·ͤΜɻ31
  23. Depending on Java libraries Depending on well-maintained Java libraries is

    a reasonable way to reduce your efforts No extra efforts when Scala major upgrade Stability (well-tested by large number of users) Downside of using Java libraries? Have to understand Java interoperability tips Have to completely encapsulate Java world which brings you lots of null +BWBMJCSBSZґଘ͸ѱ͘ͳ͍બ୒ࢶͰ৭ʑָʹͳΓ·͕͢ +BWBͳੈք؍ΛӅณ͢ΔͳͲͷέΞ͸ඞཁʹͳΔͰ͠ΐ͏ɻ 32
  24. Supported Versions 4DBMJLF+%#$͸!YVXFJLͷਚྗͷ͓͔͛΋͋Γ ΫϩεϏϧυΛ͔ͳΓؤு͍ͬͯΔϥΠϒϥϦͰ͢ɻ ScalikeJDBC Release date Scala binary versions

    sbt versions 1.0 2012-05 2.9.0 2.9.1 2.9.2 - 2.0 2014-05 2.10 2.11 0.13.x 3.0 2017-06 2.10 2.11 2.12 0.13.x 3.2 2018-01 2.10 2.11 2.12 0.13.x 1.x 34
  25. CI Build Configuration Properly configuring CI builds is the most

    important practice for cross-building If you don’t configure CI to check cross-building automatically, it’ll be very annoying to manually check the validity of pull requests $*ϏϧυΛద੾ʹઃఆͯ͠ӡ༻͢Δ͜ͱ͕࠷΋େࣄͰ͢ɻΫϩ εϏϧυΛؾʹ͠ͳ͕Β13Λड͚෇͚Δʹ͸ඞਢͰ͢ɻ 35
  26. src/main/scala-2.x Dealing with deprecated APIs between Scala major versions /

    3rd party libraries’ upgrades src/main/scala-{Scala binary version} ඇਪ঑ͱͳͬͨ"1*Λ࢖͍෼͚Δ৔߹TSDNBJOͷ഑Լʹ CJOBSZWFSTJPOຖʹҧ͏ιʔείʔυΛஔ͍ͯରॲ͠·͢ɻ36
  27. Are Java Libraries Safe? Java language itself is binary compatible

    between major versions Great efforts for backward compatibilities However, it’s not true for 3rd party Java libraries Even famous libraries sometimes break their binary compatibilities between releases (e.g) Jackson 2.x are basically binary incompatible +BWBϥΠϒϥϦͰόΠφϦޓ׵ੑΛؾʹͨ͜͠ͱ͕ͳ͍͋ͳ ͨ͸޾ӡͳਓͰ͢ɻඇޓ׵͸͠͹͠͹ൃੜ͍ͯ͠ΔͷͰ͢ɻ38
  28. How to Ensure It Use “MiMa” MiMa = Migration Manager

    lightbend/migration-manager Using it via the sbt plugin is convenient .J.BʢϛʔϚʣͱ͍͏πʔϧ͕όΠφϦඇޓ׵Λݕ஌ͯ͘͠ ΕΔͷͰ͜ΕΛTCUϓϥάΠϯܦ༝Ͱ࢖͏ͷ͕ҰൠతͰ͢ɻ 39
  29. When to Be Strict About It Up to the type

    of your library If you provide a low-level library that other libraries may depend on it you should be very careful As for directly used libraries / in-house libraries you don’t need to be so strict ࣾ಺΍ଞͷMJCSBSZ͕ґଘ͢ΔMJCSBSZΛఏڙ͠ͳ͍ͳΒɺඞ ͣ͠΋όΠφϦޓ׵ੑʹݫີʹऔΓ૊Ήඞཁੑ͸͋Γ·ͤΜɻ 40
  30. ScalikeJDBC’s Policy Bump major/minor version when introducing some binary incompatibilities

    All patch version releases are bin-compatible GitHub branch strategies Having a corresponding git branch for each bin- compatible minor version (e.g. 3.2.x, 3.1.x, .. ) master branch: the latest (3.3.x as of March 2018) ϙϦγʔɿόΠφϦඇޓ׵͸NBKPSNJOPSWFSɺQBUDIWFSؒ ͸όΠφϦޓ׵ΛอূɺHJUCSBODIΛNJOPS୯ҐͰӡ༻ 41
  31. OSS Open/Closed Principle Open/Closed Principle Open for extension, but closed

    for modification OSS Open/Closed Principle Open for contribution, Friendly for new contributors; easy-to-.. but closed for confusion (to be long-lived) CI / test suites which detect common mistakes Clearly stipulated rules (CONTRIBUTING.md) ߩݙʹ։͍͍ͯΔϓϩδΣΫτͷ৚݅ɿಈ͔͠΍͢͞ɺཧղ͠ ΍͢͞ɺ҆શͳมߋɺςετίʔυɺ໌ه͞Εͨϧʔϧ 46
  32. Just Keep Working on It Library maintainers must overcome upgrades

    Scala major upgrades / Cross-building sbt build settings upgrades I’m still taking time to adapt sbt 1.x .. It’s like a battle of patience.. Keep it simple, avoid hacky code/configuration Keep working on it at your own pace -JCSBSZϝϯςφ͸4DBMB΍TCUͷߋ৽ʹରԠ͠ଓ͚ͳ͚Ε ͹͍͚·ͤΜɻڽͬͨ͜ͱ͸ͤͣɺίπίπ࣋ٱઓͰ 47