Share
Embed
All I learned while working on a Scala OSS project for over six years #ScalaMatsuri
* http://2018.scalamatsuri.org/index_en.html
* https://www.youtube.com/watch?v=y7BxvT-Jm6w
In 2011, I started a Scala open source project named ScalikeJDBC. Thanks to many contributors, I am still working on the project after six years. In the meantime, the Scala community has been growing sharply, and the trends have been continuously changing. Despite the fact that the number of developers who work on OSS projects has increased, there are not many developers who have been working on a Scala project for several years. In this talk, I will share my experiences and pieces of knowledge through maintaining the OSS project for more than six years.
Kazuhiro Sera
March 17, 2018
Transcript
48 slides
-
All I learned while working on a Scala OSS project
for over six years Kazuhiro Sera @seratch 1 4DBMB044QSPKFDUΛҎ্ଓ͚ΔͨΊʹඞཁͳ͜ͱ ͱ͍͏ςʔϚͰɺྑ߂!TFSBUDI@KB͕͓͠͠·͢ɻ -
In 2011, I started a Scala open source project named
ScalikeJDBC. Thanks to many contributors, I am still working on the project after six years. In the meantime, the Scala community has been growing sharply, and the trends have been continuously changing. Despite the fact that the number of developers who work on OSS projects has increased, there are not many developers who have been working on a single Scala project for several years. In this talk, I will share my experiences and pieces of knowledge through maintaining a single OSS project for more than six years. ͔ΒऔΓΜͰ͖ͨ4DBMJLF+%#$Λ͡Ίͱ͢Δ 4DBMB044։ൃ͔ΒಘͨݟͳͲΛ͓͠͠·͢ɻ 2 -
Agenda Who Am I? My “Hello World” in Scala (2009)
ScalikeJDBC, My Long-lived Scala Project Tips for Continuation Avoiding Depending on Scala Libraries Cross-building Tips Ensuring binary-compatibility (MiMa) Building a Community of Friends ͜ͷΑ͏ͳ༰ʹ͍͓ͭͯ͠͠·͢ɻ্ڃऀͷํʹͪΐͬ ͱΓͳ͍༰͔͠Ε·ͤΜ͕ɺ͝༰ࣻΛɻ 3 -
Kazuhiro Sera @seratch Software Engineer, VPoE at SmartNews, Inc. We’re
a samurai sponsor Server-side software engineer (Scala, Java, Ruby) Love coding, open-source community A Scala enthusiast for 7+ years Creator of ScalikeJDBC, Skinny Framework One of the maintainers of Scalate, json4s, Scalatra 4NBSU/FXTͰಇ͍͍ͯ·͢ɻαʔόαΠυΤϯδχΞɺ 4DBMJLF+%#$ɺ4LJOOZͳͲͷ4DBMB044Λ͍ͬͯ·͢ɻ5 -
Star as an endorsement Increment the number of stargazers ͖ͳϥΠϒϥϦͷελʔΛJODSFNFOU͠·͠ΐ͏ɻ
ελʔ͕૿͑Δͷɺ͔ͳΓྭΈʹͳΔͷͰ͢ɹ 6 -
My “Hello World” in Scala In 2009, I was a
Java developer at the time Looking for a new one / unsatisfied with Groovy At the beginning, I tried with Scala 2.7.3 My first impression of Scala “IDE integration was still very poor. It’s a bit early to seriously use it at work.” Excited with hybrid paradigms, actors, concise syntax I started learning it seriously in the following year ॳΊͯͬͨͷͰ࣌+BWBΤϯδχΞͰͨ͠ɻ ࣄʹ·ͩ͑ͳ͍͚Ͳ໘ന͍ݴޠͩͳͱࢥ͍·ͨ͠ɻ 9 -
Let’s try Scala 2.7.3 4DBMBΛࢼͯ͠Έ·͠ΐ͏ɻͱɺ+BWBͰಈ͖·ͤ ΜͶɻTCU͔Β3&1-ͷىಈ͕Ͱ͖·ͤΜɻ $ java -version java
version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) $ sbt ++2.7.3 console [info] 'compiler-interface' not yet compiled for Scala 2.7.3.final. Compiling... error: error while loading Consumer, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/function/Consumer.class)' is broken (bad constant pool tag 18 at byte 15) one error found On JDK 8+, it fails to run the REPL… 10 -
Let’s try 2.8.0.. 4DBMBͳΒͲ͏͔Τϥʔ༰ҧ͍·͕͢ ͪ͜Β೦͔Βಈ࡞͠ͳ͍Α͏Ͱ͢ɻ $ sbt ++2.8.0 console [info]
'compiler-interface' not yet compiled for Scala 2.8.0.final. Compiling... error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken (bad constant pool tag 18 at byte 10) error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken (bad constant pool tag 18 at byte 76) error: error while loading Arrays, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Arrays.class)' is broken (bad constant pool tag 18 at byte 765) error: error while loading Comparator, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Comparator.class)' is broken (bad constant pool tag 18 at byte 20) /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_6e129fa2/xsbt/ExtractAPI.scala:549: error: java.util.Comparator does not take type parameters private[this] val sortClasses = new Comparator[Symbol] { ^ 5 errors found 11 On JDK 8+, it fails to run the REPL… -
Let’s try 2.9.0.. ಈ͔ͳ͍ͩͱ $ sbt ++2.9.0 console [info] 'compiler-interface'
not yet compiled for Scala 2.9.0.final. Compiling... error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken (bad constant pool tag 18 at byte 10) error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken (bad constant pool tag 18 at byte 76) error: error while loading Arrays, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Arrays.class)' is broken (bad constant pool tag 18 at byte 765) error: error while loading Comparator, class file '/Library/Java/JavaVirtualMachines/ jdk1.8.0_151.jdk/Contents/Home/jre/lib/rt.jar(java/util/Comparator.class)' is broken (bad constant pool tag 18 at byte 20) /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_a0025195/xsbt/ExtractAPI.scala:549: error: java.util.Comparator does not take type parameters private[this] val sortClasses = new Comparator[Symbol] { ^ 5 errors found 12 On JDK 8+, it fails to run the REPL… -
2.10.2+ runs on JDK 8 ☺ ͔Β+BWBͰಈ͘Α͏ʹͳΓɺͦΕ·Ͱͷόʔδϣ ϯಈ͖·ͤΜͰͨ͠ɻࠓޙࣅͨΑ͏ͳ͜ͱى͖Δ͔ɻ $ sbt
++2.10.2 console [info] 'compiler-interface' not yet compiled for Scala 2.10.2. Compiling... [info] Compilation completed in 7.522 s [info] Starting scala interpreter... [info] Welcome to Scala version 2.10.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151). Type in expressions to have them evaluated. Type :help for more information. scala> 13 Scala 2.10.2+ works on Java 8 ☺ -
2.7.3 on JDK7.. ͱ͍͏͜ͱͰΛࢼ͚ͨ͠Ε+BWBΛΘͳ͍ͱ ͓ɺ͜Εಈ͔ͳ͍ͷͰ͠ΐ͏͔ʁʁ $ export JAVA_HOME=`/usr/libexec/java_home -v 1.7`
$ java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) $ sbt ++2.7.3 console [info] 'compiler-interface' not yet compiled for Scala 2.7.3.final. Compiling... /var/folders/vl/bxx80tvs2gd8tzrtyd_b0p9r0000gp/T/sbt_7f1250db/ CompilerInterface.scala:9: error: value interactive is not a member of package scala.tools.nsc import scala.tools.nsc.interactive.RangePositions ^ (omitted) 42 errors found [error] (run-main) Error compiling sbt component 'compiler-interface' 14 Somehow, it fails to boot on Java7 .. -
2.7.3 without sbt ☺ ࣮ݱߦTCUͰ࣌ͷ4DBMBΛಈ͔͢͜ͱ͕Ͱ͖·ͤΜɻ TCUΛΘͣʹىಈ͢Ε+BWBͰવಈ࡞͠·͢ɻ $ export JAVA_HOME=`/usr/libexec/java_home -v
1.7` $ java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) $ bin/scala Welcome to Scala version 2.7.3.final (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80). Type in expressions to have them evaluated. Type :help for more information. 15 “sbt console” supports only Scala 2.9.0+. You have to use the scala script instead. -
Scala 2.7 and beyond ܦͬͯ4DBMBͲΕ͘Β͍มΘͬͨͷͰ͠ΐ͏͔ ͜ΕҎ߱ͷϝδϟʔόʔδϣϯϦϦʔεͷදͰ͢ɻ Scala 2.8 2010-07-15
Scala 2.9 2011-05-12 Scala 2.10 2012-12-20 Scala 2.11 2014-05-20 Scala 2.12 2016-10-29 Scala 2.13.0-M3 2018-01-31 Scala 3.0 (with Dotty) ??? 16 -
Missing parts (2.8, 2.9) Named and default arguments (2.8+) Package
objects (2.8+) The App trait (2.9+) JavaConverters/JavaConversions (2.9+) ͱͰ͜ΕΒͷػೳ͕Ճ͞Ε·ͨ͠ɻ ࣌YͱYʹޓੑ͕͋Γ·ͤΜͰͨ͠ɻ 17 -
Missing parts (2.10) 2.x binary compatibility (2.10+) Value classes (2.10+)
Implicit classes (2.10+) String Interpolation (2.10+) Futures and Promises (2.10+) Dynamic (2.10+) Macros (2.10+) λʔχϯάϙΠϯτͱͳΔϦϦʔεͰͨ͠ɻ όΠφϦޓੑͷվળ ଟ͘ͷػೳ͕Ճ͞Ε·ͨ͠ɻ 18 -
Missing parts (2.11, 2.12) Blackbox/whitebox macros (2.11+) Quasiquotes (2.11+) Java
8 SAM types compatibility (2.12.+) Using invokeDynamic for lambda (2.12+) ϚΫϩͷվળͱ෦తͳ࠷దԽ͕ओͰͨ͠ɻ ݴޠͷར༻ऀͱͯ͠৽͍͠ػೳ͋·Γ͋Γ·ͤΜɻ 19 -
Moderate changes recently ͜ͷΑ͏ʹҎ߱ͷݴޠͷมߋ͋·Γଟ͋͘Γ·ͤΜɻ 044Λ࢝ΊΔλΠϛϯάͱͯ͠Α͍͔͠Ε·ͤΜͶɻ Scala 2.8 2010-07-15 Scala 2.9
2011-05-12 Scala 2.10 2012-12-20 Scala 2.11 2014-05-20 Scala 2.12 2016-10-29 Scala 2.13.0-M3 2018-01-31 Scala 3.0 (with Dotty) ??? 20 -
Why I Started My Project I needed a practical DB
library in 2011 I was going to start using Scala for real projects Twitter’s querulous looked reasonable, but it supported MySQL only ScalaQuery looked hard to master for me For Fun / To learn new things At the time, I was new to both Scala and OSS I wanted to try something exciting by myself ࣌ɺ࣮༻తͳ%#MJCSBSZΛඞཁͱ͍ͯͨ͠ͷͱɺ ৽͍͜͠ͱΛͬͯΈ͍ͨͱ͍͏ؾ͔࣋ͪΒ࢝Ί·ͨ͠ɻ 22 -
Non-practical Libraries In return for fancy APIs Not designed to
solve real-world business problems New team members need long time to catch up Troubleshooting can be bothersome Insufficient information for debugging Frequently breaking API compatibilities Author suddenly stopped maintaining the library.. Χοί͍͍"1*͕ͩݱ࣮ͷʹରԠͰ͖ͳ͍ɾֶशίετ ߴ͍ɻσόοά͠ʹ͍͘ɻޓੑΛΑ่͘͢ɻϝϯς͕ಥવࢭ ·Δɻ͜ΕΒʹ֘͠ͳ͍͜ͱ͕ࢲͷʮ࣮༻తʯͷج४Ͱ͢ɻ 24 -
Philosophy of ScalikeJDBC “Just write SQL and get things done!”
Easy-to-understand Requires only Scala basics + JDBC basics No steep learning curve! (except for Scala ) Well-considered flexibility to deal with edge cases No breaking API changes as much as possible 4DBMJLF+%#$͕େࣄʹ͍ͯ͠Δ͜ͱɿ͙͢ʹཧղ͑͠Δɺ ಛघέʔεʹରԠͰ͖ΔɺޓੑΛۃྗ่͞ͳ͍ 25 -
More than 6 Years Ҏ্ಛఆͷMJCSBSZͷ։ൃΛܧଓ͖ͯ͠·ͨ͠ɻ ͜Ε4DBMBDPNNVOJUZͷதͰ͍ํͰ͠ΐ͏ɻ I’ve been working on
it for more than 6 years Scala 2.9 ~ 2.10 ~ 2.11 ~ 2.12 ~ 2.13 27 -
Tips for Continuation KISS (Keep It Simple, Stupid) Principle Keep
it Contributors Friendly Avoiding Depending on Scala Libraries Knowledge on Important Factors Cross-building Binary-compatibilities Building a Community of Friends You’re not alone! ܧଓͰେࣄͳ͜ͱɿγϯϓϧʹอͭ͜ͱɺॏཁͳཁૉʢΫϩε ϏϧυόΠφϦޓʣͷཧղɺίϛϡχςΟͷܗ 28 -
Common reasons to do that Await their releases every Scala
major upgrade If you depend on short-lived libraries.. When becoming unmaintained, you’ll have to fork and maintain them by yourself May bring binary compatibility issues 4DBMBNBKPSVQHSBEFͷʹSFMFBTFͪɺ໋MJCSBSZΛ ࠷ѱGPSL͢ΔӋɺόΠφϦޓੑͷͳͲ͋Γ͑·͢ɻ 30 -
Fetters for Cross-building So difficult to do cross-building while depending
on other Scala libraries All depending Scala libraries must do the exactly same cross-building (unrealistic..) If you give cross-building high priority, you should consider to depend on not Scala libraries but Java libraries (or no dependencies..) ΫϩεϏϧυͰଞͷ4DBMBMJCSBSZͷґଘ૬ͳ͔ͤͰ ͢ɻશͯͷґଘ͕ΫϩεϏϧυΛ͠ͳ͚Εཱ͠·ͤΜɻ31 -
Depending on Java libraries Depending on well-maintained Java libraries is
a reasonable way to reduce your efforts No extra efforts when Scala major upgrade Stability (well-tested by large number of users) Downside of using Java libraries? Have to understand Java interoperability tips Have to completely encapsulate Java world which brings you lots of null +BWBMJCSBSZґଘѱ͘ͳ͍બࢶͰ৭ʑָʹͳΓ·͕͢ +BWBͳੈք؍ΛӅณ͢ΔͳͲͷέΞඞཁʹͳΔͰ͠ΐ͏ɻ 32 -
Supported Versions 4DBMJLF+%#$!YVXFJLͷਚྗͷ͓͔͛͋Γ ΫϩεϏϧυΛ͔ͳΓؤு͍ͬͯΔϥΠϒϥϦͰ͢ɻ ScalikeJDBC Release date Scala binary versions
sbt versions 1.0 2012-05 2.9.0 2.9.1 2.9.2 - 2.0 2014-05 2.10 2.11 0.13.x 3.0 2017-06 2.10 2.11 2.12 0.13.x 3.2 2018-01 2.10 2.11 2.12 0.13.x 1.x 34 -
CI Build Configuration Properly configuring CI builds is the most
important practice for cross-building If you don’t configure CI to check cross-building automatically, it’ll be very annoying to manually check the validity of pull requests $*ϏϧυΛదʹઃఆͯ͠ӡ༻͢Δ͜ͱ͕࠷େࣄͰ͢ɻΫϩ εϏϧυΛؾʹ͠ͳ͕Β13Λड͚͚ΔʹඞਢͰ͢ɻ 35 -
src/main/scala-2.x Dealing with deprecated APIs between Scala major versions /
3rd party libraries’ upgrades src/main/scala-{Scala binary version} ඇਪͱͳͬͨ"1*Λ͍͚Δ߹TSDNBJOͷԼʹ CJOBSZWFSTJPOຖʹҧ͏ιʔείʔυΛஔ͍ͯରॲ͠·͢ɻ36 -
Are Java Libraries Safe? Java language itself is binary compatible
between major versions Great efforts for backward compatibilities However, it’s not true for 3rd party Java libraries Even famous libraries sometimes break their binary compatibilities between releases (e.g) Jackson 2.x are basically binary incompatible +BWBϥΠϒϥϦͰόΠφϦޓੑΛؾʹͨ͜͠ͱ͕ͳ͍͋ͳ ͨӡͳਓͰ͢ɻඇޓ͠͠ൃੜ͍ͯ͠ΔͷͰ͢ɻ38 -
How to Ensure It Use “MiMa” MiMa = Migration Manager
lightbend/migration-manager Using it via the sbt plugin is convenient .J.BʢϛʔϚʣͱ͍͏πʔϧ͕όΠφϦඇޓΛݕͯ͘͠ ΕΔͷͰ͜ΕΛTCUϓϥάΠϯܦ༝Ͱ͏ͷ͕ҰൠతͰ͢ɻ 39 -
When to Be Strict About It Up to the type
of your library If you provide a low-level library that other libraries may depend on it you should be very careful As for directly used libraries / in-house libraries you don’t need to be so strict ࣾଞͷMJCSBSZ͕ґଘ͢ΔMJCSBSZΛఏڙ͠ͳ͍ͳΒɺඞ ͣ͠όΠφϦޓੑʹݫີʹऔΓΉඞཁੑ͋Γ·ͤΜɻ 40 -
ScalikeJDBC’s Policy Bump major/minor version when introducing some binary incompatibilities
All patch version releases are bin-compatible GitHub branch strategies Having a corresponding git branch for each bin- compatible minor version (e.g. 3.2.x, 3.1.x, .. ) master branch: the latest (3.3.x as of March 2018) ϙϦγʔɿόΠφϦඇޓNBKPSNJOPSWFSɺQBUDIWFSؒ όΠφϦޓΛอূɺHJUCSBODIΛNJOPS୯ҐͰӡ༻ 41 -
OSS Open/Closed Principle Open/Closed Principle Open for extension, but closed
for modification OSS Open/Closed Principle Open for contribution, Friendly for new contributors; easy-to-.. but closed for confusion (to be long-lived) CI / test suites which detect common mistakes Clearly stipulated rules (CONTRIBUTING.md) ߩݙʹ։͍͍ͯΔϓϩδΣΫτͷ݅ɿಈ͔͢͠͞ɺཧղ͠ ͢͞ɺ҆શͳมߋɺςετίʔυɺ໌ه͞Εͨϧʔϧ 46 -
Just Keep Working on It Library maintainers must overcome upgrades
Scala major upgrades / Cross-building sbt build settings upgrades I’m still taking time to adapt sbt 1.x .. It’s like a battle of patience.. Keep it simple, avoid hacky code/configuration Keep working on it at your own pace -JCSBSZϝϯςφ4DBMBTCUͷߋ৽ʹରԠ͠ଓ͚ͳ͚Ε ͍͚·ͤΜɻڽͬͨ͜ͱͤͣɺίπίπ࣋ٱઓͰ 47