Lightweight data ops and seamless app distribution with coursier

Talk given at ScalaSphere 2017, Kraków, 2017/03/03

http://get-coursier.io

Transcript

1. Lightweight data ops and seamless app distribution with coursier Alexandre Archambault @alxarchambault github.com/alexarchambault gif courtesy of @paulp
2. Outline 1. What’s coursier? 2. (New) stuff that can be done with it 3. Development / goals
3. Me contributed to shapeless coursier, jupyter-scala Teads.tv
4. What’s coursier?
5. Core Manage dependencies from Maven / Ivy repos Idiomatic Scala Separation of concerns resolution <> caching Hybrid Maven / Ivy model Allows to be used straightaway from sbt
6. Launcher Some inspiration from conscript More user-friendly conscript Slightly different concepts $ coursier launch com.lihaoyi:ammonite_2.12.1:0.8.2 coursier command: ~12 kB JAR
7. sbt plugin project/plugins.sbt addSbtPlugin("io.get-coursier" % "sbt-coursier" % "1.0.0-M15") In project/project/plugins.sbt too! - aka @paulp’s trick Limitations Doesn’t handle fetching sbt itself Doesn’t handle initial resolution fetching sbt-coursier
8. (New) stuff that can be done with it
9. sbt-shading libraryDependencies ++= Seq( "org" %% "b" % "0.1.0" % "shaded", "org" %% "c" % "0.1.0" ) enablePlugins(coursier.ShadingPlugin) shadingNamespace := "myapp.shaded" $ sbt shading:publish b a common c addSbtPlugin("io.get-coursier" % "sbt-shading" % "1.0.0-M15-3") Limitations Possibly does awful things with sbt configurations Shading extends Compile
10. Launching applications Scala Index $ coursier launch ammonite ammonite looked up though the Scala Index $ coursier launch com.lihaoyi:ammonite_2.12.1:0.8.2
11. $ coursier spark-submit \ -r https://nexus.foo.com \ com.foo::job:2.2.3 \ -- \ --num-executors 32 \ … Launching Spark applications \ -- \ arg1 arg2 …
12. Launching Spark applications Publish your jobs with sbt publish no assemblies no provided dependencies Ensure the cluster config is around (/etc/hadoop/conf…) No need of spark distributions coursier spark-submit … Automatically fetches all the required spark JARs (from Maven Central) Limitations Only YARN, only used against one type of clusters (AWS EMR ones) $ coursier spark-submit \ -r https://nexus.foo.com \ com.foo::job:2.2.3 \ -- \ --num-executors 32 \ … \ -- \ arg1 arg2 …
13. sbt launcher sbt: expects to be passed instances of stuff from its launcher-interface "components" reloading Fetch all that sbt needs Comes… csbt Fetch all that sbt needs
14. sbt launcher project/build.properties… or sbt.properties sbt.version=0.13.8 plugins = [ "io.get-coursier:sbt-coursier:1.0.0-M15-2" "io.get-coursier:sbt-shading:1.0.0-M15-2" "org.xerial.sbt:sbt-pack:0.8.2" "org.scala-js:sbt-scalajs:0.6.14" "com.jsuereth:sbt-pgp:1.0.0" "org.scoverage:sbt-scoverage:1.4.0" "org.tpolecat:tut-plugin:0.4.8" "com.typesafe.sbt:sbt-proguard:0.2.2" "com.typesafe:sbt-mima-plugin:0.1.13" ] dependencies = [ "org.scala-sbt:scripted-plugin:"${sbt.version} "org.anarres.jarjar:jarjar-core:1.0.1-coursier-SNAPSHOT" ] Scrap ~/.sbt, ~/.ivy2/cache, and project/! Do we really need meta-builds?
15. sbt launcher That looks cool but… Slower start once cache is full :-| Concurrency issues with zinc? $ sbt test:compile Help needed!
17. Wish: intellij launcher? $ coursier launch intellij:2016.3
18. Wish: better spark launch coursier spark-submit or notebook sessions Lots of JARs get moved around (Almost) always the same JARs! Ideally ship only Maven coordinates Spark containers re-use things from local cache
19. Maybe: coursier install $ coursier install scalafmt $ coursier install ammonite $ coursier install sbt Puts scalafmt / ammonite / sbt launchers in /usr/local/bin $ coursier update Makes sure all the launchers launch the latest versions
20. Development / goals
21. Why the CI will fail Scala 2.10, 2.11, 2.12 2 sbt plugins with scripted tests scala-js build and tests Windows CI (Appveyor) Integration tests with external stuff Binary compatibility checks (mima) Compiled with Java 7 or 8, parts must be fine with Java 6
22. Lower barrier to contributing sbt-microsite CONTRIBUTING.md
23. Questions?