もう一つのビルドツール mill で作る Docker イメージ / Build docker image with mill the yet another build tool

for Scala Aki Matsuri (https://scala-aki-matsuri.connpass.com/event/142817/)

Transcript

1. ΋͏ҰͭͷϏϧυπʔϧ
   mill Ͱ࡞Δ Docker Πϝʔδ
   Ѩ઒ ߞ࢘ @atty303
   AI tech studio / CyberAgent, Inc.
   © 2019 CyberAgent, Inc.
   1/33
   

   View Slide

2. Ѩ઒ ߞ࢘
   ιϑτ΢ΣΞΤϯδχΞ
   @CyberAgent, Inc.
   • ۀ຿ྺ 14 ೥ / Scala ྺ 4 ೥
   • ήʔϜ͕޷͖ (Steam ϝΠϯ)
   ීஈ͸ઃܭͨ͠ΓόοΫΤϯυ
   Λ΍Γͭͭɺͨ·ʹ Web ϑϩϯ
   τΤϯυ΍ Android Λ৮ͬͨΓ
   ΋͠·͢ɻ
   © 2019 CyberAgent, Inc. 2/33
   

   View Slide

3. !
   Ϗϧυπʔϧ
   © 2019 CyberAgent, Inc. 3/33
   

   View Slide

4. 2018 Scala Developer Survey 1
   Which build tool are you currently using? 2
   2 https://typesafe.co1.qualtrics.com/results/public/
   dHlwZXNhZmUtVVJfNlB4cWNSMXdub0liVExmLTVhZjMwZDc4MjAzMGVkMDAxNDhkOTc4
   OA==
   1 https://www.scala-lang.org/news/survey-2018.html
   © 2019 CyberAgent, Inc. 4/33
   

   View Slide

5. mill
   © 2019 CyberAgent, Inc. 5/33
   

   View Slide

6. Your shiny new Java/Scala build
   tool!
   • https://github.com/lihaoyi/mill
   • Made by Li Hayoi
   • Ammonite / fastparse / utest / upickle
   © 2019 CyberAgent, Inc. 6/33
   

   View Slide

7. యܕతͳMillϏϧυઃఆ
   // build.sc
   import mill._, scalalib._
   object foo extends ScalaModule {
   def scalaVersion = T("2.12.10")
   def scalacOptions = T(Seq(
   "-Ypartial-unification",
   ))
   def ivyDeps = T(Agg(
   ivy"com.lihaoyi::upickle:0.5.1",
   ))
   }
   © 2019 CyberAgent, Inc. 7/33
   

   View Slide

8. IntelliJ IDEA ΁ͷΠϯϙʔτ
   ૢ࡞ sbt mill
   ॳճΠϯϙʔτ 04:07 01:45
   ࠶Πϯϙʔτ 02:25 00:23
   • ಉ͡ MacBook
   • ϓϩδΣΫτ਺͸ 28
   © 2019 CyberAgent, Inc. 8/33
   

   View Slide

9. mill ͷ֦ு
   import mill._, scalalib._
   trait ExampleModule extends Module {
   def hello = T { "Hello" }
   def firstName = T { "Koji" }
   def lastName = T { "AGAWA" }
   def fullName = T { s"${firstName()} ${lastName()}" }
   def message = T { s"${hello()}, ${fullName()}" }
   }
   object example extends ScalaModule with ExampleModule {
   def scalaVersion = "2.13.0"
   }
   © 2019 CyberAgent, Inc. 9/33
   

   View Slide

10. λεΫͷ࣮ߦ
    $ ./mill example.message
    $
    ஋Λฦ͍ͯ͠Δ͚ͩͳͷͰԿ΋ى͜Βͳ͍
    © 2019 CyberAgent, Inc. 10/33
    

    View Slide

11. λεΫͷ஋Λग़ྗ
    $ ./mill show example.message
    [1/1] show
    "Hello, Koji AGAWA"
    $
    © 2019 CyberAgent, Inc. 11/33
    

    View Slide

12. λεΫͷґଘؔ܎
    $ ./mill visualize example._ 3
    $ open out/visualize/out.png
    3 ExampleModule ʹؔ܎͢Δ෦෼ͷΈൈਮ
    © 2019 CyberAgent, Inc. 12/33
    

    View Slide

13. Docker
    © 2019 CyberAgent, Inc. 13/33
    

    View Slide

14. Docker
    • શͯͷΞϓϦ͸ Docker Ͱ؅ཧ͍ͯ͠Δ
    • sbt-native-packager ͷΑ͏ʹ Docker ΠϝʔδΛ࡞
    Γ͍ͨ
    © 2019 CyberAgent, Inc. 14/33
    

    View Slide

15. contrib.docker.DockerModule 4
    4 http://www.lihaoyi.com/mill/page/contrib-modules.html#docker
    © 2019 CyberAgent, Inc. 15/33
    

    View Slide

16. ͋Δ΍Μ
    © 2019 CyberAgent, Inc. 16/33
    

    View Slide

17. ίʔυྔগͳ͍
    package mill
    package contrib.docker
    import mill.scalalib.JavaModule
    import os.Shellable.IterableShellable
    import scala.collection.immutable._
    trait DockerModule { outer: JavaModule =>
    trait DockerConfig extends mill.Module {
    /**
    * Tags that should be applied to the built image
    * In the standard registry/repository:tag format
    */
    def tags: T[Seq[String]] = T(List(outer.artifactName()))
    def labels: T[Map[String, String]] = Map.empty[String, String]
    def baseImage: T[String] = "gcr.io/distroless/java:latest"
    def pullBaseImage: T[Boolean] = T(baseImage().endsWith(":latest"))
    private def baseImageCacheBuster: T[(Boolean, Double)] = T.input {
    val pull = pullBaseImage()
    if(pull) (pull, Math.random()) else (pull, 0d)
    }
    def dockerfile: T[String] = T {
    val jarName = assembly().path.last
    val labelRhs = labels()
    .map { case (k, v) =>
    val lineBrokenValue = v
    .replace("\r\n", "\\\r\n")
    .replace("\n", "\\\n")
    .replace("\r", "\\\r")
    s""""$k"="$lineBrokenValue""""
    }
    .mkString(" ")
    val labelLine = if(labels().isEmpty) "" else s"LABEL $labelRhs"
    s"""
    |FROM ${baseImage()}
    |$labelLine
    |COPY $jarName /$jarName
    |ENTRYPOINT ["java", "-jar", "/$jarName"]
    """.stripMargin
    }
    final def build = T {
    val dest = T.ctx().dest
    val asmPath = outer.assembly().path
    os.copy(asmPath, dest / asmPath.last)
    os.write(dest / "Dockerfile", dockerfile())
    val log = T.ctx().log
    val tagArgs = tags().flatMap(t => List("-t", t))
    val (pull, _) = baseImageCacheBuster()
    val pullLatestBase = IterableShellable(if(pull) Some("--pull") else None)
    val result = os
    .proc("docker", "build", tagArgs, pullLatestBase, dest)
    .call(stdout = os.Inherit, stderr = os.Inherit)
    log.info(s"Docker build completed ${if(result.exitCode == 0) "successfully" else "unsuccessfully"} with ${result.exitCode}")
    tags()
    }
    final def push() = T.command {
    val tags = build()
    tags.foreach(t => os.proc("docker", "push", t).call(stdout = os.Inherit, stderr = os.Inherit))
    }
    }
    }
    © 2019 CyberAgent, Inc. 17/33
    

    View Slide

18. ͔͠͠…
    final def build = T {
    val dest = T.ctx().dest
    val asmPath = outer.assembly().path
    os.copy(asmPath, dest / asmPath.last)
    os.write(dest / "Dockerfile", dockerfile())
    © 2019 CyberAgent, Inc. 18/33
    

    View Slide

19. assembly…
    ஗͍ΑͶ…
    © 2019 CyberAgent, Inc. 19/33
    

    View Slide

20. Jib Core 5
    Java library for building containers
    by Google
    5 https://github.com/GoogleContainerTools/jib/tree/master/jib-core
    © 2019 CyberAgent, Inc. 20/33
    

    View Slide

21. Jib Core
    • Docker daemon ෆཁ !!!
    © 2019 CyberAgent, Inc. 21/33
    

    View Slide

22. JavaContainerBuilder
    • Java ΞϓϦ޲͚ͷϏϧμʔΛఏڙ !!!
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 22/33
    

    View Slide

23. JavaContainerBuilder
    JavaContainerBuilder
    .from("gcr.io/distroless/java:11")
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 23/33
    

    View Slide

24. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass("glaze.progad.driver.router.Boot")
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 24/33
    

    View Slide

25. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(Seq("~/.cache/coursier/...").asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 25/33
    

    View Slide

26. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(Seq("out/progad/driver/router/out.jar").asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 26/33
    

    View Slide

27. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(Seq("-XX:MaxRAMPercentage=75").asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 27/33
    

    View Slide

28. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(Seq("progad-router:latest").asJava)
    .containerize(cont)
    © 2019 CyberAgent, Inc. 28/33
    

    View Slide

29. JavaContainerBuilder
    JavaContainerBuilder
    .from(baseImage)
    .setMainClass(mainClass)
    .addDependencies(deps.asJava)
    .addProjectDependencies(projectDeps.asJava)
    .addJvmFlags(jvmFlags.asJava)
    .toContainerBuilder
    .setLabels(labels.asJava)
    .containerize(containizer)
    © 2019 CyberAgent, Inc. 29/33
    

    View Slide

30. Containizer
    • ϩʔΧϧͷ Docker daemon ΁Πϯϙʔτ
    Containerizer.to(DockerDaemonImage.named(imageName))
    • ϨδετϦ΁௚઀ϓογϡ
    Containerizer.to(RegistryImage.named(imageName))
    © 2019 CyberAgent, Inc. 30/33
    

    View Slide

31. ޾ͤ
    © 2019 CyberAgent, Inc. 31/33
    

    View Slide

32. ੒Ռ෺
    • Jib Core ͷ͓ӄͰ mill ͷϞδϡʔϧԽ͸؆୯
    • ۀ຿ϓϩδΣΫτʹಋೖ
    • ScalaPB ϞδϡʔϧΛ sbt ൛ʹ͚ۙͮͨΓ
    • GenIdea Ϟδϡʔϧ͕ ScalaPB ͷੜ੒ιʔεΛ
    ೝࣝ͢ΔΑ͏ʹ
    • ޙ೔ Pull Request ͠·͢
    © 2019 CyberAgent, Inc. 32/33
    

    View Slide

33. ΈΜͳ΋ mill Λ
    ࢖ͬͯΈΑ͏
    © 2019 CyberAgent, Inc. 33/33
    

    View Slide