How do we Optimise Scala Build Times?

Slides from a talk I gave at Func Prog Sweden. First I discuss the incremental compilation algorithm used to compile Scala code by build tools such as sbt and Mill.

Next I explore other ways to optimise build times, such as build pipelining, and outline typing. I provide benchmarks and profiling data to demonstrate the effects of these modes of compilation.

Transcript

1. How do we optimise Scala Build times? • Twitter: @bishabosha • Email: scala.center@epfl.ch Jamie Thompson
2. What is Scala?
3. What is Scala?
4. What is Scala? >100 phases of transformation
5. Agenda Explaining the Scala Build What happens when you build a project? How do build tools optimise? Incremental compilation, Pipelined builds Takeaways Which steps can you take today to improve build times? What does the future hold? Can we add more innovations to speed up builds?
6. Research Open Source 8000+ Github repos 1,200,000+ artifacts Industry
7. • Created in 2016 at EPFL • Not for profit • Team of 10 people: ◦ administration ◦ communication ◦ engineering • Advisory Board: ◦ 2 Community Representatives ◦ 5 Companies
8. Fundraising Campaign
9. Explaining the Scala Build “Premature optimisation is the root of all evil” - Donald Knuth
10. Example: Small Server App 50 source files 10 library dependencies A.scala lib.jar 1 project directory webservice
11. Example: Small Server App ~/workspace » scala run webservice
12. 1. Fetch dependencies 2. Generate source files Example: Small Server App once per config update ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM)
13. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform slow!!! ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) once per config update Example: Small Server App
14. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App
15. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App Compiling project (Scala 3.3.1, JVM)
16. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App Compiling project (Scala 3.3.1, JVM)
17. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App Compiling project (Scala 3.3.1, JVM)
18. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App Compiling project (Scala 3.3.1, JVM)
19. 1. Fetch dependencies 2. Generate source files 3. Compile source files to runtime platform 4. Create a launcher and execute it fast ~/workspace » scala run webservice Compiling project (Scala 3.3.1, JVM) Compiled project (Scala 3.3.1, JVM) running server on localhost:8080 once per config update slow!!! Example: Small Server App Compiling project (Scala 3.3.1, JVM) * goal *
20. Incremental Compilation Compile only the changed definitions and their uses, compared to the previous compilation.
21. Incremental Compilation A.scala B.scala C.scala D.scala E.scala F.scala G.scala H.scala Instead of compiling all of these files…
22. Incremental Compilation First, detect A.scala has changed, compile it. C.scala D.scala E.scala F.scala G.scala H.scala B.scala A.scala
23. Incremental Compilation Next, compile the dependencies of any changed definitions A.scala C.scala D.scala E.scala F.scala G.scala H.scala B.scala
24. Incremental Compilation No more changes, compilation success A.scala C.scala D.scala E.scala F.scala G.scala H.scala B.scala ✅
25. Incremental Compilation Correct Performant compile everything that is necessary don’t compile more than necessary efficient invalidation The two unbreakable rules
26. Introducing Zinc Zinc is an incremental compiler for the Scala language, it maximizes correctness and performance with the name hashing algorithm.
27. Incremental Compilation Zinc scalac build tool “compile 50 scala files” A.scala A.scala A.scala lib.jar
28. Incremental Compilation Zinc scalac A.scala A.scala A.scala lib.jar “there is no cache, we must compile everything!” build tool
29. Incremental Compilation Zinc scalac A.scala A.scala A.scala lib.jar build tool
30. Incremental Compilation Zinc scalac A.scala A.scala A.scala lib.jar Analysis A.class B.class C.class … build tool
31. Incremental Compilation Zinc scalac A.scala A.scala A.scala lib.jar Analysis A.class B.class C.class … “compile succeeded” build tool
32. Incremental Compilation Zinc scalac A.scala A.scala A.scala lib.jar Analysis A.class B.class C.class … build tool
33. Incremental Compilation Zinc scalac “compile 50 scala files” A.scala A.scala A.scala lib.jar Analysis A.class B.class C.class … build tool
34. Incremental Compilation Zinc scalac lib.jar Analysis A.class B.class C.class … “Actually, only A.scala changed” A.scala A.scal a B.scala A.scala build tool
35. A.scala A.scal a B.scala Incremental Compilation Zinc scalac lib.jar Analysis A.scala A.class B.class C.class … build tool
36. Incremental Compilation Zinc scalac lib.jar Analysis A.scala A.scal a B.scala A.scala A.class build tool
37. Incremental Compilation Zinc scalac lib.jar Analysis “recompile D.scala and F.scala, they depend on definitions of A.scala that changed.” D.scala F.scala A.scala A.scal a A.scala A.class build tool
38. Incremental Compilation Zinc scalac lib.jar Analysis D.scala F.scala A.scala A.scal a A.scala A.class build tool
39. Incremental Compilation Zinc scalac lib.jar Analysis D.scala F.scala A.scala A.scal a A.scala D.class F.class build tool
40. Incremental Compilation Zinc scalac lib.jar Analysis No more changes detected! A.scala A.scala A.scala D.class F.class build tool
41. Incremental Compilation Zinc scalac lib.jar Analysis “Compile succeeded” A.scala A.scala A.scala D.class F.class build tool
42. Takeaways Tip No. 1 Use small files! A more granular dependency graph avoids unnecessary recompilation
43. Incremental Compilation Analysis APIs Dependencies tree data structure, representing signatures pairs of used-name, file of origin Stamps timestamps, file hashes
44. Incremental Compilation Analysis Stamps timestamps, file hashes “A.scala has different bytes than the last time I saw it, it should be recompiled.”
45. Incremental Compilation Analysis APIs APIs in A.scala: class A: def foo: Int def foo(x: Int): Int def foo(x: Int, y: String): Int val bar: Boolean tree data structure, representing signatures
46. Incremental Compilation Analysis APIs - Name Hashes class A defines names: foo = 0x8523a23 bar = 0x4d65e65 computed by Zinc aggregate all “foo” API aggregate all “bar” API
47. Incremental Compilation Analysis APIs - Name Hashes class A defines names: foo = 0xfb191c7 bar = 0x4d65e65 “Some definition A.foo has a changed API”
48. Incremental Compilation Analysis Dependencies pairs of used-name, class of origin class D inherits from class A class F uses name foo class F uses a member of class A “both class D and class F depend on changed API’s of class A!”
49. Incremental Compilation Analysis Summary The combination of stamps, name hashes and dependencies are sufficient to maximise performance and correctness of the name hashing algorithm.
50. Incremental Compilation Zinc scalac A.scala A.class A.tasty
51. Incremental Compilation Zinc A.scala genBCode pickler reads text into AST checks AST is valid, adds types writes AST to TASTy outputs class files A.class A.tasty typer parser
52. Incremental Compilation Zinc A.scala genBCode reads text into AST checks AST is valid, adds types outputs class files A.class A.tasty typer parser pickler writes AST to TASTy
53. Incremental Compilation Zinc A.scala genBCode A.class A.tasty typer parser pickler deps api callback callback record dependencies record API trees Analysis
54. Multi-project builds structure your project as a collection of modular libraries that cooperate to form a cohesive whole.
55. Multi-project Builds A.scala B.scala C.scala D.scala E.scala F.scala G.scala H.scala Split up the source files into modular groups
56. Multi-project Builds each group can be compiled in separate stages A.scala B.scala C.scala D.scala E.scala F.scala G.scala H.scala
57. Multi-project Builds before webservice
58. Multi-project Builds after server database auth model common
59. Multi-project Builds auth service model common database single threaded build multithreading off
60. Multi-project Builds on multithreaded build saved time! service common auth model common database multithreading
61. Takeaways Tip No. 2 With a more granular project graph, you can introduce parallelism. Split apps into smaller projects!
62. Multi-project Builds on can we do even better? saved time! service common auth model common database multithreading
63. Pipelined Builds When possible, begin downstream compilation before the upstream completes.
64. Pipelined Builds - Morgan Stanley OBT (optimus platform) - Bloop with Zinc fork (Jorge Vicente Cantero) - Experimental support today in sbt Prior Work
65. Pipelined Builds
66. Pipelined Builds Project B Project A Project C multithreaded, standard build off pipelining on multithreading
67. Pipelined Builds Project B Project C saved time! on pipelining on multithreading Project A pipelined multithreaded build no macros no macros
68. Pipelined Build Example off pipelining lampepfl/dotty
69. Pipelined Build Example NEW! off pipelining on pipelining lampepfl/dotty
70. Pipeline Compiler Zinc A.scala genBCode A.class A.tasty typer parser pickler deps api callback callback Analysis before
71. Zinc Pipeline Compiler A.scala A.class A.tasty typer parser pickler deps api saved time! rest of compile is now a background task write TASTy parallel thread TASTy files written early early.jar NEW!
72. Benchmarks Enough! show me the numbers!
73. Testing on MacBook Pro 2019 (i9 8-core 2.3GHz 16GB RAM) From cold sbt start: - clean; compile 2x to warm up - then take mean time of next 7 cycles. Benchmarks - pipelining
74. “clean compile” time & memory 308,829 LOC lichess-org/lila The key takeaway seems to be that you trade time overall for peak memory. 72s 6GB 55s 8.3GB 3.3.0 (standard) 3.3.2-SNAPSHOT (pipelined) 5600 lines/s Benchmarks - pipelining
75. Scaladex 31% improved 3.3.2-SNAPSHOT (pipelined) Time to finish “clean compile” time Your mileage may vary Other projects Benchmarks - pipelining
76. data server infra template webclient coreJS coreJVM Scaladex project layout Benchmarks - pipelining
77. facia faciaPress article applications sport discussion diagnostics admin identity commercial onward archive rss 77 common guardian frontend project layout preview Benchmarks - pipelining
78. 78 pipelining on multithreading facia common faciaPress article preview applications … off Benchmarks - pipelining
79. 79 pipelining on multithreading facia common faciaPress … preview on article Benchmarks - pipelining is there another way?
80. Takeaways Tip No. 3 Thread starvation, duplicate work Don’t make projects too small!
81. outline compile Compile in two passes, first quick to signatures, and a parallel second pass.
82. Standard Compile inlining lowering, erasure backend deps, api, pickler type checking pipelining start
83. Outline Compile inlining lowering, erasure backend pipelining start (full compile) outline type checking api, pickler
84. Outline Compile NEW! inlining second type check pipelining start (incremental compile) lowering, erasure backend pipelining start (full compile) outline type checking api, pickler deps
85. Outline Compile saved time! NEW! outline type checking api, pickler
86. Outline Build Example off outlining lampepfl/dotty
87. Outline Build Example off outlining NEW! on outlining lampepfl/dotty
88. Outline Build Example 158,000 LOC 30s 1.54x faster! we could still do better… 19.5s 3.3.2-SNAPSHOT (2-pass, 3 workers) 3.3.1 (single pass) 8119 lines/s 5277 lines/s lampepfl/dotty
89. Takeaways Tip No. 4 The faster the better, right?! Try out pipelining!