Simple and Safe Pickling of Closures in Scala 3

Transcript

1. Simple and Safe Pickling of Closures in Scala 3 Philipp Haller Joint work with Jonas Spenger KTH Royal Institute of Technology, Stockholm, Sweden Scala Winter Meetup @ Spotify, January 14, 2026, Stockholm, Sweden 1
2. About the Authors 2 Philipp Haller @philippkhaller (X) @phaller (GitHub) linkedin.com/in/hallerp https://people.kth.se/~phaller/ Jonas Spenger @jspenger (GitHub) https://jspenger.github.io/ • Defended his PhD on Dec 11, 2025 at KTH Royal Institute of Technology, Stockholm, Sweden (thesis) • Durable computation (dataflow streaming, futures, actors) • Associate Professor at KTH Royal Institute of Technology, Stockholm, Sweden • Concurrency (actors, futures/async, joins) • Distributed programming • Type systems (capabilities, uniqueness)
3. Closures and Environments 3 • Note: threshold is not defined within the blue anonymous function! • Thus: threshold is a called a free variable of the blue anonymous function • Original definition of the term closure: An anonymous function “whose open bindings (free variables) have been closed by the lexical environment” (Peter J. Landin, 1964)" val numbers = List(6, 3, 9, 2, 4) def below(threshold: Int) = numbers.filter(num => num < threshold) assert(below(5) == List(3, 2, 4)) P. J. Landin: The Mechanical Evaluation of Expressions. Comput. J. 6(4): 308-320 (1964) https://doi.org/10.1093/comjnl/6.4.308 The blue lambda refers to threshold in its lexical context
4. object Global { val fld = 5 } val fun = { (num: Int) => num < Global.fld } assert(numbers.filter(fun), List(3, 2, 4)) What is Captured and What isn't? 4 object Global { val fld = 5 } val fun = { (num: Int) => num < Global.fld } assert(numbers.filter(fun), List(3, 2, 4)) val numbers = List(6, 3, 9, 2, 4) def mkFilter(threshold: Int) = (num: Int) => num < threshold val fun = mkFilter(5) assert(numbers.filter(fun), List(3, 2, 4)) class Filter(threshold: Int) extends Int => Bool: def apply(num: Int) = num < threshold val fun = Filter(5) assert(numbers.filter(fun), List(3, 2, 4)) Top-level object captured "captured" Not captured
5. class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4, 5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 5 Using Apache Spark™ Exception in thread "main" org.apache.spark.SparkException: Task not serializable ...
6. class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4, 5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 6 • distData is a distributed data set → each remote worker has a piece of the data • map sends its argument closure to each worker → argument closure must be serialized Uses transform method from enclosing scope
7. class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4, 5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => this.transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 7 • distData is a distributed data set → each remote worker has a piece of the data • map sends its argument closure to each worker → argument closure must be serialized Actually: closure captures this! Closure is sent to remote workers → this must be serialized The type of this is SparkExample which is not serializable, hence... Exception in thread "main" org.apache.spark.SparkException: Task not serializable ...
8. Example: Concurrency 8 val customerData: mutable.Map[Int, CustomerInfo] = ... def averageAge(customers: List[Customer]) = Future { val infos = customers.flatMap { c => customerData.get(c.customerNo) match case Some(info) => List(info) case None => List() } val sumAges = infos.foldLeft(0)(_ + _.age).toFloat if (infos.nonEmpty) sumAges / infos.size else 0.0f } Possible data race! Mutable map Async execution
9. Problematic Uses of Closures • Using closures in distributed settings is a safety risk ◦ Example: serializing closures can result in runtime errors (e.g., java.io.NotSerializableException on the JVM) • Using closures in concurrent settings is a safety risk ◦ Example: running a closure on a concurrent thread could cause a data race if a captured variable refers to a shared mutable object • Anything else? ◦ Send a closure from a Scala.js frontend to a Scala/JVM backend? ◦ Requires portable pickling/serialization! 9
10. Short History of Spores • Goals of Spores: ◦ Safety checking of closures ◦ Increased flexibility, e.g., portable serialization • Main idea: “Spore” = a special kind of closure that ◦ consists of (1) a capture-free function and (2) an explicit environment ◦ where the environment is constrained using type classes • Spores were first proposed in SIP-21 in 2013 and presented at ECOOP'14 10 Miller, Haller, and Odersky. Spores: a type-based foundation for closures in the age of concurrency and distribution. ECOOP 2014 (Google Scholar: 57 citations) https://doi.org/10.1007/978-3-662-44202-9_13
11. Spores3 • A new take on Spores for Scala 3 • Uses a new approach for type-class-based pickling • A simpler and more robust implementation • Talk and paper (open access): 11 Philipp Haller: Enhancing closures in Scala 3 with Spores3. ACM SIGPLAN Scala Symposium, Berlin, Germany, June 2022. https://doi.org/10.1145/3550198.3550428 Talk at Strange Loop 2022: video, slides
12. This Talk • Spores are now even more flexible in the new version 0.2 ◦ Partial application of Spores ◦ Support for context functions ◦ ... computation on pickled data made simple! 12
13. The Spore Factory // Spore without environment val spore = Spore { (x: Int) => x + 1 } // with optional type annotation val spore: Spore[Int => Int] = Spore { x => x + 1 } // Spore with environment val num = 12 val spore: Spore[Int => Int] = Spore(num) { (env: Int) => (x: Int) => x + env } ^^^ ^^^ ^^^ env is replaced with the value of num in body 13
14. // Spore with context environment val num = 12 val spore: Spore[Int => Int] = Spore.applyWithCtx(num) { (env: Int) ?=> (x: Int) => x + summon[Int]} The Spore Factory 14
15. Spore Auto Capture // Automatically captures environment variables // y, z captured* val y = 12 val z = “World” val spore: Spore[Int => String] = Spore.auto { (x: Int) => (x + y).toString() + z } ^ ^ 15 *if not top-level
16. Unwrapping / Applying a Spore 16 // unwraps the value in the spore val spore: Spore[Int => Int] = Spore { ... } spore.unwrap().apply(12) // or, using imported implicit conversion import spores.default.given spore(12) // Spore[Int => Int] is converted to Int => Int
17. val y = 12 val spore: Spore[Int => Int] = Spore { (x: Int) => x + y } The Spore Factory: Compile-Time Checks 17 More examples here and here
18. val y = 12 val spore: Spore[Int => Int] = Spore { (x: Int) => x + y } ^^^ Error: Invalid capture of variable `y`. The Spore Factory: Compile-Time Checks 18 More examples here and here
19. class Foo: val x = 12 def spore = Spore.apply { () => 42 * x } The Spore Factory: Compile-Time Checks 19 More examples here and here
20. class Foo: val x = 12 def spore = Spore.apply { () => 42 * x } ^^^ (i.e., this.x) Error: Invalid capture of `this` from class Foo. The Spore Factory: Compile-Time Checks 20 More examples here and here
21. // What about this example? class Outer: class Foo def spore = Spore.apply { () => class Bar extends Foo new Bar() } The Spore Factory: Compile-Time Checks 21
22. // What about this example? class Outer: class Foo def spore = Spore.apply { () => class Bar extends Foo ^^^ new Bar() } Error: Invalid capture of variable `Outer.Foo`. The Spore Factory: Compile-Time Checks 22
23. // What about this example? ... class Bar def foo(using Bar): Spore[Bar] = Spore.apply { summon } The Spore Factory: Compile-Time Checks 23
24. // What about this example? ... class Bar def foo(using Bar): Spore[Bar] = Spore.apply { summon } ^^^^^^ Error: Invalid capture of variable `x$1`. The Spore Factory: Compile-Time Checks 24 Captured context parameter!
25. • Problematic capturing is prevented • This ensures that it is always safe to pickle / unpickle a Spore • Pickling example: import upickle.default.* val spore: Spore[Int => Int] = ... val pickled: String = write(spore) // JSON String val unpickled = read[Spore[Int => Int]](pickled) unpickled: Spore[Int => Int] Safe Pickling of Spores 25 ‘write’ to JSON String ‘read’ from JSON String
26. Macro for Checking Captures • Macros.checkBodyExpr(bodyExpr) ◦ Collect all symbols which might be captured ▪ Ident, This, ClassDefs, New, etc. ▪ Ignore some special cases, e.g. ignore U in new T[U]() ◦ Remove symbols owned by the closure (parameters, etc.) ◦ Remove top-level symbols ◦ Remaining symbols are captured 26
27. • Functions are capture free ◦ Checked by macros • Captured environment variables ◦ By partial application Spores3 Approach 27
28. // Apply (partially) a Spore to a value val spore: Spore[Int => Int => String] spore.withEnv(12): Spore[Int => String] ^^ requires a given: Spore[ReadWriter[Int]] // Apply (partially) a Spore to a Spore val spore: Spore[Int => Int => String] = ... val spore12: Spore[Int] = Spore { 12 } spore.withEnv2(spore12): Spore[Int => String] ^^^^^^^ no given needed Partial Application 28
29. // Also works for context parameters val spore2: Spore[Int ?=> Int => String] spore2.withCtx(12): Spore[Int => String] ... spore2.withCtx2(spore12): Spore[Int => String] Partial Application 29
30. Creating a `given Spore[ReadWriter[T]]` 30 // Works if the summoned ReadWriter is top-level given Spore[ReadWriter[Int]] = Spore { summon } // Works for all top-level ReadWriters inline given [T: ReadWriter]: Spore[ReadWriter[T]] = Spore { summon[ReadWriter[T]] }
31. • Use Spores for serializing actor behaviors • Periodically serialize execution state • Recover state after crash • https://github.com/jspenger/durable-actor (WIP) ◦ API inspired by Akka Typed Example: Durable Actors 31
32. Preliminary Microbenchmarks • Compare Spores3* to Java** • Pickling performance: ◦ Speedup between 1.3x and 5.4x ◦ Size reduction between 2.5x and 9.2x • More details in the backup slides 32 *Serialized Spore by upickle writeBinary **Serialized by java.io.ObjectInputStream / java.io.ObjectOutputStream
33. Summary • Spores3: making closures in Scala safer and more flexible ◦ Spores are always safe to pickle / unpickle ◦ Captured environment is checked ◦ Version 0.2 has the latest improvements: https://github.com/phaller/spores3 • Plans for version 0.3: ◦ More flexible Spore type: Spore[F[_], +T] ▪ Generic over the evidence type (so far fixed to ReadWriter[_]) 33
34. Summary • Spores3: making closures in Scala safer and more flexible ◦ Spores are always safe to pickle / unpickle ◦ Captured environment is checked ◦ Version 0.2 has the latest improvements: https://github.com/phaller/spores3 34 Philipp Haller @philippkhaller (X) @phaller (GitHub) linkedin.com/in/hallerp https://people.kth.se/~phaller/
35. Preliminary Microbenchmarks • Compare Spores3* to Java** • Function1: x => () => x + 1 • Function2: (x => () => x + 1).apply(7) • Function3: (list => () => list.sum).apply(List(1, …, 10)) • Function4: (pred => list => list.filter(pred)).apply(x => x % 2 == 0) 35 *Serialized Spore by upickle writeBinary **Serialized by java.io.ObjectInputStream / java.io.ObjectOutputStream
36. Operations per Second • Measure time to serialize and deserialize with JMH ◦ MOPS (million operations per second) • Function1: Spores3: 0.70 Java: 0.13 (MOPS) • Function2: Spores3: 0.16 Java: 0.12 • Function3: Spores3: 0.11 Java: 0.050 • Function4: Spores3: 0.22 Java: 0.10 • Speedup between 1.3x and 5.4x 36
37. • Serialize to Array[Byte] ◦ Measure size in Byte • Function1: Spores3: 67 Java: 622 (Byte) • Function2: Spores3: 223 Java: 562 • Function3: Spores3: 359 Java: 1166 • Function4: Spores3: 186 Java: 862 • Size reduction between 2.5x and 9.2x Serialization Size 37