Share
Embed
Transcript
37 slides
-
Simple and Safe Pickling of Closures in Scala 3 Philipp
Haller Joint work with Jonas Spenger KTH Royal Institute of Technology, Stockholm, Sweden Scala Winter Meetup @ Spotify, January 14, 2026, Stockholm, Sweden 1 -
About the Authors 2 Philipp Haller @philippkhaller (X) @phaller (GitHub)
linkedin.com/in/hallerp https://people.kth.se/~phaller/ Jonas Spenger @jspenger (GitHub) https://jspenger.github.io/ • Defended his PhD on Dec 11, 2025 at KTH Royal Institute of Technology, Stockholm, Sweden (thesis) • Durable computation (dataflow streaming, futures, actors) • Associate Professor at KTH Royal Institute of Technology, Stockholm, Sweden • Concurrency (actors, futures/async, joins) • Distributed programming • Type systems (capabilities, uniqueness) -
Closures and Environments 3 • Note: threshold is not defined
within the blue anonymous function! • Thus: threshold is a called a free variable of the blue anonymous function • Original definition of the term closure: An anonymous function “whose open bindings (free variables) have been closed by the lexical environment” (Peter J. Landin, 1964)" val numbers = List(6, 3, 9, 2, 4) def below(threshold: Int) = numbers.filter(num => num < threshold) assert(below(5) == List(3, 2, 4)) P. J. Landin: The Mechanical Evaluation of Expressions. Comput. J. 6(4): 308-320 (1964) https://doi.org/10.1093/comjnl/6.4.308 The blue lambda refers to threshold in its lexical context -
object Global { val fld = 5 } val fun
= { (num: Int) => num < Global.fld } assert(numbers.filter(fun), List(3, 2, 4)) What is Captured and What isn't? 4 object Global { val fld = 5 } val fun = { (num: Int) => num < Global.fld } assert(numbers.filter(fun), List(3, 2, 4)) val numbers = List(6, 3, 9, 2, 4) def mkFilter(threshold: Int) = (num: Int) => num < threshold val fun = mkFilter(5) assert(numbers.filter(fun), List(3, 2, 4)) class Filter(threshold: Int) extends Int => Bool: def apply(num: Int) = num < threshold val fun = Filter(5) assert(numbers.filter(fun), List(3, 2, 4)) Top-level object captured "captured" Not captured -
class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4,
5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 5 Using Apache Spark™ Exception in thread "main" org.apache.spark.SparkException: Task not serializable ... -
class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4,
5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 6 • distData is a distributed data set → each remote worker has a piece of the data • map sends its argument closure to each worker → argument closure must be serialized Uses transform method from enclosing scope -
class SparkExample { val distData = sc.parallelize(Array(1, 2, 3, 4,
5)) def transform(x: Int): Int = x+1 def test(): Unit = { val transformed = distData.map(elem => this.transform(elem)) transformed.collect().foreach(elem => println(elem)) } } Trouble in Paradise 7 • distData is a distributed data set → each remote worker has a piece of the data • map sends its argument closure to each worker → argument closure must be serialized Actually: closure captures this! Closure is sent to remote workers → this must be serialized The type of this is SparkExample which is not serializable, hence... Exception in thread "main" org.apache.spark.SparkException: Task not serializable ... -
Example: Concurrency 8 val customerData: mutable.Map[Int, CustomerInfo] = ... def
averageAge(customers: List[Customer]) = Future { val infos = customers.flatMap { c => customerData.get(c.customerNo) match case Some(info) => List(info) case None => List() } val sumAges = infos.foldLeft(0)(_ + _.age).toFloat if (infos.nonEmpty) sumAges / infos.size else 0.0f } Possible data race! Mutable map Async execution -
Problematic Uses of Closures • Using closures in distributed settings
is a safety risk ◦ Example: serializing closures can result in runtime errors (e.g., java.io.NotSerializableException on the JVM) • Using closures in concurrent settings is a safety risk ◦ Example: running a closure on a concurrent thread could cause a data race if a captured variable refers to a shared mutable object • Anything else? ◦ Send a closure from a Scala.js frontend to a Scala/JVM backend? ◦ Requires portable pickling/serialization! 9 -
Short History of Spores • Goals of Spores: ◦ Safety
checking of closures ◦ Increased flexibility, e.g., portable serialization • Main idea: “Spore” = a special kind of closure that ◦ consists of (1) a capture-free function and (2) an explicit environment ◦ where the environment is constrained using type classes • Spores were first proposed in SIP-21 in 2013 and presented at ECOOP'14 10 Miller, Haller, and Odersky. Spores: a type-based foundation for closures in the age of concurrency and distribution. ECOOP 2014 (Google Scholar: 57 citations) https://doi.org/10.1007/978-3-662-44202-9_13 -
Spores3 • A new take on Spores for Scala 3
• Uses a new approach for type-class-based pickling • A simpler and more robust implementation • Talk and paper (open access): 11 Philipp Haller: Enhancing closures in Scala 3 with Spores3. ACM SIGPLAN Scala Symposium, Berlin, Germany, June 2022. https://doi.org/10.1145/3550198.3550428 Talk at Strange Loop 2022: video, slides -
This Talk • Spores are now even more flexible in
the new version 0.2 ◦ Partial application of Spores ◦ Support for context functions ◦ ... computation on pickled data made simple! 12 -
The Spore Factory // Spore without environment val spore =
Spore { (x: Int) => x + 1 } // with optional type annotation val spore: Spore[Int => Int] = Spore { x => x + 1 } // Spore with environment val num = 12 val spore: Spore[Int => Int] = Spore(num) { (env: Int) => (x: Int) => x + env } ^^^ ^^^ ^^^ env is replaced with the value of num in body 13 -
// Spore with context environment val num = 12 val
spore: Spore[Int => Int] = Spore.applyWithCtx(num) { (env: Int) ?=> (x: Int) => x + summon[Int]} The Spore Factory 14 -
Spore Auto Capture // Automatically captures environment variables // y,
z captured* val y = 12 val z = “World” val spore: Spore[Int => String] = Spore.auto { (x: Int) => (x + y).toString() + z } ^ ^ 15 *if not top-level -
Unwrapping / Applying a Spore 16 // unwraps the value
in the spore val spore: Spore[Int => Int] = Spore { ... } spore.unwrap().apply(12) // or, using imported implicit conversion import spores.default.given spore(12) // Spore[Int => Int] is converted to Int => Int -
val y = 12 val spore: Spore[Int => Int] =
Spore { (x: Int) => x + y } The Spore Factory: Compile-Time Checks 17 More examples here and here -
val y = 12 val spore: Spore[Int => Int] =
Spore { (x: Int) => x + y } ^^^ Error: Invalid capture of variable `y`. The Spore Factory: Compile-Time Checks 18 More examples here and here -
class Foo: val x = 12 def spore = Spore.apply
{ () => 42 * x } The Spore Factory: Compile-Time Checks 19 More examples here and here -
class Foo: val x = 12 def spore = Spore.apply
{ () => 42 * x } ^^^ (i.e., this.x) Error: Invalid capture of `this` from class Foo. The Spore Factory: Compile-Time Checks 20 More examples here and here -
// What about this example? class Outer: class Foo def
spore = Spore.apply { () => class Bar extends Foo new Bar() } The Spore Factory: Compile-Time Checks 21 -
// What about this example? class Outer: class Foo def
spore = Spore.apply { () => class Bar extends Foo ^^^ new Bar() } Error: Invalid capture of variable `Outer.Foo`. The Spore Factory: Compile-Time Checks 22 -
// What about this example? ... class Bar def foo(using
Bar): Spore[Bar] = Spore.apply { summon } The Spore Factory: Compile-Time Checks 23 -
// What about this example? ... class Bar def foo(using
Bar): Spore[Bar] = Spore.apply { summon } ^^^^^^ Error: Invalid capture of variable `x$1`. The Spore Factory: Compile-Time Checks 24 Captured context parameter! -
• Problematic capturing is prevented • This ensures that it
is always safe to pickle / unpickle a Spore • Pickling example: import upickle.default.* val spore: Spore[Int => Int] = ... val pickled: String = write(spore) // JSON String val unpickled = read[Spore[Int => Int]](pickled) unpickled: Spore[Int => Int] Safe Pickling of Spores 25 ‘write’ to JSON String ‘read’ from JSON String -
Macro for Checking Captures • Macros.checkBodyExpr(bodyExpr) ◦ Collect all symbols
which might be captured ▪ Ident, This, ClassDefs, New, etc. ▪ Ignore some special cases, e.g. ignore U in new T[U]() ◦ Remove symbols owned by the closure (parameters, etc.) ◦ Remove top-level symbols ◦ Remaining symbols are captured 26 -
• Functions are capture free ◦ Checked by macros •
Captured environment variables ◦ By partial application Spores3 Approach 27 -
// Apply (partially) a Spore to a value val spore:
Spore[Int => Int => String] spore.withEnv(12): Spore[Int => String] ^^ requires a given: Spore[ReadWriter[Int]] // Apply (partially) a Spore to a Spore val spore: Spore[Int => Int => String] = ... val spore12: Spore[Int] = Spore { 12 } spore.withEnv2(spore12): Spore[Int => String] ^^^^^^^ no given needed Partial Application 28 -
// Also works for context parameters val spore2: Spore[Int ?=>
Int => String] spore2.withCtx(12): Spore[Int => String] ... spore2.withCtx2(spore12): Spore[Int => String] Partial Application 29 -
Creating a `given Spore[ReadWriter[T]]` 30 // Works if the summoned
ReadWriter is top-level given Spore[ReadWriter[Int]] = Spore { summon } // Works for all top-level ReadWriters inline given [T: ReadWriter]: Spore[ReadWriter[T]] = Spore { summon[ReadWriter[T]] } -
• Use Spores for serializing actor behaviors • Periodically serialize
execution state • Recover state after crash • https://github.com/jspenger/durable-actor (WIP) ◦ API inspired by Akka Typed Example: Durable Actors 31 -
Preliminary Microbenchmarks • Compare Spores3* to Java** • Pickling performance:
◦ Speedup between 1.3x and 5.4x ◦ Size reduction between 2.5x and 9.2x • More details in the backup slides 32 *Serialized Spore by upickle writeBinary **Serialized by java.io.ObjectInputStream / java.io.ObjectOutputStream -
Summary • Spores3: making closures in Scala safer and more
flexible ◦ Spores are always safe to pickle / unpickle ◦ Captured environment is checked ◦ Version 0.2 has the latest improvements: https://github.com/phaller/spores3 • Plans for version 0.3: ◦ More flexible Spore type: Spore[F[_], +T] ▪ Generic over the evidence type (so far fixed to ReadWriter[_]) 33 -
Summary • Spores3: making closures in Scala safer and more
flexible ◦ Spores are always safe to pickle / unpickle ◦ Captured environment is checked ◦ Version 0.2 has the latest improvements: https://github.com/phaller/spores3 34 Philipp Haller @philippkhaller (X) @phaller (GitHub) linkedin.com/in/hallerp https://people.kth.se/~phaller/ -
Preliminary Microbenchmarks • Compare Spores3* to Java** • Function1: x
=> () => x + 1 • Function2: (x => () => x + 1).apply(7) • Function3: (list => () => list.sum).apply(List(1, …, 10)) • Function4: (pred => list => list.filter(pred)).apply(x => x % 2 == 0) 35 *Serialized Spore by upickle writeBinary **Serialized by java.io.ObjectInputStream / java.io.ObjectOutputStream -
Operations per Second • Measure time to serialize and deserialize
with JMH ◦ MOPS (million operations per second) • Function1: Spores3: 0.70 Java: 0.13 (MOPS) • Function2: Spores3: 0.16 Java: 0.12 • Function3: Spores3: 0.11 Java: 0.050 • Function4: Spores3: 0.22 Java: 0.10 • Speedup between 1.3x and 5.4x 36 -
• Serialize to Array[Byte] ◦ Measure size in Byte •
Function1: Spores3: 67 Java: 622 (Byte) • Function2: Spores3: 223 Java: 562 • Function3: Spores3: 359 Java: 1166 • Function4: Spores3: 186 Java: 862 • Size reduction between 2.5x and 9.2x Serialization Size 37