pICKLES &sPORES HEATHER MILLER On IMPROVING SCALA’S SUPPORT FOR DISTRIBUTED PROGRAMMING

plug: http://lampwww.epfl.ch/~hmiller/scala2013/

pICKLES &sPORES HEATHER MILLER On IMPROVING SCALA’S SUPPORT FOR DISTRIBUTED PROGRAMMING

with: PHILIPP HALLER EUGENE BURMAKO MARTIN ODERSKY TYPESAFE EPFL EPFL/TYPESAFE

wHAT IS THIS TALK ABOUT?

wHAT IS THIS TALK ABOUT? Making distributed programming easier in scala

This kind of distributed system

This kind of distributed system insert social network of your choice here ALso!

Bottomline: Machines Communicating

Bottomline: Machines Communicating How can we simplify distribution at the language-level?

sCALA pICKLING Agenda macros & typeclasses Spores

sCALA pICKLING Agenda Spores

Pickles! scala.pickling https://github.com/scala/pickling

What is it? https://github.com/scala/pickling

What is it? PICKLING == SERIALIZATION == MARSHALLING https://github.com/scala/pickling

What is it? PICKLING == SERIALIZATION == MARSHALLING very different from java serialization https://github.com/scala/pickling

wait, why do we care? https://github.com/scala/pickling

Slow! wait, why do we care? https://github.com/scala/pickling

Closed! Slow! wait, why do we care? https://github.com/scala/pickling

Closed! Slow! wait, why do we care? not serializable exceptions at runtime https://github.com/scala/pickling

Closed! Slow! wait, why do we care? not serializable exceptions at runtime can’t retroactively make classes serializable https://github.com/scala/pickling

Enter: Scala Pickling https://github.com/scala/pickling

Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. https://github.com/scala/pickling

Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable https://github.com/scala/pickling

Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time https://github.com/scala/pickling

Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! https://github.com/scala/pickling

Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! typesafe: Picklers are type-specialized. Catch errors at compile-time! https://github.com/scala/pickling

What does it look like? https://github.com/scala/pickling

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36)

What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36) scala>  res0.pickle res1:  scala.pickling.json.JSONPickle  =   JSONPickle({    "tpe":  "Person",    "name":  "John  Oliver",    "age":  36 })

and... it’s pretty fast https://github.com/scala/pickling

collections: Time Benchmarks

Benchmarks collections: free Memory (more is better)

Benchmarks collections: size

Benchmarks geotrellis: time

Benchmarks evactor: time Java runs out of memory

Benchmarks evactor: time (no java, more events)

that’s just the btw, default behavior... https://github.com/scala/pickling

that’s just the btw, default behavior... you can really customize scala pickling too. https://github.com/scala/pickling

Previous examples used default behavior Customizing Pickling Pickling is very customizable Before we can show these things,let's have a look at the building block of the framework... Generated picklers Standard pickle format Custom picklers for specific types Custom pickle format

Pickler Combinators trait  Pickler[T]  {    //  returns  next  write  position    def  pickle(arr:  Array[Byte],  i:  Int,  x:  T):  Int    //  returns  result  plus  next  read  position    def  unpickle(arr:  Array[Byte],  i:  Int):  (T,  Int) } Elegant programming pearl that comes from functional programming. A composable and "constructive" way to think about persisting data. Compose picklers for simple types to build picklers for more complicated types What is a pickler? simplified version of what’s actually used in scala- pickling https://github.com/scala/pickling

https://github.com/scala/pickling Pickler Combinators We need 2 things: fully-implemented picklers for some basic types like primitives 1 Picklers for base types Functions that combine existing picklers to build compound picklers 2 example: combinator that takes a Pickler[T] and returns a Pickler[List[T]]

Pickler Combinators Build a pickler for pairs (Int, String), combine an Int pickler and a String pickler val  myPairPickler  =  tuple2Pickler(intPickler,  stringPickler) What’s the type? Can we combine them automatically? Pickler[T], can pickle objects of type T def pickle(implicit pickler: Pickler[(Int, String)]) = { pickler.pickle((32, “yay!”)) } Goal: Can take intPickler and stringPickler as implicit parameters tuple2Pickler can be an implicit def https://github.com/scala/pickling

Implicit Picklers case  class  Person(name:  String,  age:  Int,  salary:  Int) class  CustomPersonPickler(implicit  val  format:  PickleFormat)  extends  SPickler[Person]  {    def  pickle(picklee:  Person,  builder:  PBuilder):  Unit  =  {        builder.beginEntry(picklee)        builder.putField("name",  b  =>            b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())        builder.putField("age",  b  =>            b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())        builder.endEntry()    } } implicit  def  genCustomPersonPickler(implicit  format:  PickleFormat)  =    new  CustomPersonPickler customize what you pickle! https://github.com/scala/pickling

Pickle Format        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling

Pickle Format output any format!        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling

Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format. toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }

Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format. talk to a clojure app toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }

What can be pickled? if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... can’t (yet): instances of inner classes, Types https://github.com/scala/pickling

What can be pickled? if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... IMPORTANT: The implicit picklers are used in the generation! can’t (yet): instances of inner classes, Types https://github.com/scala/pickling

Status Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling

Status Scala 2.11 as target Goal: Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling

Status Scala 2.11 as target Goal: Plan: 1.0 release within the next few months SIP for Scala 2.11 Integration with sbt, Spark, and Akka, ... Experiment: use Scala-pickling to speed up Scala compiler Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling

And now onto something completely different.

spores! scala http://docs.scala-lang.org/sips/pending/spores.html

What are they?

small units of mobile functional behavior What are they?

http://docs.scala-lang.org/sips/pending/spores.html proposed for inclusion in scala 2.11 small units of mobile functional behavior What are they?

First, some Motivation http://docs.scala-lang.org/sips/pending/spores.html

Motivation def  receive  =  {    case  Request(data)  =>        future  {            val  result  =  transform(data)            sender  !  Response(result)        } } Potential hazards when using closures incorrectly: • Memory leaks • Race conditions, due to capturing mutable references • Runtime serialization errors, due to unintended capture of references akka actor spawns a future to concurrently process incoming reqs not a stable value! http://docs.scala-lang.org/sips/pending/spores.html

Serialization case  class  Helper(name:  String) class  Main  {    val  helper  =  Helper("the  helper")    val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {        val  result  =  x  +  "  "  +  helper.toString        println("The  result  is:  "  +  result)    } } Serialization of fun throws a NotSerializableException. Why? doesn’t extend serializable http://docs.scala-lang.org/sips/pending/spores.html

Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled

Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure.

Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure. The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either 1. parameters of the closure, or 2.declared in the preceding sequence of local value declarations.

Inspiration Spores are not limited to distributed programming Make your APIs safer by expecting a Spore instead of a function By requiring spores in your public APIs you can prevent users from introducing hazards, like race conditions. Make your users more happy by preventing them from shooting themselves in the foot! do you have state & asynchrony? http://docs.scala-lang.org/sips/pending/spores.html

Status SIP-21 Spores: on docs.scala-lang.org/sips now! Get involved in the discussion! Pull request for Scala 2.11 and Akka 2.2.1 in preparation Integration with Scala-pickling planned http://docs.scala-lang.org/sips/pending/spores.html

qUESTIONS ? heather.miller@epfl.ch @heathercmiller Contact