Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On Pickles & Spores: Improving Support for Distributed Programming in Scala

On Pickles & Spores: Improving Support for Distributed Programming in Scala

A talk about a new serialization framework for Scala, and a new idea for sending closures over the wire.

Slides from the ScalaDays 2013 talk.

Heather Miller

June 12, 2013
Tweet

More Decks by Heather Miller

Other Decks in Programming

Transcript

  1. pICKLES
    &sPORES
    HEATHER MILLER
    On
    IMPROVING SCALA’S SUPPORT FOR
    DISTRIBUTED PROGRAMMING

    View full-size slide

  2. plug:
    http://lampwww.epfl.ch/~hmiller/scala2013/

    View full-size slide

  3. pICKLES
    &sPORES
    HEATHER MILLER
    On
    IMPROVING SCALA’S SUPPORT FOR
    DISTRIBUTED PROGRAMMING

    View full-size slide

  4. with:
    PHILIPP HALLER
    EUGENE BURMAKO
    MARTIN ODERSKY
    TYPESAFE
    EPFL
    EPFL/TYPESAFE

    View full-size slide

  5. wHAT IS THIS TALK ABOUT?

    View full-size slide

  6. wHAT IS THIS TALK ABOUT?
    Making distributed
    programming
    easier in scala

    View full-size slide

  7. This kind of distributed system

    View full-size slide

  8. This kind of distributed system
    insert social
    network of
    your choice
    here
    ALso!

    View full-size slide

  9. Bottomline:
    Machines Communicating

    View full-size slide

  10. Bottomline:
    Machines Communicating
    How can we simplify distribution
    at the language-level?

    View full-size slide

  11. sCALA pICKLING
    Agenda
    macros & typeclasses
    Spores

    View full-size slide

  12. sCALA pICKLING
    Agenda
    Spores

    View full-size slide

  13. Pickles!
    scala.pickling
    https://github.com/scala/pickling

    View full-size slide

  14. What is it?
    https://github.com/scala/pickling

    View full-size slide

  15. What is it?
    PICKLING == SERIALIZATION == MARSHALLING
    https://github.com/scala/pickling

    View full-size slide

  16. What is it?
    PICKLING == SERIALIZATION == MARSHALLING
    very different from
    java serialization
    https://github.com/scala/pickling

    View full-size slide

  17. wait, why do we care?
    https://github.com/scala/pickling

    View full-size slide

  18. Slow!
    wait, why do we care?
    https://github.com/scala/pickling

    View full-size slide

  19. Closed!
    Slow!
    wait, why do we care?
    https://github.com/scala/pickling

    View full-size slide

  20. Closed!
    Slow!
    wait, why do we care?
    not serializable
    exceptions at runtime
    https://github.com/scala/pickling

    View full-size slide

  21. Closed!
    Slow!
    wait, why do we care?
    not serializable
    exceptions at runtime
    can’t retroactively make
    classes serializable
    https://github.com/scala/pickling

    View full-size slide

  22. Enter: Scala Pickling
    https://github.com/scala/pickling

    View full-size slide

  23. Enter: Scala Pickling
    fast: Serialization code generated at compile-
    time and inlined at the use-site.
    https://github.com/scala/pickling

    View full-size slide

  24. Enter: Scala Pickling
    fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern, retroactively make
    types serializable
    https://github.com/scala/pickling

    View full-size slide

  25. Enter: Scala Pickling
    fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern, retroactively make
    types serializable
    NO BOILERPLATE:
    Typeclass instances generated at compile-time
    https://github.com/scala/pickling

    View full-size slide

  26. Enter: Scala Pickling
    fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern, retroactively make
    types serializable
    NO BOILERPLATE:
    Typeclass instances generated at compile-time
    pluggable formats:
    Effortlessly change format of serialized data:
    binary, JSON, invent your own!
    https://github.com/scala/pickling

    View full-size slide

  27. Enter: Scala Pickling
    fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern, retroactively make
    types serializable
    NO BOILERPLATE:
    Typeclass instances generated at compile-time
    pluggable formats:
    Effortlessly change format of serialized data:
    binary, JSON, invent your own!
    typesafe:
    Picklers are type-specialized. Catch errors at
    compile-time! https://github.com/scala/pickling

    View full-size slide

  28. What does it look like?
    https://github.com/scala/pickling

    View full-size slide

  29. What does it look like?
    scala>  import  scala.pickling._
    import  scala.pickling._
    https://github.com/scala/pickling

    View full-size slide

  30. What does it look like?
    scala>  import  scala.pickling._
    import  scala.pickling._
    https://github.com/scala/pickling
    scala>  import  json._
    import  json._

    View full-size slide

  31. What does it look like?
    scala>  import  scala.pickling._
    import  scala.pickling._
    https://github.com/scala/pickling
    scala>  import  json._
    import  json._
    scala>  case  class  Person(name:  String,  age:  Int)
    defined  class  Person
    scala>  Person("John  Oliver",  36)
    res0:  Person  =  Person(John  Oliver,36)

    View full-size slide

  32. What does it look like?
    scala>  import  scala.pickling._
    import  scala.pickling._
    https://github.com/scala/pickling
    scala>  import  json._
    import  json._
    scala>  case  class  Person(name:  String,  age:  Int)
    defined  class  Person
    scala>  Person("John  Oliver",  36)
    res0:  Person  =  Person(John  Oliver,36)
    scala>  res0.pickle
    res1:  scala.pickling.json.JSONPickle  =  
    JSONPickle({
       "tpe":  "Person",
       "name":  "John  Oliver",
       "age":  36
    })

    View full-size slide

  33. and...
    it’s pretty fast
    https://github.com/scala/pickling

    View full-size slide

  34. collections: Time
    Benchmarks

    View full-size slide

  35. Benchmarks
    collections: free Memory
    (more is better)

    View full-size slide

  36. Benchmarks
    collections: size

    View full-size slide

  37. Benchmarks
    geotrellis: time

    View full-size slide

  38. Benchmarks
    evactor: time
    Java runs out of memory

    View full-size slide

  39. Benchmarks
    evactor: time (no java, more events)

    View full-size slide

  40. that’s just the
    btw,
    default behavior...
    https://github.com/scala/pickling

    View full-size slide

  41. that’s just the
    btw,
    default behavior...
    you can really customize scala
    pickling too.
    https://github.com/scala/pickling

    View full-size slide

  42. Previous examples used
    default behavior
    Customizing Pickling
    Pickling is very customizable
    Before we can show these things,let's have a
    look at the building block of the framework...
    Generated picklers
    Standard pickle format
    Custom picklers for specific types
    Custom pickle format

    View full-size slide

  43. Pickler Combinators
    trait  Pickler[T]  {
       //  returns  next  write  position
       def  pickle(arr:  Array[Byte],  i:  Int,  x:  T):  Int
       //  returns  result  plus  next  read  position
       def  unpickle(arr:  Array[Byte],  i:  Int):  (T,  Int)
    }
    Elegant programming pearl that comes from
    functional programming.
    A composable and "constructive" way to think about
    persisting data.
    Compose picklers for simple types to build picklers
    for more complicated types
    What is a pickler?
    simplified version of what’s
    actually used in scala-
    pickling
    https://github.com/scala/pickling

    View full-size slide

  44. https://github.com/scala/pickling
    Pickler Combinators
    We need 2 things:
    fully-implemented picklers
    for some basic types like
    primitives
    1 Picklers for base types
    Functions that combine existing picklers
    to build compound picklers
    2
    example: combinator that
    takes a Pickler[T] and returns a
    Pickler[List[T]]

    View full-size slide

  45. Pickler Combinators
    Build a pickler for pairs (Int, String),
    combine an Int pickler and a String pickler
    val  myPairPickler  =  tuple2Pickler(intPickler,  stringPickler)
    What’s the type?
    Can we combine them automatically?
    Pickler[T], can pickle objects of type T
    def pickle(implicit pickler: Pickler[(Int, String)]) = {
    pickler.pickle((32, “yay!”))
    }
    Goal:
    Can take intPickler and stringPickler as implicit parameters
    tuple2Pickler can be an implicit def
    https://github.com/scala/pickling

    View full-size slide

  46. Implicit Picklers
    case  class  Person(name:  String,  age:  Int,  salary:  Int)
    class  CustomPersonPickler(implicit  val  format:  PickleFormat)  extends  SPickler[Person]  {
       def  pickle(picklee:  Person,  builder:  PBuilder):  Unit  =  {
           builder.beginEntry(picklee)
           builder.putField("name",  b  =>
               b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())
           builder.putField("age",  b  =>
               b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())
           builder.endEntry()
       }
    }
    implicit  def  genCustomPersonPickler(implicit  format:  PickleFormat)  =
       new  CustomPersonPickler
    customize what you pickle!
    https://github.com/scala/pickling

    View full-size slide

  47. Pickle Format
           trait  PickleFormat  {
                   type  PickleType  <:  Pickle
                   def  createBuilder():  PBuilder
                   def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader
               }
           trait  PBuilder  extends  Hintable  {
               def  beginEntry(picklee:  Any):  this.type
               def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type
               def  endEntry():  Unit
               def  beginCollection(length:  Int):  this.type
               def  putElement(pickler:  this.type  =>  Unit):  this.type
               def  endCollection(length:  Int):  Unit
               def  result():  Pickle
           }
    https://github.com/scala/pickling

    View full-size slide

  48. Pickle Format
    output any format!
           trait  PickleFormat  {
                   type  PickleType  <:  Pickle
                   def  createBuilder():  PBuilder
                   def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader
               }
           trait  PBuilder  extends  Hintable  {
               def  beginEntry(picklee:  Any):  this.type
               def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type
               def  endEntry():  Unit
               def  beginCollection(length:  Int):  this.type
               def  putElement(pickler:  this.type  =>  Unit):  this.type
               def  endCollection(length:  Int):  Unit
               def  result():  Pickle
           }
    https://github.com/scala/pickling

    View full-size slide

  49. Pickle Format
    https://gist.github.com/heathermiller/5760171
    example
    Output edn, Clojure’s data transfer format.
    toy builder implementation:
    scala>  import  scala.pickling._
    import  scala.pickling._
    scala>  import  edn._
    import  edn._
    scala>  case  class  Person(name:  String,  kidsAges:  Array[Int])
    defined  class  Person
    scala>  val  joe  =  Person("Joe",  Array(3,  4,  13))
    joe:  Person  =  Person(Joe,[I@3d925789)
    scala>  joe.pickle.value
    res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }

    View full-size slide

  50. Pickle Format
    https://gist.github.com/heathermiller/5760171
    example
    Output edn, Clojure’s data transfer format.
    talk to a clojure app
    toy builder implementation:
    scala>  import  scala.pickling._
    import  scala.pickling._
    scala>  import  edn._
    import  edn._
    scala>  case  class  Person(name:  String,  kidsAges:  Array[Int])
    defined  class  Person
    scala>  val  joe  =  Person("Joe",  Array(3,  4,  13))
    joe:  Person  =  Person(Joe,[I@3d925789)
    scala>  joe.pickle.value
    res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }

    View full-size slide

  51. What can be pickled?
    if there is an implicit of type Pickler[Foo] in scope,
    you can pickle instances of it
    types for which an implicit pickler is in scope
    types for which our framework can
    generate picklers
    classes
    case classes
    generic classes
    singleton objects
    primitives & primitive arrays
    ...
    can’t (yet): instances of inner classes, Types
    https://github.com/scala/pickling

    View full-size slide

  52. What can be pickled?
    if there is an implicit of type Pickler[Foo] in scope,
    you can pickle instances of it
    types for which an implicit pickler is in scope
    types for which our framework can
    generate picklers
    classes
    case classes
    generic classes
    singleton objects
    primitives & primitive arrays
    ...
    IMPORTANT: The implicit picklers
    are used in the generation!
    can’t (yet): instances of inner classes, Types
    https://github.com/scala/pickling

    View full-size slide

  53. Status
    Release 0.8.0 for Scala 2.10.2
    No support for inner classes, yet
    ScalaCheck tests
    Very soon: support for cyclic object graphs (for release 0.9.0)
    https://github.com/scala/pickling

    View full-size slide

  54. Status
    Scala 2.11 as target
    Goal:
    Release 0.8.0 for Scala 2.10.2
    No support for inner classes, yet
    ScalaCheck tests
    Very soon: support for cyclic object graphs (for release 0.9.0)
    https://github.com/scala/pickling

    View full-size slide

  55. Status
    Scala 2.11 as target
    Goal:
    Plan:
    1.0 release within the next few months
    SIP for Scala 2.11
    Integration with sbt, Spark, and Akka, ...
    Experiment: use Scala-pickling to speed up Scala compiler
    Release 0.8.0 for Scala 2.10.2
    No support for inner classes, yet
    ScalaCheck tests
    Very soon: support for cyclic object graphs (for release 0.9.0)
    https://github.com/scala/pickling

    View full-size slide

  56. And now onto
    something
    completely
    different.

    View full-size slide

  57. spores!
    scala
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  58. What are they?

    View full-size slide

  59. small units of mobile
    functional
    behavior
    What are they?

    View full-size slide

  60. http://docs.scala-lang.org/sips/pending/spores.html
    proposed for inclusion in scala 2.11
    small units of mobile
    functional
    behavior
    What are they?

    View full-size slide

  61. First, some
    Motivation
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  62. Motivation
    def  receive  =  {
       case  Request(data)  =>
           future  {
               val  result  =  transform(data)
               sender  !  Response(result)
           }
    }
    Potential hazards when using closures incorrectly:
    • Memory leaks
    • Race conditions, due to capturing mutable references
    • Runtime serialization errors, due to unintended capture of references
    akka actor spawns a
    future to concurrently
    process incoming reqs
    not a stable value!
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  63. Serialization
    case  class  Helper(name:  String)
    class  Main  {
       val  helper  =  Helper("the  helper")
       val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {
           val  result  =  x  +  "  "  +  helper.toString
           println("The  result  is:  "  +  result)
       }
    }
    Serialization of fun throws a
    NotSerializableException.
    Why?
    doesn’t extend
    serializable
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  64. Spores
    val s = spore {
    val h = helper
    (x: Int) => {
    val result = x + " " + h.toString
    println("The result is: " + result)
    }
    }
    an alternative way to create closure-like objects, in a way
    where the environment is controlled

    View full-size slide

  65. Spores
    val s = spore {
    val h = helper
    (x: Int) => {
    val result = x + " " + h.toString
    println("The result is: " + result)
    }
    }
    an alternative way to create closure-like objects, in a way
    where the environment is controlled
    The body of a spore consists of two parts:
    1. a sequence of local value (val) declarations only, and
    2.a closure.

    View full-size slide

  66. Spores
    val s = spore {
    val h = helper
    (x: Int) => {
    val result = x + " " + h.toString
    println("The result is: " + result)
    }
    }
    an alternative way to create closure-like objects, in a way
    where the environment is controlled
    The body of a spore consists of two parts:
    1. a sequence of local value (val) declarations only, and
    2.a closure.
    The closure of a spore has to satisfy the following rule.
    All free variables of the closure body have to be either
    1. parameters of the closure, or
    2.declared in the preceding sequence of local value declarations.

    View full-size slide

  67. Inspiration
    Spores are not limited to distributed programming
    Make your APIs safer by expecting a Spore
    instead of a function
    By requiring spores in your public APIs you can
    prevent users from introducing hazards, like race conditions.
    Make your users more happy by preventing
    them from shooting themselves in the foot!
    do you have state & asynchrony?
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  68. Status
    SIP-21 Spores: on docs.scala-lang.org/sips now!
    Get involved in the discussion!
    Pull request for Scala 2.11 and Akka 2.2.1 in preparation
    Integration with Scala-pickling planned
    http://docs.scala-lang.org/sips/pending/spores.html

    View full-size slide

  69. qUESTIONS
    ?
    [email protected]
    @heathercmiller
    Contact

    View full-size slide