$30 off During Our Annual Pro Sale. View Details »

Spores: Distributable Functions in Scala

Heather Miller
September 19, 2013

Spores: Distributable Functions in Scala

Pickles & Spores: Improving Support for Distributed Programming in Scala

Spores are "small units of possibly mobile functional behavior". They're a closure-like abstraction meant for use in distributed or concurrent environments.

Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment.

The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala's type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What's more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints.

The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.

Part of a series of talks on Improving Support for Distributed Programming in Scala.

Presented at Strange Loop 2013

Heather Miller

September 19, 2013
Tweet

More Decks by Heather Miller

Other Decks in Programming

Transcript

  1. Heather Miller
    SPORES
    Improving Support for
    Distributed Programmingin
    @heathercmiller
    heather.miller@epfl.ch
    Distributable
    functions
    :
    in

    View Slide

  2. PICKLES
    Heather Miller
    SPORES
    &
    Improving Support for
    Distributed Programmingin
    @heathercmiller
    heather.miller@epfl.ch

    View Slide

  3. with:
    Philipp Haller
    Eugene Burmako
    Martin Odersky
    Typesafe
    EPFL
    EPFL/Typesafe

    View Slide

  4. What is this talk about?

    View Slide

  5. What is this talk about?
    Making distributed
    programming
    easier in scala

    View Slide

  6. This kind of distributed system

    View Slide

  7. This kind of distributed system
    insert social
    network of
    your choice
    here
    !

    View Slide

  8. Bottomline:
    M C

    View Slide

  9. Bottomline:
    How can we simplify
    distribution at the
    language-level?
    M C

    View Slide

  10. Spores
    pickling

    View Slide

  11. Spores
    pickling
    75%

    View Slide

  12. Spores
    pickling
    75%
    25%

    View Slide

  13. THIS STUFF IS BOTH
    RESEARCH & intended for production
    Spores
    pickling

    View Slide

  14. Spores
    pickling
    Instant Pickles: Generating Object-Oriented Pickler
    Combinators for Fast and Extensible Serialization,
    Heather Miller, Philipp Haller, Eugene Burmako, Martin Odersky.
    @OOPSLA’13, Indianapolis, IN, October 26-31, 2013.
    RESEARCH Accepted for publication at OOPSLA’13
    Used by a handful of companies, Scala
    Language Proposal in the works
    Practice

    View Slide

  15. Spores
    pickling

    View Slide

  16. Spores
    pickling
    RESEARCH
    Academic paper on the foundations and practical
    benefits of Scala’s spores in the works. Draft will
    be available in the coming months.
    Practice
    Scala Improvement Proposal posted, lots of user
    feedback, helped reformulate the design.
    Release soon upcoming of what will come in the
    Scala 2.11 distribution.

    View Slide

  17. SPORES!
    . p r

    View Slide

  18. SINGLE
    MACHINE
    SCENARIO
    imagine the...
    r ,

    View Slide

  19. closures are
    wonderful

    View Slide

  20. are wonderful
    C r
    How do you do fp without them?

    View Slide

  21. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?

    View Slide

  22. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.

    View Slide

  23. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.

    View Slide

  24. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.
    essentially, you’re sending the
    closure to the data.

    View Slide

  25. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.
    essentially, you’re sending the
    closure to the data.
    even java’s going to get closures
    O ,

    View Slide

  26. distributed
    back to thinking
    O ,

    View Slide

  27. Closures are awesome.

    View Slide

  28. but
    we
    can’t
    really
    distributethem.
    Closures are awesome.

    View Slide

  29. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?

    View Slide

  30. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?
    instead of compile-time
    checks.
    runtime errors

    View Slide

  31. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?
    instead of compile-time
    checks.
    runtime errors
    for a user, often
    unclear whether it’s a user-error or the
    framework
    Who’s fault is it?

    View Slide

  32. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...

    View Slide

  33. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.

    View Slide

  34. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.
    When picking battles, framework designers
    tend to avoid issues with closures.

    View Slide

  35. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.
    rightfully so.
    When picking battles, framework designers
    tend to avoid issues with closures.

    View Slide

  36. Have you heard this before?:

    View Slide

  37. fUNCTIONAL PROGRAMMING
    The Way To Go™
    ...for parallelism/concurrency/distribution
    Have you heard this before?:

    View Slide

  38. fUNCTIONAL PROGRAMMING
    The Way To Go™
    ...for parallelism/concurrency/distribution
    Have you heard this before?:
    ...And composing and passing functions
    around is the way to do FP.
    However, managing closures in a concurrent
    or distributed environment, or writing APIs
    to be used by clients in such an environment,
    remains considerably precarious.

    View Slide

  39. But wouldn’t it be nice
    if we could...
    More reliably
    develop new frameworks
    for distributed programming based
    on passing functions ?
    develop new types of frameworks
    passing functions
    ...foolproof dist collections, dist streams, ...

    View Slide

  40. eNTER:
    spores

    View Slide

  41. eNTER:
    spores
    w :
    1
    spores with type constraints
    2 mainline spores
    proposed for inclusion in Scala 2.11
    research project @EPFL,
    research paper in the works

    View Slide

  42. 2
    spores
    w :
    spores with type constraints
    research project @EPFL,
    research paper in the works
    proposed for inclusion in Scala 2.11
    mainline spores
    1

    View Slide

  43. Spores
    what are they?
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    behavior
    small units of possibly mobile
    functional

    View Slide

  44. Spores
    what are they?
    A closure-like abstraction for use in
    distributed or concurrent environments.
    goal:
    Well-behaved closures with controlled
    environments that can avoid various
    hazards.
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11

    View Slide

  45. Spores
    what are they?
    A closure-like abstraction for use in
    distributed or concurrent environments.
    goal:
    Well-behaved closures with controlled
    environments that can avoid various
    hazards.
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    Potential hazards when using closures
    incorrectly:
    • memory leaks
    • race conditions due to capturing mutable references
    • runtime serialization errors due to unintended
    capture of references

    View Slide

  46. spark
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    class MyCoolRddApp {
    val param = 3.14
    val log = new Log(...)
    ...
    def work(rdd: RDD[Int]) {
    rdd.map(x => x + param)
    .reduce(...)
    }
    }

    View Slide

  47. spark
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    class MyCoolRddApp {
    val param = 3.14
    val log = new Log(...)
    ...
    def work(rdd: RDD[Int]) {
    rdd.map(x => x + param)
    .reduce(...)
    }
    }
    Problem:
    not serializable
    because it captures
    this of type
    MyCoolRddApp
    which is itself not
    serializable
    (x => x + param)

    View Slide

  48. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results

    View Slide

  49. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results
    akka actor spawns a
    future to concurrently
    process incoming reqs

    View Slide

  50. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results
    akka actor spawns a
    future to concurrently
    process incoming reqs
    not a stable value!
    it’s a method call!

    View Slide

  51. Serialization
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    case class Helper(name: String)
    class Main {
    val helper = Helper("the helper")
    val fun: Int => Unit = (x: Int) => {
    val result = x + " " + helper.toString
    println("The result is: " + result)
    }
    }
    Problem:
    fun not serializable.
    Accidentally captures
    this since
    helper.toString
    is really
    this.helper.toString,
    and Main (the type of
    this) is not serializable.

    View Slide

  52. Serialization
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    case class Helper(name: String)
    class Main {
    val helper = Helper("the helper")
    val fun: Int => Unit = (x: Int) => {
    val result = x + " " + helper.toString
    println("The result is: " + result)
    }
    }
    Problem:
    fun not serializable.
    Accidentally captures
    this since
    helper.toString
    is really
    this.helper.toString,
    and Main (the type of
    this) is not serializable.

    View Slide

  53. We need safer closures
    O . G .
    for concurrent &
    distributed scenarios.
    sure.

    View Slide

  54. We need safer closures
    O . G .
    for concurrent &
    distributed scenarios.
    sure.
    what do these things
    look like?

    View Slide

  55. What do spores look like?
    B :
    val  s  =  spore  {
       val  h  =  helper
       (x:  Int)  =>  {
           val  result  =  x  +  "  "  +  h.toString
           println("The  result  is:  "  +  result)
       }
    }
    THE BODY OF A SPORE CONSISTS OF 2 PARTS
    2
    a closure
    a sequence of local value (val) declarations
    only (the “spore header”), and
    1
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11

    View Slide

  56. Spore
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    1. All captured variables are declared in
    the spore header, or using capture
    2. The initializers of captured variables
    are executed once, upon creation of
    the spore
    3. References to captured variables do
    not change during the spore’s execution
    v closures
    ( )
    G r ...

    View Slide

  57. Spores&
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    closures
    Evaluation semantics:
    Remove the spore marker, and the code
    behaves as before
    spores & closures are related:
    You can write a full function literal and pass
    it to something that expects a spore.
    (Of course, only if the function literal
    satisfies the spore rules.)

    View Slide

  58. How can you use a spore?
    O . S .
    In APIs
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  sendOverWire(s:  Spore[Int,  Int]):  Unit  =  ...
    //  ...
    sendOverWire((x:  Int)  =>  x  *  x  -­‐  2)
    If you want parameters to be spores,
    then you can write it this way

    View Slide

  59. How can you use a spore?
    O . S .
    for-comprehensions
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  lookup(i:  Int):  DCollection[Int]  =  ...
    val  indices:  DCollection[Int]  =  ...
    for  {  i  <-­‐  indices
               j  <-­‐  lookup(i)
    }  yield  j  +  capture(i)
    trait  DCollection[A]  {
     def  map[B](sp:  Spore[A,  B]):  DCollection[B]
     def  flatMap[B](sp:  Spore[A,  DCollection[B]]):  DCollection[B]
    }

    View Slide

  60. R ,
    get you?
    what does
    all of that

    View Slide

  61. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    Since...
    Captured expressions are evaluated
    upon spore creation.
    Spores are like function values with
    an immutable environment.
    Plus, environment is specified and
    checked, no accidental capturing.
    That means...

    View Slide

  62. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2

    View Slide

  63. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’ ?
    ?

    View Slide

  64. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’
    5 ‘a’
    ?
    ?

    View Slide

  65. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’
    5 ‘a’
    ?
    ?
    I’m in ur stuff
    draggin around ur object graf

    View Slide

  66. So now that we have
    spores, what kind of
    Gr .
    cool patter can
    they enable?
    Cool Patterns

    View Slide

  67. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Stage and Ship
    Build up a computation graph such that
    the behavior is represented as spores.
    Once the computation has been built, it
    can be safely shipped to remote nodes.
    could be pipeline stages that are
    plugged together

    View Slide

  68. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Function-passing Style Concurrency
    Pass spores between concurrent entities.
    Compose spores with other spores to
    form larger, composite spores, and then
    send.

    View Slide

  69. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Hot-swapping actor behavior
    Have a running actor and want to
    change its behavior.
    Send it a spore and instruct the actor to
    use the spore from now on to process its
    incoming messages.

    View Slide

  70. NEW PATTERNS
    ➡NEW FRAMEWORKS
    Distributed Collections
    Distributed pipelines, stream processing
    Function-passing style frameworks
    SOME IDEAS for potential FRAMEWORKS
    that spores might help
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11

    View Slide

  71. C .

    View Slide

  72. C .
    Socket?
    what if I
    capture a

    View Slide

  73. C .
    Socket?
    what if I
    capture a
    RESEARCH
    REALM OF

    View Slide

  74. proposed for inclusion in Scala 2.11
    mainline spores
    1
    spores
    w :
    2 spores with type constraints
    research project @EPFL,
    research paper in the works

    View Slide

  75. C .
    Socket?
    what if I
    capture a

    View Slide

  76. We don't have the means yet for
    frameworks to express these kinds of
    constraints or to enforce them when we
    create and compose spores.
    RESEARCH
    Wouldn't it be nice if we could
    add these constraints, in a
    friendly, and composable way?

    View Slide

  77. keep track of captured types
    RESEARCH
    I :
    The spore macro can synthesize precise types
    automatically for newly created spores:
    Spore[Int,  ...]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }
    w/ r
    Cr spores
    ...at compile-time
    spore  {  val  x:  Int  =  list.size;  val  a:  ActorRef  =  this.sender
       (y:  Int)  =>  ...
    }  exclude[Actor]
    synthesized type:
    (a whitebox macro)

    View Slide

  78. w/ r
    C p spores
    RESEARCH
    basic composition operators
    (same as for
    regular functions)
    andThen
    compose
    How do we synthesize the result type of
    s1 andThen s2?
    result type synthesized by andThen macro
    type member Facts takes “union” of the facts of
    s1 and s2
    type member Excluded: conjunction of excluded
    types, needs to check Facts to see if possible

    View Slide

  79. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
       type  Facts  =  Captured[Actor]
    }
    s1  andThen  s2    //  does  not  compile

    View Slide

  80. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
       type  Facts  =  Captured[Actor]
    }
    s1  andThen  s2    //  does  not  compile

    View Slide

  81. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
    }
    s1  andThen  s2:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]  with  
    NoCapture[RDD[Int]]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }

    View Slide

  82. what do type constraints
    buy us?
    Stronger constraints checked at
    compile time (not "just" basic spore
    rules)
    Frameworks can make stronger
    assumptions about spores created by
    users.
    Confidence in consuming, creating,
    and composing spores:
    Constraints accumulate monotonically
    Constraints are never lost when
    composing spores
    Less brittleness.

    View Slide

  83. And now onto
    something
    completely
    different.

    View Slide

  84. Pickles!
    .p

    View Slide

  85. W ?
    https://github.com/scala/pickling

    View Slide

  86. W ?
    PICKLING == SERIALIZATION == MARSHALLING
    https://github.com/scala/pickling

    View Slide

  87. W ?
    PICKLING == SERIALIZATION == MARSHALLING
    very different from
    java serialization
    https://github.com/scala/pickling

    View Slide

  88. https://github.com/scala/pickling
    wait, why do we care?

    View Slide

  89. https://github.com/scala/pickling
    S w!
    wait, why do we care?

    View Slide

  90. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?

    View Slide

  91. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?
    not serializable exceptions
    at runtime

    View Slide

  92. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?
    not serializable exceptions
    at runtime
    can’t retroactively make
    classes serializable

    View Slide

  93. fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern,
    retroactively make types serializable
    Typeclass instances generated at
    compile-time
    pluggable formats:
    Effortlessly change format of serialized
    data: binary, JSON, invent your own!
    typesafe:
    Picklers are type-specialized. Catch
    errors at compile-time!
    Enter:S P
    NO BOILERPLATE:

    View Slide

  94. https://github.com/scala/pickling

    View Slide

  95. https://github.com/scala/pickling
    W ?

    View Slide

  96. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._

    View Slide

  97. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._

    View Slide

  98. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._
    scala> case class Person(name: String, age: Int)
    defined class Person
    scala> Person("John Oliver", 36)
    res0: Person = Person(John Oliver,36)

    View Slide

  99. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._
    scala> case class Person(name: String, age: Int)
    defined class Person
    scala> Person("John Oliver", 36)
    res0: Person = Person(John Oliver,36)
    scala> res0.pickle
    res1: scala.pickling.json.JSONPickle =
    JSONPickle({
    "tpe": "Person",
    "name": "John Oliver",
    "age": 36
    })

    View Slide

  100. and...
    ’ pr f
    https://github.com/scala/pickling

    View Slide

  101. collections: Time
    B r

    View Slide

  102. B r
    collections: free Memory
    (more is better)

    View Slide

  103. B r
    collections: size

    View Slide

  104. B r
    geotrellis: time

    View Slide

  105. B r
    evactor: time
    Java runs out of
    memory

    View Slide

  106. B r
    evactor: time (no java, more events)

    View Slide

  107. https://github.com/scala/pickling
    that’s just the
    b w,
    default behavior...

    View Slide

  108. https://github.com/scala/pickling
    that’s just the
    b w,
    default behavior...
    you can really customize scala
    pickling too.

    View Slide

  109. Previous examples used
    default behavior
    C
    Pickling is very customizable
    Generated picklers
    Standard pickle format
    Custom picklers for specific types
    Custom pickle format
    https://github.com/scala/pickling
    pickling

    View Slide

  110. case class Person(name: String, age: Int, salary: Int)
    class CustomPersonPickler(implicit val format: PickleFormat) extends SPickler[Person] {
    def pickle(picklee: Person, builder: PBuilder): Unit = {
    builder.beginEntry(picklee)
    builder.putField("name", b =>
    b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())
    builder.putField("age", b =>
    b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())
    builder.endEntry()
    }
    }
    implicit def genCustomPersonPickler(implicit format: PickleFormat) =
    new CustomPersonPickler
    customize what you pickle!
    https://github.com/scala/pickling
    I p picklers

    View Slide

  111. output any format!
    trait PickleFormat {
    type PickleType <: Pickle
    def createBuilder(): PBuilder
    def createReader(pickle: PickleType, mirror: Mirror): PReader
    }
    trait PBuilder extends Hintable {
    def beginEntry(picklee: Any): PBuilder
    def putField(name: String, pickler: PBuilder => Unit): PBuilder
    def endEntry(): Unit
    def beginCollection(length: Int): PBuilder
    def putElement(pickler: PBuilder => Unit): PBuilder
    def endCollection(length: Int): Unit
    def result(): Pickle
    }
    https://github.com/scala/pickling
    F r
    pickle

    View Slide

  112. https://gist.github.com/heathermiller/5760171
    example
    Output edn, Clojure’s data transfer format.
    talk to a clojure app
    toy builder implementation:
    scala> import scala.pickling._
    import scala.pickling._
    scala> import edn._
    import edn._
    scala> case class Person(name: String, kidsAges: Array[Int])
    defined class Person
    scala> val joe = Person("Joe", Array(3, 4, 13))
    joe: Person = Person(Joe,[I@3d925789)
    scala> joe.pickle.value
    res0: String = #pickling/Person { :name "Joe" :kidsAges [3, 4, 13] }
    F r
    pickle

    View Slide

  113. Scala 2.11 as target
    G :
    P :
    1.0 release within the next few months
    SIP for Scala 2.11
    Integration with sbt, Spark, and Akka, ...
    Release 0.8.0 for Scala 2.10.2
    No support for inner classes, yet
    ScalaCheck tests
    Support for cyclic object graphs, and most Scala types
    https://github.com/scala/pickling
    Status

    View Slide

  114. qUESTIONS
    ?
    heather.miller@epfl.ch
    @heathercmiller
    C

    View Slide