Spores: Distributable Functions in Scala

Pickles & Spores: Improving Support for Distributed Programming in Scala

Spores are "small units of possibly mobile functional behavior". They're a closure-like abstraction meant for use in distributed or concurrent environments.

Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment.

The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala's type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What's more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints.

The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.

Part of a series of talks on Improving Support for Distributed Programming in Scala.

Presented at Strange Loop 2013

Transcript

1. Heather Miller
   SPORES
   Improving Support for
   Distributed Programmingin
   @heathercmiller
   heather.miller@epfl.ch
   Distributable
   functions
   :
   in
   

   View Slide

2. PICKLES
   Heather Miller
   SPORES
   &
   Improving Support for
   Distributed Programmingin
   @heathercmiller
   heather.miller@epfl.ch
   

   View Slide

3. with:
   Philipp Haller
   Eugene Burmako
   Martin Odersky
   Typesafe
   EPFL
   EPFL/Typesafe
   

   View Slide

4. What is this talk about?
   

   View Slide

5. What is this talk about?
   Making distributed
   programming
   easier in scala
   

   View Slide

6. This kind of distributed system
   

   View Slide

7. This kind of distributed system
   insert social
   network of
   your choice
   here
   !
   

   View Slide

8. Bottomline:
   M C
   

   View Slide

9. Bottomline:
   How can we simplify
   distribution at the
   language-level?
   M C
   

   View Slide

10. Spores
    pickling
    

    View Slide

11. Spores
    pickling
    75%
    

    View Slide

12. Spores
    pickling
    75%
    25%
    

    View Slide

13. THIS STUFF IS BOTH
    RESEARCH & intended for production
    Spores
    pickling
    

    View Slide

14. Spores
    pickling
    Instant Pickles: Generating Object-Oriented Pickler
    Combinators for Fast and Extensible Serialization,
    Heather Miller, Philipp Haller, Eugene Burmako, Martin Odersky.
    @OOPSLA’13, Indianapolis, IN, October 26-31, 2013.
    RESEARCH Accepted for publication at OOPSLA’13
    Used by a handful of companies, Scala
    Language Proposal in the works
    Practice
    

    View Slide

15. Spores
    pickling
    

    View Slide

16. Spores
    pickling
    RESEARCH
    Academic paper on the foundations and practical
    benefits of Scala’s spores in the works. Draft will
    be available in the coming months.
    Practice
    Scala Improvement Proposal posted, lots of user
    feedback, helped reformulate the design.
    Release soon upcoming of what will come in the
    Scala 2.11 distribution.
    

    View Slide

17. SPORES!
    . p r
    

    View Slide

18. SINGLE
    MACHINE
    SCENARIO
    imagine the...
    r ,
    

    View Slide

19. closures are
    wonderful
    

    View Slide

20. are wonderful
    C r
    How do you do fp without them?
    

    View Slide

21. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    

    View Slide

22. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    

    View Slide

23. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.
    

    View Slide

24. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.
    essentially, you’re sending the
    closure to the data.
    

    View Slide

25. are wonderful
    C r
    Ok, ok, anonymous inner classes. But still.
    How do you do fp without them?
    fp is all about transformations on
    immutable data.
    These transformations are just closures.
    monads backed by data, like lists, options
    or futures.
    Typically, you pass closures to higher-order
    functions.
    essentially, you’re sending the
    closure to the data.
    even java’s going to get closures
    O ,
    

    View Slide

26. distributed
    back to thinking
    O ,
    

    View Slide

27. Closures are awesome.
    

    View Slide

28. but
    we
    can’t
    really
    distributethem.
    Closures are awesome.
    

    View Slide

29. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?
    

    View Slide

30. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?
    instead of compile-time
    checks.
    runtime errors
    

    View Slide

31. we can’t really distribute them
    B ,
    WHY?
    because they
    capture stuff that’s not serializable.
    oFTEN NOT SERIALIZABLE
    Easy to reference
    something and unknowingly capture it.
    ACCIDENTAL CAPTURE.
    ...enclosing this, anyone?
    instead of compile-time
    checks.
    runtime errors
    for a user, often
    unclear whether it’s a user-error or the
    framework
    Who’s fault is it?
    

    View Slide

32. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    

    View Slide

33. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.
    

    View Slide

34. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.
    When picking battles, framework designers
    tend to avoid issues with closures.
    

    View Slide

35. we can’t really distribute them
    B ,
    Consequences that follow from
    these problems...
    ...not just in their public APIs, but private
    ones too.
    framework builders avoid them
    Users shoot themselves in the foot and blame framework.
    rightfully so.
    When picking battles, framework designers
    tend to avoid issues with closures.
    

    View Slide

36. Have you heard this before?:
    

    View Slide

37. fUNCTIONAL PROGRAMMING
    The Way To Go™
    ...for parallelism/concurrency/distribution
    Have you heard this before?:
    

    View Slide

38. fUNCTIONAL PROGRAMMING
    The Way To Go™
    ...for parallelism/concurrency/distribution
    Have you heard this before?:
    ...And composing and passing functions
    around is the way to do FP.
    However, managing closures in a concurrent
    or distributed environment, or writing APIs
    to be used by clients in such an environment,
    remains considerably precarious.
    

    View Slide

39. But wouldn’t it be nice
    if we could...
    More reliably
    develop new frameworks
    for distributed programming based
    on passing functions ?
    develop new types of frameworks
    passing functions
    ...foolproof dist collections, dist streams, ...
    

    View Slide

40. eNTER:
    spores
    

    View Slide

41. eNTER:
    spores
    w :
    1
    spores with type constraints
    2 mainline spores
    proposed for inclusion in Scala 2.11
    research project @EPFL,
    research paper in the works
    

    View Slide

42. 2
    spores
    w :
    spores with type constraints
    research project @EPFL,
    research paper in the works
    proposed for inclusion in Scala 2.11
    mainline spores
    1
    

    View Slide

43. Spores
    what are they?
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    behavior
    small units of possibly mobile
    functional
    

    View Slide

44. Spores
    what are they?
    A closure-like abstraction for use in
    distributed or concurrent environments.
    goal:
    Well-behaved closures with controlled
    environments that can avoid various
    hazards.
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    

    View Slide

45. Spores
    what are they?
    A closure-like abstraction for use in
    distributed or concurrent environments.
    goal:
    Well-behaved closures with controlled
    environments that can avoid various
    hazards.
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    Potential hazards when using closures
    incorrectly:
    • memory leaks
    • race conditions due to capturing mutable references
    • runtime serialization errors due to unintended
    capture of references
    

    View Slide

46. spark
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    class MyCoolRddApp {
    val param = 3.14
    val log = new Log(...)
    ...
    def work(rdd: RDD[Int]) {
    rdd.map(x => x + param)
    .reduce(...)
    }
    }
    

    View Slide

47. spark
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    class MyCoolRddApp {
    val param = 3.14
    val log = new Log(...)
    ...
    def work(rdd: RDD[Int]) {
    rdd.map(x => x + param)
    .reduce(...)
    }
    }
    Problem:
    not serializable
    because it captures
    this of type
    MyCoolRddApp
    which is itself not
    serializable
    (x => x + param)
    

    View Slide

48. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results
    

    View Slide

49. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results
    akka actor spawns a
    future to concurrently
    process incoming reqs
    

    View Slide

50. Akka/futures
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  receive  =  {
     case  Request(data)  =>
         future  {
             val  result  =  transform(data)
             sender  !  Response(result)
         }
    }
    Problem: Akka actor spawns future to
    concurrently process incoming results
    akka actor spawns a
    future to concurrently
    process incoming reqs
    not a stable value!
    it’s a method call!
    

    View Slide

51. Serialization
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    case class Helper(name: String)
    class Main {
    val helper = Helper("the helper")
    val fun: Int => Unit = (x: Int) => {
    val result = x + " " + helper.toString
    println("The result is: " + result)
    }
    }
    Problem:
    fun not serializable.
    Accidentally captures
    this since
    helper.toString
    is really
    this.helper.toString,
    and Main (the type of
    this) is not serializable.
    

    View Slide

52. Serialization
    v p :
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    case class Helper(name: String)
    class Main {
    val helper = Helper("the helper")
    val fun: Int => Unit = (x: Int) => {
    val result = x + " " + helper.toString
    println("The result is: " + result)
    }
    }
    Problem:
    fun not serializable.
    Accidentally captures
    this since
    helper.toString
    is really
    this.helper.toString,
    and Main (the type of
    this) is not serializable.
    

    View Slide

53. We need safer closures
    O . G .
    for concurrent &
    distributed scenarios.
    sure.
    

    View Slide

54. We need safer closures
    O . G .
    for concurrent &
    distributed scenarios.
    sure.
    what do these things
    look like?
    

    View Slide

55. What do spores look like?
    B :
    val  s  =  spore  {
       val  h  =  helper
       (x:  Int)  =>  {
           val  result  =  x  +  "  "  +  h.toString
           println("The  result  is:  "  +  result)
       }
    }
    THE BODY OF A SPORE CONSISTS OF 2 PARTS
    2
    a closure
    a sequence of local value (val) declarations
    only (the “spore header”), and
    1
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    

    View Slide

56. Spore
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    1. All captured variables are declared in
    the spore header, or using capture
    2. The initializers of captured variables
    are executed once, upon creation of
    the spore
    3. References to captured variables do
    not change during the spore’s execution
    v closures
    ( )
    G r ...
    

    View Slide

57. Spores&
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    closures
    Evaluation semantics:
    Remove the spore marker, and the code
    behaves as before
    spores & closures are related:
    You can write a full function literal and pass
    it to something that expects a spore.
    (Of course, only if the function literal
    satisfies the spore rules.)
    

    View Slide

58. How can you use a spore?
    O . S .
    In APIs
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  sendOverWire(s:  Spore[Int,  Int]):  Unit  =  ...
    //  ...
    sendOverWire((x:  Int)  =>  x  *  x  -­‐  2)
    If you want parameters to be spores,
    then you can write it this way
    

    View Slide

59. How can you use a spore?
    O . S .
    for-comprehensions
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    def  lookup(i:  Int):  DCollection[Int]  =  ...
    val  indices:  DCollection[Int]  =  ...
    for  {  i  <-­‐  indices
               j  <-­‐  lookup(i)
    }  yield  j  +  capture(i)
    trait  DCollection[A]  {
     def  map[B](sp:  Spore[A,  B]):  DCollection[B]
     def  flatMap[B](sp:  Spore[A,  DCollection[B]]):  DCollection[B]
    }
    

    View Slide

60. R ,
    get you?
    what does
    all of that
    

    View Slide

61. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    Since...
    Captured expressions are evaluated
    upon spore creation.
    Spores are like function values with
    an immutable environment.
    Plus, environment is specified and
    checked, no accidental capturing.
    That means...
    

    View Slide

62. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    

    View Slide

63. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’ ?
    ?
    

    View Slide

64. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’
    5 ‘a’
    ?
    ?
    

    View Slide

65. what does all of that
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    get you?
    or, graphically...
    During
    execution
    Right after
    creation
    Spores closures
    1
    2
    5 ‘a’
    5 ‘a’
    ?
    ?
    I’m in ur stuff
    draggin around ur object graf
    

    View Slide

66. So now that we have
    spores, what kind of
    Gr .
    cool patter can
    they enable?
    Cool Patterns
    

    View Slide

67. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Stage and Ship
    Build up a computation graph such that
    the behavior is represented as spores.
    Once the computation has been built, it
    can be safely shipped to remote nodes.
    could be pipeline stages that are
    plugged together
    

    View Slide

68. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Function-passing Style Concurrency
    Pass spores between concurrent entities.
    Compose spores with other spores to
    form larger, composite spores, and then
    send.
    

    View Slide

69. http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    eNABLED BY SPORES
    P r
    Hot-swapping actor behavior
    Have a running actor and want to
    change its behavior.
    Send it a spore and instruct the actor to
    use the spore from now on to process its
    incoming messages.
    

    View Slide

70. NEW PATTERNS
    ➡NEW FRAMEWORKS
    Distributed Collections
    Distributed pipelines, stream processing
    Function-passing style frameworks
    SOME IDEAS for potential FRAMEWORKS
    that spores might help
    http://docs.scala-lang.org/sips/pending/spores.html
    Proposed for inclusion in Scala 2.11
    

    View Slide

71. C .
    

    View Slide

72. C .
    Socket?
    what if I
    capture a
    

    View Slide

73. C .
    Socket?
    what if I
    capture a
    RESEARCH
    REALM OF
    

    View Slide

74. proposed for inclusion in Scala 2.11
    mainline spores
    1
    spores
    w :
    2 spores with type constraints
    research project @EPFL,
    research paper in the works
    

    View Slide

75. C .
    Socket?
    what if I
    capture a
    

    View Slide

76. We don't have the means yet for
    frameworks to express these kinds of
    constraints or to enforce them when we
    create and compose spores.
    RESEARCH
    Wouldn't it be nice if we could
    add these constraints, in a
    friendly, and composable way?
    

    View Slide

77. keep track of captured types
    RESEARCH
    I :
    The spore macro can synthesize precise types
    automatically for newly created spores:
    Spore[Int,  ...]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }
    w/ r
    Cr spores
    ...at compile-time
    spore  {  val  x:  Int  =  list.size;  val  a:  ActorRef  =  this.sender
       (y:  Int)  =>  ...
    }  exclude[Actor]
    synthesized type:
    (a whitebox macro)
    

    View Slide

78. w/ r
    C p spores
    RESEARCH
    basic composition operators
    (same as for
    regular functions)
    andThen
    compose
    How do we synthesize the result type of
    s1 andThen s2?
    result type synthesized by andThen macro
    type member Facts takes “union” of the facts of
    s1 and s2
    type member Excluded: conjunction of excluded
    types, needs to check Facts to see if possible
    

    View Slide

79. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
       type  Facts  =  Captured[Actor]
    }
    s1  andThen  s2    //  does  not  compile
    

    View Slide

80. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
       type  Facts  =  Captured[Actor]
    }
    s1  andThen  s2    //  does  not  compile
    

    View Slide

81. RESEARCH
    w/ r
    E p : spores
    C p
    val  s1:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }  =  ...
    val  s2:  Spore[String,  String]  {
       type  Excluded  =  NoCapture[RDD[Int]]
    }
    s1  andThen  s2:  Spore[Int,  String]  {
       type  Excluded  =  NoCapture[Actor]  with  
    NoCapture[RDD[Int]]
       type  Facts  =  Captured[Int]  with  Captured[ActorRef]
    }
    

    View Slide

82. what do type constraints
    buy us?
    Stronger constraints checked at
    compile time (not "just" basic spore
    rules)
    Frameworks can make stronger
    assumptions about spores created by
    users.
    Confidence in consuming, creating,
    and composing spores:
    Constraints accumulate monotonically
    Constraints are never lost when
    composing spores
    Less brittleness.
    

    View Slide

83. And now onto
    something
    completely
    different.
    

    View Slide

84. Pickles!
    .p
    

    View Slide

85. W ?
    https://github.com/scala/pickling
    

    View Slide

86. W ?
    PICKLING == SERIALIZATION == MARSHALLING
    https://github.com/scala/pickling
    

    View Slide

87. W ?
    PICKLING == SERIALIZATION == MARSHALLING
    very different from
    java serialization
    https://github.com/scala/pickling
    

    View Slide

88. https://github.com/scala/pickling
    wait, why do we care?
    

    View Slide

89. https://github.com/scala/pickling
    S w!
    wait, why do we care?
    

    View Slide

90. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?
    

    View Slide

91. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?
    not serializable exceptions
    at runtime
    

    View Slide

92. https://github.com/scala/pickling
    C !
    S w!
    wait, why do we care?
    not serializable exceptions
    at runtime
    can’t retroactively make
    classes serializable
    

    View Slide

93. fast: Serialization code generated at compile-
    time and inlined at the use-site.
    Flexible:
    Using typeclass pattern,
    retroactively make types serializable
    Typeclass instances generated at
    compile-time
    pluggable formats:
    Effortlessly change format of serialized
    data: binary, JSON, invent your own!
    typesafe:
    Picklers are type-specialized. Catch
    errors at compile-time!
    Enter:S P
    NO BOILERPLATE:
    

    View Slide

94. https://github.com/scala/pickling
    

    View Slide

95. https://github.com/scala/pickling
    W ?
    

    View Slide

96. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    

    View Slide

97. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._
    

    View Slide

98. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._
    scala> case class Person(name: String, age: Int)
    defined class Person
    scala> Person("John Oliver", 36)
    res0: Person = Person(John Oliver,36)
    

    View Slide

99. https://github.com/scala/pickling
    W ?
    scala> import scala.pickling._
    import scala.pickling._
    scala> import json._
    import json._
    scala> case class Person(name: String, age: Int)
    defined class Person
    scala> Person("John Oliver", 36)
    res0: Person = Person(John Oliver,36)
    scala> res0.pickle
    res1: scala.pickling.json.JSONPickle =
    JSONPickle({
    "tpe": "Person",
    "name": "John Oliver",
    "age": 36
    })
    

    View Slide

100. and...
     ’ pr f
     https://github.com/scala/pickling
     

     View Slide

101. collections: Time
     B r
     

     View Slide

102. B r
     collections: free Memory
     (more is better)
     

     View Slide

103. B r
     collections: size
     

     View Slide

104. B r
     geotrellis: time
     

     View Slide

105. B r
     evactor: time
     Java runs out of
     memory
     

     View Slide

106. B r
     evactor: time (no java, more events)
     

     View Slide

107. https://github.com/scala/pickling
     that’s just the
     b w,
     default behavior...
     

     View Slide

108. https://github.com/scala/pickling
     that’s just the
     b w,
     default behavior...
     you can really customize scala
     pickling too.
     

     View Slide

109. Previous examples used
     default behavior
     C
     Pickling is very customizable
     Generated picklers
     Standard pickle format
     Custom picklers for specific types
     Custom pickle format
     https://github.com/scala/pickling
     pickling
     

     View Slide

110. case class Person(name: String, age: Int, salary: Int)
     class CustomPersonPickler(implicit val format: PickleFormat) extends SPickler[Person] {
     def pickle(picklee: Person, builder: PBuilder): Unit = {
     builder.beginEntry(picklee)
     builder.putField("name", b =>
     b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())
     builder.putField("age", b =>
     b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())
     builder.endEntry()
     }
     }
     implicit def genCustomPersonPickler(implicit format: PickleFormat) =
     new CustomPersonPickler
     customize what you pickle!
     https://github.com/scala/pickling
     I p picklers
     

     View Slide

111. output any format!
     trait PickleFormat {
     type PickleType <: Pickle
     def createBuilder(): PBuilder
     def createReader(pickle: PickleType, mirror: Mirror): PReader
     }
     trait PBuilder extends Hintable {
     def beginEntry(picklee: Any): PBuilder
     def putField(name: String, pickler: PBuilder => Unit): PBuilder
     def endEntry(): Unit
     def beginCollection(length: Int): PBuilder
     def putElement(pickler: PBuilder => Unit): PBuilder
     def endCollection(length: Int): Unit
     def result(): Pickle
     }
     https://github.com/scala/pickling
     F r
     pickle
     

     View Slide

112. https://gist.github.com/heathermiller/5760171
     example
     Output edn, Clojure’s data transfer format.
     talk to a clojure app
     toy builder implementation:
     scala> import scala.pickling._
     import scala.pickling._
     scala> import edn._
     import edn._
     scala> case class Person(name: String, kidsAges: Array[Int])
     defined class Person
     scala> val joe = Person("Joe", Array(3, 4, 13))
     joe: Person = Person(Joe,[I@3d925789)
     scala> joe.pickle.value
     res0: String = #pickling/Person { :name "Joe" :kidsAges [3, 4, 13] }
     F r
     pickle
     

     View Slide

113. Scala 2.11 as target
     G :
     P :
     1.0 release within the next few months
     SIP for Scala 2.11
     Integration with sbt, Spark, and Akka, ...
     Release 0.8.0 for Scala 2.10.2
     No support for inner classes, yet
     ScalaCheck tests
     Support for cyclic object graphs, and most Scala types
     https://github.com/scala/pickling
     Status
     

     View Slide

114. qUESTIONS
     ?
     heather.miller@epfl.ch
     @heathercmiller
     C
     

     View Slide