Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instant Pickles: Generating Object-Oriented Pickler Combinators for Fast and Extensible Serialization

Heather Miller
October 30, 2013

Instant Pickles: Generating Object-Oriented Pickler Combinators for Fast and Extensible Serialization

Presented at OOPSLA'13, Indianapolis, IN, USA
Full publication: http://infoscience.epfl.ch/record/187787/files/oopsla-pickling_1.pdf

Serialization or pickling, i.e., persisting runtime objects by converting them into a binary or text representation is ubiquitous in distributed programming. Pickler combinators are a popular approach from functional programming designed to alleviate some of the tedium of writing pickling code by hand, but they don't translate well to object-oriented programming due to qualities like open class hierarchies and subtyping polymorphism. Furthermore, both functional pickler combinators and Java-based serialization frameworks tend to be tied to a specific pickle format, leaving programmers no choice of how their data is persisted. In this paper, we present object-oriented pickler combinators and a framework for generating them at compile-time, designed to be the default serialization mechanism of the Scala programming language. Our framework is extensible; (1) using Scala's implicit parameters, users can add their own easily-swappable pickle format, (2) using the type class pattern, users can provide their own custom picklers to override the default behavior of the Scala pickling framework. In addition to extensibility and need for little to no boilerplate, the static generation of our OO picklers achieves a factor 6 speedup over Java Serialization, and performs on par or up to 3 times faster than popular "fast" Java serialization frameworks like Kryo.

Heather Miller

October 30, 2013
Tweet

More Decks by Heather Miller

Other Decks in Research

Transcript

  1. Pickles Heather Miller heather.miller@epfl.ch : instant Generating Object-Oriented Pickler Combinators

    for Fast and Extensible Serialization Philipp Haller Eugene Burmako Martin Odersky OOPSLA’13, Indianapolis, IN, USA October 30th, 2013
  2. M v

  3. M v More and more traditional applications are moving to

    the cloud or to mobile devices. many applications are becoming distributed
  4. M v More and more traditional applications are moving to

    the cloud or to mobile devices. many applications are becoming distributed These applications run on a variety of distributed systems
  5. this kind of distributed system a cluster of commodity machines

    churning through a massive data-parallel job
  6. insert social network of your choice here This kind of

    distributed system ! a smartphone interacting with a service running in the cloud
  7. communication is central interoperability between different services increasing in importance.

    many message formats required often within the same application Serialization is central to communication
  8. communication is central Efficient communication required. interoperability between different services

    increasing in importance. many message formats required often within the same application Serialization is central to communication
  9. https://github.com/scala/pickling S w! The overhead of Java serialization nearly always

    dominates other communication costs. “ ” Object Serialization for Marshalling Data in a Java Interface to MPI Carpenter et al. Serialization slows communication by two orders of magnitude “ ” Java and others all have built-in support for serialization. why do we care?
  10. https://github.com/scala/pickling S w! Java and others all have built-in support

    for serialization. why do we care? Java Serialization is still slow. Industry knows this. jvm- serialization- group
  11. C ! https://github.com/scala/pickling S w! Java and others all have

    built-in support for serialization. why do we care?
  12. /p 1 new approach to serialization Compile-time pickler generation. json

    binary xml protobuf or, invent your own format! https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
  13. /p 1 new approach to serialization Compile-time pickler generation. Customize

    how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
  14. new scala module 3 (Incubating.) Increasing adoption, community libraries, external

    contributors. /p 1 new approach to serialization Compile-time pickler generation. Customize how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
  15.          

                
  16.          

                 ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL
  17.          

                 ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Some Topics Covered in the Paper Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing
  18.          

                 ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Topics Covered in this presentation Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing
  19. https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name:

    String, age: Int) val p = Person("John Oliver", 36) scala> p.pickle res1: scala.pickling.json.JSONPickle = JSONPickle({ "tpe": "Person", "name": "John Oliver", "age": 36 }) pickling:
  20. https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name:

    String, age: Int) val p = Person("John Oliver", 36) unpickling:
  21. https://github.com/scala/pickling W ? val pckl = p.pickle import scala.pickling._ import

    json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:
  22. https://github.com/scala/pickling W ? val pckl = p.pickle scala>  pckl.unpickle[Person] res0:

     Person  =  Person(John  Oliver,36) import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:
  23. B r evactor: time Java runs out of memory [#events]

    String Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String
  24. B r evactor: time (no java, more events) [#events] String

    Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String
  25. 6000 8000 10000 12000 14000 0 5 10 15 20

    25 30 35 40 Number of Wikipedia Nodes Time [ms] Wikipedia Cyclic Object Graph, Pickle Only 6 0 5 10 15 20 25 30 35 40 Time [ms] W Java Kryo v2 Scala Pickling (a) B r wikipedia: time (pickle only) cyclic object graph final  class  Vertex(    val  label:  String,      var  neighbors:  List[Vertex])
  26. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for

    some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } basics
  27. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for

    some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() basics
  28. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for

    some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() inserted by the framework basics
  29. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for

    some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() imported pickle format basics
  30. https://github.com/scala/pickling W b subtyping? Example: class Position(val title: String, val

    person: Person) What if we want to pickle Position, and Person has subclasses? dynamically, the person field could point to an instance of a subclass of Person of course... A modularly re-usable pickler for type Person must be able to pickle all possible subtypes of Person
  31. https://github.com/scala/pickling S b : Not all picklers are required to

    handle all possible subtypes. Dynamic PIcklers exist to handle subtypes. trait  DynamicPickler[T]  {    def  pickle(obj:  T,  builder:  PickleBuilder):  Unit } Contract: pickles any object of type T (including subtypes of T) the solution
  32. https://github.com/scala/pickling S b :how it works Within pickle, select Pickler[S]

    based on the dynamic type of the picklee: class  Person class  Firefighter  extends  Person   picklee  match  {    case  OfClass[Person]  =>  personPickler    case  OfClass[Firefighter]  =>  ffPickler    ...    case  p:  Pickleable  =>  p.pickler    case  _  =>  RuntimePickler.for(picklee.getClass)   }
  33. GeneratingP r case class Person(name: String, age: Int) Let’s generate

    a pickler for Job case class Job(p: Person, title: String) Scala/pickling recursively generates and composes type-class instances that represent picklers at compile-time. essence Rather than diving into type- classes, let’s just develop some intuition...
  34. GeneratingP r case class Person(name: String, age: Int) case class

    Job(p: Person, title: String) graphical Intuition Let’s generate a pickler for Job
  35. GeneratingP r case class Person(name: String, age: Int) case class

    Job(p: Person, title: String) graphical Intuition Pickler[Job] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
  36. GeneratingP r case class Person(name: String, age: Int) case class

    Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
  37. GeneratingP r case class Person(name: String, age: Int) case class

    Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
  38. GeneratingP r graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] Pickler[String]

    intPickler type sought pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
  39. GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] intPickler type sought

    pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
  40. GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] type sought pickler stringPickler

    stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
  41. GeneratingP r graphical Intuition Pickler[Job] type sought pickler stringPickler personPickler

    stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
  42. GeneratingP r graphical Intuition Pickler[Job] type sought pickler personPickler stringPickler

    intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)
  43. GeneratingP r graphical Intuition type sought pickler jobPickler personPickler stringPickler

    intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)
  44. 1. Picklers are implicit values that are looked up by

    type (at compile time) 2. Custom picklers: implicit picklers with higher precedence than generated picklers customP r jobPickler myPersonPickler stringPickler intPickler stringPickler generated pickler custom user pickler
  45. customP F r class  JSONPickleFormat  extends  PickleFormat  {  ...  }

    package  object  json  {    implicit  val  pickleFormat  =  new  JSONPickleFormat } same principle as custom picklers
  46. an extension of pickler combinators that supports subtyping polymorphism; Contributions

    of the Scala/Pickling project: a framework that enables retrofitting existing types with pickling support and that supports pluggable pickle formats; S r a composable mechanism for static pickler generation; an experimental evaluation that shows that statically-generated pickler combinators can outperform state-of-the-art serialization frameworks and reduce memory usage.