Pickles Heather Miller heather.miller@epfl.ch : instant Generating Object-Oriented Pickler Combinators for Fast and Extensible Serialization Philipp Haller Eugene Burmako Martin Odersky OOPSLA’13, Indianapolis, IN, USA October 30th, 2013

M v

M v More and more traditional applications are moving to the cloud or to mobile devices. many applications are becoming distributed

M v More and more traditional applications are moving to the cloud or to mobile devices. many applications are becoming distributed These applications run on a variety of distributed systems

this kind of distributed system a cluster of commodity machines churning through a massive data-parallel job

insert social network of your choice here This kind of distributed system ! a smartphone interacting with a service running in the cloud

communication is central

communication is central Serialization is central to communication

communication is central interoperability between different services increasing in importance. Serialization is central to communication

communication is central interoperability between different services increasing in importance. many message formats required often within the same application Serialization is central to communication

communication is central Efficient communication required. interoperability between different services increasing in importance. many message formats required often within the same application Serialization is central to communication

solved problem? B ,isn’t serialization a

https://github.com/scala/pickling Java and others all have built-in support for serialization.

https://github.com/scala/pickling Java and others all have built-in support for serialization. why do we care?

https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?

https://github.com/scala/pickling S w! The overhead of Java serialization nearly always dominates other communication costs. “ ” Object Serialization for Marshalling Data in a Java Interface to MPI Carpenter et al. Serialization slows communication by two orders of magnitude “ ” Java and others all have built-in support for serialization. why do we care?

https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care? Java Serialization is still slow. Industry knows this. jvm- serialization- group

https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?

C ! https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?

/p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling Enter:

/p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling FAST Enter:

/p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:

/p 1 new approach to serialization Compile-time pickler generation. json binary xml protobuf or, invent your own format! https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:

/p 1 new approach to serialization Compile-time pickler generation. Customize how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:

new scala module 3 (Incubating.) Increasing adoption, community libraries, external contributors. /p 1 new approach to serialization Compile-time pickler generation. Customize how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:

No content

No content

                      

                       ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL

                       ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Some Topics Covered in the Paper Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing

                       ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Topics Covered in this presentation Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ Pickler Basics Extensibility

https://github.com/scala/pickling W ? pickling:

https://github.com/scala/pickling W ? import scala.pickling._ pickling:

https://github.com/scala/pickling W ? import scala.pickling._ import json._ pickling:

https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) pickling:

https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) scala> p.pickle res1: scala.pickling.json.JSONPickle = JSONPickle({ "tpe": "Person", "name": "John Oliver", "age": 36 }) pickling:

https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:

https://github.com/scala/pickling W ? val pckl = p.pickle import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:

https://github.com/scala/pickling W ? val pckl = p.pickle scala>  pckl.unpickle[Person] res0:  Person  =  Person(John  Oliver,36) import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ Pickler Basics Extensibility

collections: Time B r size [#elements] scala.collection.immutable.Vector ... scala-pickling Kryo 2 pickler-comb unsafe-pickler-comb

B r collections: free Memory (more is better) size [#elements] scala.collection.immutable.Vector ...

B r collections: size size [#elements] scala.collection.immutable.Vector ... scala-pickling Kryo

B r evactor: time Java runs out of memory [#events] String Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String

B r evactor: time (no java, more events) [#events] String Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String

6000 8000 10000 12000 14000 0 5 10 15 20 25 30 35 40 Number of Wikipedia Nodes Time [ms] Wikipedia Cyclic Object Graph, Pickle Only 6 0 5 10 15 20 25 30 35 40 Time [ms] W Java Kryo v2 Scala Pickling (a) B r wikipedia: time (pickle only) cyclic object graph final  class  Vertex(    val  label:  String,      var  neighbors:  List[Vertex])

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics Extensibility

https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } basics

https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() basics

https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() inserted by the framework basics

https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() imported pickle format basics

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility

https://github.com/scala/pickling W b subtyping? Example: class Position(val title: String, val person: Person) What if we want to pickle Position, and Person has subclasses? dynamically, the person field could point to an instance of a subclass of Person of course... A modularly re-usable pickler for type Person must be able to pickle all possible subtypes of Person

https://github.com/scala/pickling S b : Not all picklers are required to handle all possible subtypes. Dynamic PIcklers exist to handle subtypes. trait  DynamicPickler[T]  {    def  pickle(obj:  T,  builder:  PickleBuilder):  Unit } Contract: pickles any object of type T (including subtypes of T) the solution

https://github.com/scala/pickling S b :how it works Within pickle, select Pickler[S] based on the dynamic type of the picklee: class  Person class  Firefighter  extends  Person   picklee  match  {    case  OfClass[Person]  =>  personPickler    case  OfClass[Firefighter]  =>  ffPickler    ...    case  p:  Pickleable  =>  p.pickler    case  _  =>  RuntimePickler.for(picklee.getClass)   }

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔

GeneratingP r case class Person(name: String, age: Int) Let’s generate a pickler for Job case class Job(p: Person, title: String) Scala/pickling recursively generates and composes type-class instances that represent picklers at compile-time. essence Rather than diving into type- classes, let’s just develop some intuition...

GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Let’s generate a pickler for Job

GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] stringPickler intPickler type sought pickler Let’s generate a pickler for Job

GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job

GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job

GeneratingP r graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] Pickler[String] intPickler type sought pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job

GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] intPickler type sought pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job

GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] type sought pickler stringPickler stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job

GeneratingP r graphical Intuition Pickler[Job] type sought pickler stringPickler personPickler stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job

GeneratingP r graphical Intuition Pickler[Job] type sought pickler personPickler stringPickler intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)

GeneratingP r graphical Intuition type sought pickler jobPickler personPickler stringPickler intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔ ✔

1. Picklers are implicit values that are looked up by type (at compile time) 2. Custom picklers: implicit picklers with higher precedence than generated picklers customP r jobPickler myPersonPickler stringPickler intPickler stringPickler generated pickler custom user pickler

customP F r class  JSONPickleFormat  extends  PickleFormat  {  ...  } package  object  json  {    implicit  val  pickleFormat  =  new  JSONPickleFormat } same principle as custom picklers

Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔ ✔ ✔

an extension of pickler combinators that supports subtyping polymorphism; Contributions of the Scala/Pickling project: a framework that enables retrofitting existing types with pickling support and that supports pluggable pickle formats; S r a composable mechanism for static pickler generation; an experimental evaluation that shows that statically-generated pickler combinators can outperform state-of-the-art serialization frameworks and reduce memory usage.

qUESTIONS ? heather.miller@epfl.ch @heathercmiller C