Practical Binary with Scodec and Shapeless

The Shapeless library provides generic programming facilities for Scala. It does this by employing various techniques and language features, including singleton types, implicit witness proofs, path dependent types, and refinement types.

In this talk, we’ll look at how Shapeless is used in scodec to support conversion of binary data to Scala objects and vice versa. It will provide a brief overview of Shapeless HLists and Coproducts, as well as records and unions, and show how these pieces are used in something as mundane as binary encoding and decoding.

Presented at Scala World 2015 in Penrith, UK.

Transcript

1. Practical Binary with Scodec: Applying Typelevel Programming and Shapeless to the Mundane September 2015
2. Agenda 2 Introduction to scodec HLists Coproducts 1 2 3 4 Records & Unions Explore usage of Shapeless in scodec, to understand how type level programming can be put to practical use.
3. About Me… • Michael Pilquist (@mpilquist) • Using Scala professionally since 2008 • Primary author of scodec • Work at CCAD, LLC. (Combined Conditional Access Development) • Joint-venture between Comcast and ARRIS Group, Inc. • Build state of the art control systems that manage set-top boxes and consumer devices 3
4. scodec • Suite of Scala libraries for working with binary data • Pure functional • Compositional • http://scodec.org • http://gitter.im/scodec • Interop libraries for Cats, Scalaz, Spire, and Akka 4 scodec-bits scodec-core scodec-stream scalaz-stream shapeless
5. scodec, approximately… 5 trait Encoder[-A] { def encode(value: A): Attempt[BitVector] def sizeBound: SizeBound } trait Decoder[+A] { def decode(bits: BitVector): Attempt[DecodeResult[A]] } case class DecodeResult[+A](value: A, remainder: BitVector) trait Codec[A] extends Encoder[A] with Decoder[A] Similar to Try, \/, Or, Xor, right-biased Either
6. Introduction to scodec 6 case class Point(x: Int, y: Int) case class Line(start: Point, end: Point) case class Arrangement(lines: Vector[Line]) val arr = Arrangement(Vector( Line(Point(0, 0), Point(10, 10)), Line(Point(0, 10), Point(10, 0))))
7. Introduction to scodec 7 import scodec.Codec import scodec.codecs.implicits._ // arr: Arrangement = Arrangement(Vector(Line(Point(… val arrBinary = Codec.encode(arr).require // arrBinary: BitVector = BitVector(288 bits, 0x0000000200000000000000000000000a0000000a00000000000… val decoded = Codec[Arrangement]. decode(arrBinary).require.value // decoded: Arrangement = Arrangement(Vector(
 Line(Point(0, 0),Point(10, 10)),
 Line(Point(0, 10),Point(10, 0)))) Attempt[A] => A (throws if failure)
8. Controlling codec derivation with implicits 8 import scodec.codecs.implicits.{ implicitIntCodec => _, _ } implicit val ci = scodec.codecs.uint8 // arr: Arrangement = Arrangement(Vector(Line(Point(… val arrBinary = Codec.encode(arr).require // arrBinary: BitVector = BitVector(72 bits, 0x0200000a0a000a0a00) val result = Codec.encode( Arrangement(Vector( Line(Point(0, 0), Point(1, 1)), Line(Point(0, 0), Point(1, -1)) )) ) // result: Attempt[BitVector] = Failure(lines/1/end/y: -1 
 is less than minimum value 0 for 8-bit unsigned integer) Error contains path to problematic field All ints are encoded in 8 bits
9. Explicit codecs 9 case class Pid(value: Int) { require(value >= Pid.MinValue && value <= Pid.MaxValue) } object Pid { val MinValue = 0 val MaxValue = 8191 implicit val codec: Codec[Pid] = uint(13).xmap(Pid.apply, _.value) // Alternatively… uint(13).as[Pid] } Design-wise, scodec is optimized for defining explicit codecs Transforms a Codec[Int] => Codec[Pid] Shapeless auto-generates xmap invocation
10. Explicit codecs 10 case class TransportStreamHeader( transportErrorIndicator: Boolean, payloadUnitStartIndicator: Boolean, transportPriority: Boolean, pid: Pid, scramblingControl: Int, adaptationFieldControl: Int, continuityCounter: ContinuityCounter )
11. Explicit codecs 11 object TransportStreamHeader { implicit val codec: Codec[TransportStreamHeader] = "transport_stream_header" | fixedSizeBytes(4, ("syncByte" | constant(0x47) ) :: ("transportErrorIndicator" | bool ) :: ("payloadUnitStartIndicator" | bool ) :: ("transportPriority" | bool ) :: ("pid" | Codec[Pid] ) :: ("scramblingControl" | uint2 ) :: ("adaptationFieldControl" | uint2 ) :: ("continuityCounter" | Codec[ContinuityCounter]) ).as[TransportStreamHeader] } Case class bindings via codec transformations
12. Examples Rely Heavily on Shapeless 12 HLists Records Type class derivation Type operators Singleton types Labelled generic representations Generic representations Lazy implicit resolution
13. Heterogen(e)ous Lists 13 • Equivalent to TupleN for all N (given infinite memory and time) • More precisely, equivalent to right-nested tuples 
 (x0, (x1, … (xn, HNil)…)) • Collection-like API • head/tail, take/drop, map, flatMap, filter, etc. • Like tuples, type of each element is encoded in the type • Unlike tuples, size is encoded in the type in a way that supports abstraction over arity val xs = 1 :: "hello" :: 3 :: HNil // xs: shapeless.::[Int,shapeless.::[String,shapeless.:: [Int,shapeless.HNil]]] = 1 :: hi :: 3 :: HNil
14. Building a Codec[L] from an HList of Codecs 14 import shapeless.{ ::, HNil } import scodec.codecs._ val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil Consider an HList where each component type is a Codec[Xi]
15. Building a Codec[L] from an HList of Codecs 15 Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil 0 8 16 Let’s flip the order of the type constructors — that is, swap Codec and HCons: Codec[Int :: Unit :: Boolean :: Int :: HNil] cs.toCodec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil
16. Sequencing HLists 16 F[X0 :: X1 :: … :: Xn :: HNil] F[X0] :: F[X1] :: … :: F[Xn] :: HNil Generally, we can perform the following shape conversion for any type constructor F: …if F is an applicative functor or an invariant lax monoidal functor. • Codec is an invariant lax monoidal functor • This transformation is the primary transformation needed for building codecs of product types (hence, case classes)
17. Building Codec[L] directly 17 val c = int(6) :: ignore(4) :: bool :: int(6) // Codec[Int :: Unit :: Boolean :: Int :: HNil] This transformation is so critical, it has its own special syntax — the :: method on Codec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil
18. val c = int(6) :: ignore(4) :: bool :: int(6) Building Codec[L] directly 18 This transformation is so critical, it has its own special syntax — the :: method on Codec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil Codec[Int]
19. val c = int(6) :: ignore(4) :: bool :: int(6) Building Codec[L] directly 19 This transformation is so critical, it has its own special syntax — the :: method on Codec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil Codec[Boolean :: Int :: HNil]
20. val c = int(6) :: ignore(4) :: bool :: int(6) Building Codec[L] directly 20 This transformation is so critical, it has its own special syntax — the :: method on Codec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil Codec[Unit :: Boolean :: Int :: HNil]
21. val c = int(6) :: ignore(4) :: bool :: int(6) Building Codec[L] directly 21 This transformation is so critical, it has its own special syntax — the :: method on Codec val cs = int(6) :: ignore(4) :: bool :: int(6) :: HNil // Codec[Int] :: Codec[Unit] :: Codec[Boolean] :: Codec[Int] :: HNil Codec[Int :: Unit :: Boolean :: Int :: HNil]
22. flatPrepend 22 case class PcapHeader( ordering: ByteOrdering, versionMajor: Int, versionMinor: Int, thiszone: Int, sigfigs: Long, snaplen: Long, network: Long) implicit val pcapHeader: Codec[PcapHeader] = { ("magic_number" | byteOrdering) >>:~ { implicit ordering => ("version_major" | guint16 ) :: ("version_minor" | guint16 ) :: ("thiszone" | gint32 ) :: ("sigfigs" | guint32 ) :: ("snaplen" | guint32 ) :: ("network" | guint32 ) }}.as[PcapHeader] Lots of other combinators for building 
 Codec[L <: HList] ::, :+, :~>:, :::, 
 flatPrepend / >>:~, 
 flatAppend, 
 flatConcat
23. Transforming codecs via as combinator 23 Codec[PcapHeader] case class PcapHeader( ordering: ByteOrdering, versionMajor: Int, versionMinor: Int, thiszone: Int, sigfigs: Long, snaplen: Long, network: Long) Codec[ ByteOrdering :: Int :: Int :: Int :: Long :: Long :: Long :: HNil] .as[PcapHeader] • Converts a Codec[L <: HList] to a Codec[CC] if L is the generic representation of the case class • i.e., L is an HList of the component types of the case class • Supports single element case classes • Supports automatic removal of Unit values case class Point(x: Int, y: Int) val c: Codec[Unit :: Int :: Unit :: Int :: HNil] = ??? c.as[Point]
24. Generic Representations of Case Classes 24 • Converting a case class to/from its generic representation is accomplished by using Shapeless’s Generic trait Generic[T] { type Repr def to(t : T) : Repr def from(r : Repr) : T } • Instances of Generic are acquired via the apply method in the companion 
 object Generic { type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 } def apply[T](implicit gen: Generic[T]): Aux[T, gen.Repr] = gen implicit def materialize[T, R]: Aux[T, R] = macro GenericMacros.materialize[T, R] } Representation is specified as a type member Implemented with an implicit macro
25. Coproducts 25 • Equivalent to right-nested eithers
 Either[X0, Either[X1, … Either[Xn-1, Xn]…]] • Collection-like API
 head/tail, take/drop, map, flatMap, filter, etc. • Dual to HLists (products) import shapeless.{ :+:, Coproduct, CNil } type Num = Int :+: Long :+: Float :+: Double :+: CNil val Num = Coproduct[Num] Num(1) // res0: Num = 1 Num(2L) // res1: Num = 2 res3.select[Float] // Option[Float] = None res3.select[Double] // Option[Double] = Some(4.0) Num(3.0f) // res2: Num = 3.0 Num(4.0d) // res3: Num = 4.0
26. Binary Unions 26 type ISB = Int :+: String :+: Boolean :+: CNil val ISB = Coproduct[ISB] val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). choice codec.encode(ISB(1)).require // BitVector(32 bits, 0x00000001) codec.encode(ISB("Hi")).require // BitVector(48 bits, 0x000000024869) codec.encode(ISB(true)).require // BitVector(1 bit, 0x8) • Coproduct codecs represent binary unions • Choice combinator describes an untagged union • Decoding performed by using first successful result when decoding with each of the component codecs
27. Discriminated Binary Unions 27 type ISB = Int :+: String :+: Boolean :+: CNil val ISB = Coproduct[ISB] val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedByIndex(uint8) codec.encode(ISB(1)).require // BitVector(40 bits, 0x0000000001) codec.encode(ISB("Hi")).require // BitVector(56 bits, 0x01000000024869) codec.encode(ISB(true)).require // BitVector(9 bits, 0x028) • Also supports discriminated binary unions • The uint8 is used as the discriminator • 0 → int32 • 1 → utf8_32 • 2 → bool • Not typical to discriminate by coproduct component index
28. Discriminated Binary Unions with Explicit Discriminators 28 type ISB = Int :+: String :+: Boolean :+: CNil val ISB = Coproduct[ISB] val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "s", "b")) codec.encode(ISB(1)).require // BitVector(40 bits, 0x6900000001) codec.encode(ISB("Hi")).require // BitVector(56 bits, 0x73000000024869) codec.encode(ISB(true)).require // BitVector(9 bits, 0x628) • More typical to provide explicit discriminator values for each component type • Sized collection of discriminator values passed to the using method
29. Discriminated Binary Unions with Explicit Discriminators 29 val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "b", "s"))
30. Discriminated Binary Unions with Explicit Discriminators 30 val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "b", "s")) CoproductCodecBuilder[ String :+: Boolean :+: CNil, Codec[String] :: Codec[Boolean] :: HNil, String :+: Boolean :+: CNil ]
31. Discriminated Binary Unions with Explicit Discriminators 31 val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "b", "s")) CoproductCodecBuilder[ Int :+: String :+: Boolean :+: CNil, Codec[Int] :: Codec[String] :: Codec[Boolean] :: HNil, Int :+: String :+: Boolean :+: CNil ]
32. Discriminated Binary Unions with Explicit Discriminators 32 val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "b", "s")) CoproductCodecBuilder[ Int :+: String :+: Boolean :+: CNil, Codec[Int] :: Codec[String] :: Codec[Boolean] :: HNil, Int :+: String :+: Boolean :+: CNil ]#NeedDiscriminators[String]
33. Discriminated Binary Unions with Explicit Discriminators 33 val codec: Codec[ISB] = (int32 :+: utf8_32 :+: bool). discriminatedBy(fixedSizeBytes(1, utf8)). using(Sized("i", "s", "b")) using(Sized("i", "s", "b")) (int32 :+: utf8_32 :+: bool).
34. Generic Representations of Sealed Class Hierarchies 34 sealed trait Tuning object Tuning { case class BySourceId(value: Int) extends Tuning case class ByPid( frequency: Frequency, modulation: ModulationType, pid: Pid ) extends Tuning case class ByProgram( frequency: Frequency, modulation: ModulationType, programNumber: ProgramNumber ) extends Tuning } Generic[Tuning]{ type Repr = Tuning.BySourceId :+: Tuning.ByPid :+: Tuning.ByProgram :+: CNil } A sealed class hierarchy can be represented generically as a coproduct of all concrete leaf types
35. Codec for a Sealed Class Hierarchy 35 val frequency: Codec[Frequency] = ??? val modulation: Codec[ModulationType] = ??? val pid: Codec[Pid] = ??? val program: Codec[ProgramNumber] = ??? val bySourceId: Codec[Tuning.BySourceId] = uint16.as[Tuning.BySourceId] val byPid: Codec[Tuning.ByPid] = (frequency :: modulation :: ignore(3) :: pid).as[Tuning.ByPid] val byProgram: Codec[Tuning.ByProgram] = (frequency :: modulation :: program).as[Tuning.ByProgram] val tuning: Codec[Tuning] = (bySourceId :+: byPid :+: byProgram). discriminatedBy(uint8).using(Sized(0, 1, 2)).as[Tuning] We can combine manual coproduct codecs with the as combinator and the Generic[Tuning] to create a Codec[Tuning]
36. Codec for a Sealed Class Hierarchy 36 val tuning: Codec[Tuning] = (bySourceId :+: byPid :+: byProgram). discriminatedBy(uint8).using(Sized(0, 1, 2)).as[Tuning] • Will only compile if the there’s exactly 1 codec for each concrete leaf subtype of Tuning • Order of codecs is irrelevant — any permutation will compile • Signal-to-noise ratio: • Signal: codec for each subtype, codec for discriminator, discriminator value for each subtype • Noise: construction of Codec[C <: Coproduct], construction of sized discriminator collection, transformation to Codec[Tuning] • We can do better, but we’ll need more constructs from Shapeless first…
37. Records, Unions, and Labelled Generics 37 • We can add metadata to the component types in both products and coproducts • Conceptually, we’ll add a unique label to each component type • The Labelled Generic representation of a: • case class: record with labels = field names • sealed class hierarchy: union with labels = leaf class names Int :: String :: Boolean :: HNil Int :+: String :+: Boolean :+: CNil IntFoo :: StringBar :: BooleanBaz :: HNil IntFoo :+: StringBar :+: BooleanBaz :+: CNil
38. Improving SNR for Codec[Tuning] 38 val tuning: Codec[Tuning] = (bySourceId :+: byPid :+: byProgram). discriminatedBy(uint8).using(Sized(0, 1, 2)).as[Tuning] • Signal-to-noise ratio: • Signal: codec for each subtype, codec for discriminator, discriminator value for each subtype • Noise: construction of Codec[C <: Coproduct], construction of sized discriminator collection, transformation to Codec[Tuning] • How can we use records, unions, and labelled generics to improve SNR?
39. Improving SNR for Codec[Tuning] 39 implicit val bySourceId: Codec[Tuning.BySourceId] = uint16.as[Tuning.BySourceId] implicit val byPid: Codec[Tuning.ByPid] = (frequency :: modulation :: ignore(3) :: pid).as[Tuning.ByPid] implicit val byProgram: Codec[Tuning.ByProgram] = (frequency :: modulation :: program).as[Tuning.ByProgram] val tuning: Codec[Tuning] = Codec.coproduct[Tuning]. discriminatedBy(uint8).using(Sized(0, 1, 2)) The Codec[C <: Coproduct] can be automatically generated, if for each concrete leaf type Xi , there’s an implicit Codec[Xi] in scope
40. Improving SNR for Codec[Tuning] 40 val tuning: Codec[Tuning] = Codec.coproduct[Tuning]. discriminatedBy(uint8).using(Sized(0, 1, 2)) CoproductCodecBuilder[ Tuning.ByPidByPid :+: Tuning.ByProgramByProgram :+: Tuning.BySourceIdBySourceId :+: CNil, Codec[Tuning.ByPidByPid] :: Codec[Tuning.ByProgramByProgram] :: Codec[Tuning.BySourceIdBySourceId] :: HNil, Tuning ]
41. Improving SNR for Codec[Tuning] 41 val tuning: Codec[Tuning] = Codec.coproduct[Tuning]. discriminatedBy(uint8).using(Sized(0, 1, 2)) CoproductCodecBuilder[ Tuning.ByPidByPid :+: Tuning.ByProgramByProgram :+: Tuning.BySourceIdBySourceId :+: CNil, Codec[Tuning.ByPidByPid] :: Codec[Tuning.ByProgramByProgram] :: Codec[Tuning.BySourceIdBySourceId] :: HNil, Tuning ]#NeedDiscriminators[Int]
42. Improving SNR for Codec[Tuning] 42 val tuning: Codec[Tuning] = Codec.coproduct[Tuning]. discriminatedBy(uint8).using(Sized(0, 1, 2)) ❌ Bug - the order of the coproduct has changed! CoproductCodecBuilder[ Tuning.ByPidByPid :+: Tuning.ByProgramByProgram :+: Tuning.BySourceIdBySourceId :+: CNil, Codec[Tuning.ByPidByPid] :: Codec[Tuning.ByProgramByProgram] :: Codec[Tuning.BySourceIdBySourceId] :: HNil, Tuning ]#NeedDiscriminators[Int]
43. Improving SNR for Codec[Tuning] 43 val tuning: Codec[Tuning] = Codec.coproduct[Tuning]. discriminatedBy(uint8). using( ('BySourceId ->> 0) :: ('ByPid ->> 1) :: ('ByProgram ->> 2) :: HNil) Addressed by using a record to specify each discriminator! • Signal-to-noise evaluation: • Noise: construction of Codec[C <: Coproduct], construction of sized discriminator collection, transformation to Codec[Tuning] • Noise: sized discriminator collection now duplicates type names! • We can do better different…
44. Improving SNR for Codec[Tuning] 44 implicit val d = Discriminated[Tuning, Int](uint8) implicit val d0 = d.bind[Tuning.BySourceId](0) implicit val d1 = d.bind[Tuning.ByPid](1) implicit val d2 = d.bind[Tuning.ByProgram](2) val tuning: Codec[Tuning] = Codec.coproduct[Tuning].auto Use implicits to associate discriminator values with concrete leaf types
45. Improving SNR for Codec[Tuning] 45 implicit val d = Discriminated[Tuning, Int](uint8) implicit val d0 = d.bind[Tuning.BySourceId](0) implicit val d1 = d.bind[Tuning.ByPid](1) implicit val d2 = d.bind[Tuning.ByProgram](2) val tuning: Codec[Tuning] = Codec.coproduct[Tuning].auto Use implicits to associate discriminator values with concrete leaf types Witnesses that subtypes of Tuning are discriminated by integers, represented in binary as a uint8
46. Improving SNR for Codec[Tuning] 46 implicit val d = Discriminated[Tuning, Int](uint8) implicit val d0 = d.bind[Tuning.BySourceId](0) implicit val d1 = d.bind[Tuning.ByPid](1) implicit val d2 = d.bind[Tuning.ByProgram](2) val tuning: Codec[Tuning] = Codec.coproduct[Tuning].auto Use implicits to associate discriminator values with concrete leaf types Constructs a new Discriminator, which witnesses that the BySourceId type is discriminated against other subtypes of Tuning by the integer 0, when the discriminator type is an Int Discriminator[Tuning, Tuning.BySourceId, Int]
47. Improving SNR for Codec[Tuning] 47 implicit val d = Discriminated[Tuning, Int](uint8) implicit val d0 = d.bind[Tuning.BySourceId](0) implicit val d1 = d.bind[Tuning.ByPid](1) implicit val d2 = d.bind[Tuning.ByProgram](2) val tuning: Codec[Tuning] = Codec.coproduct[Tuning].auto // Alternatively… val tuning: Codec[Tuning] = Codec[Tuning] Use implicits to associate discriminator values with concrete leaf types • Signal-to-noise evaluation: • Noise: construction of Codec[C <: Coproduct], construction of sized discriminator collection, transformation to Codec[Tuning] • Noise: implicit definitions and implicit scope management • scope issue can be mitigated by defining in companions
48. Parting Thoughts • Lots more Shapeless usage in scodec: automatic derivation, cached implicit instances, scodec.Transform type class • Scodec greatly simplifies working with binary data, in large part due to abstractions provided by Shapeless • Shapeless is useful in many disciplines — not limited to academic interests of language geeks! • Thanks to Miles Sabin, Travis Brown, and the Shapeless community for help with scodec design and for the inspiration to reduce my boilerplate 48
49. Bonus Slides
50. Sequencing HLists 50 F[X0 :: X1 :: … :: Xn :: HNil] F[X0] :: F[X1] :: … :: F[Xn] :: HNil Generally, we can perform the following shape conversion for any type constructor F: …if F is an applicative functor or an invariant lax monoidal functor.
51. Invariant Sequencing of HLists 51 def isequence[F[_], L <: HList, M <: HList](l: L)(implicit ??? ): F[M] = { ??? }
52. Invariant Sequencing of HLists 52 def isequence[F[_], L <: HList, M <: HList](l: L)(implicit im: InvariantMonoidal[F] ): F[M] = { ??? } Constrain F to just those type constructors with invariant lax monoidal functors
53. Invariant Sequencing of HLists 53 def isequence[F[_], L <: HList, M <: HList](l: L)(implicit im: InvariantMonoidal[F], utcc: *->*[F]#λ[L] ): F[M] = { ??? } Unary type class constraint that witnesses every element of L is an F[?]
54. Invariant Sequencing of HLists 54 def isequence[F[_], L <: HList, M <: HList](l: L)(implicit im: InvariantMonoidal[F], utcc: *->*[F]#?[L], folder: RightFolder.Aux[L, F[HNil], ISequenceFolder.type, F[M]] ): F[M] = { l.foldRight(im.pure(HNil: HNil))(ISequenceFolder) } object ISequenceFolder extends Poly2 { implicit def caseInvariantMonoidal[F[_], A, B <: HList]( implicit im: InvariantMonoidal[F] ) = at[F[A], F[B]] { (fa, fb) => val zipped: F[(A, B)] = im.zip(fa, fb) im.xmap(zipped)(ab => ab._1 :: ab._2)(abl => (abl.head, abl.tail)) } } Fold over L, starting with an F[HNil], combining each F[Xi] with an accumulated F[K <: HList]
55. Generic Representations of Enumerations 55 sealed trait ModulationType object ModulationType { case object Qam64 extends ModulationType case object Qam256 extends ModulationType } Generic[ModulationType]{ type Repr = ModulationType.Qam256.type :+: ModulationType.Qam64.type :+: CNil } The generic representation of an enumeration is a coproduct of the singleton type of each value
56. Enumeration Codecs 56 sealed trait ModulationType object ModulationType { case object Qam64 extends ModulationType case object Qam256 extends ModulationType } val modulationType: Codec[ModulationType] = { (provide(ModulationType.Qam64) :+: provide(ModulationType.Qam256) ). discriminatedBy(uint8). using(Sized(1, 2)). as[ModulationType] } We can combine manual coproduct codecs with the 'as' combinator and the Generic[ModulationType] to create a Codec[ModulationType]
57. Records and Unions 57 • How can we represent a type along with its associated label in the type system? Phantom types! type FieldType[K, +V] = V with KeyTag[K, V] trait KeyTag[K, +V]
 • Labels are (approximately) expressed as the singleton type of a Symbol("label") IntFoo :: StringBar :: BooleanBaz :: HNil FieldType[Symbol("Foo"), Int] :: FieldType[Symbol("Bar"), String] :: FieldType[Symbol("Baz"), Boolean] :: HNil
58. Records 58 • A product of FieldType[Ki, Vi] is called a record • Constructing a record is done by building an HList of fields, where each field is constructed using the ->> operator: import shapeless.syntax.singleton._ val car = ('make ->> Make("Tesla")) :: ('model ->> Model("S")) :: ('year ->> Year(2015)) :: HNil • In addition to HList operations, there are some special record operations car('make) // res0: Make = Make(Tesla) car('model) // res1: Model = Model(S) car('year) // res2: Year = Year(2015) car + ('year ->> Year(2016)) // res3: … = Make(Tesla) :: Model(S) :: Year(2016) :: HNil
59. Unions 59 • A coproduct of FieldType[Ki, Vi] is called a discriminated union • Constructing a discriminated union is done by building a Coproduct of fields, where each field is constructed using the ->> operator:
 import shapeless.union._ import shapeless.syntax.singleton._ type U = Union.`'i -> Int, 's -> String, 'b -> Boolean`.T val U = Coproduct[U] val x = U('i ->> 1) • In addition to Coproduct operations, there are some special union operations x('i) // res0: Option[Int] = Some(1) x('s) // res1: Option[String] = None x('b) // res2: Option[Boolean] = None
60. Labelled Generic Representations of Case Classes 60 • A case class can be represented generically as a record of its component fields — known as the “labelled generic representation” case class Car(make: Make, model: Model, year: Year) val car = ('make ->> Make("Tesla")) :: ('model ->> Model("S")) :: ('year ->> Year(2015)) :: HNil • A sealed class hierarchy can be represented generically as a coproduct of fields, where the field keys correspond to the concrete leaf types • Converting a value to/from its labelled generic representation is accomplished by using LabelledGeneric instead of Generic • Same API as Generic — implicit materialization is built off of Generic macro and DefaultSymbolicLabelling type operator