AUTOMATIC TYPE CLASS DERIVATION WITH SHAPELESS Shi Forward Tech Talks July 6, 2017 / Joao Azevedo The purpose of this talk is to give the basics on how to do automatic type class derivation using Shapeless. At the end of the talk, attendees should be aware of the building blocks Shapeless provides and be able to apply some of the described patterns to derive their own type classes. 

INTRODUCTION 

TYPE CLASSES A definition of behaviour in the form of operations that must be supported by a given type. A way to implement ad hoc polymorphism. 

TYPE CLASSES def mean[T](xs: Vector[T])(implicit ev: NumberLike[T]): T = ev.divide(xs.reduce(ev.plus(_, _)), xs.size) 

SHAPELESS You must be shapeless, formless, like water. When you pour water in a cup, it becomes the cup. When you pour water in a bottle, it becomes the bottle. When you pour water in a teapot, it becomes the teapot. Water can drip and it can crash. Become like water my friend. -- Bruce Lee 

SHAPELESS A type class and dependent type based generic programming library for Scala. Generic programming provides ways to exploit similarities between types to avoid repetition. case class Employee(name: String, number: Int, manager: Boolean) case class IceCream(name: String, numCherries: Int, inCone: Boolean) Sometimes, types are too specific and we want to explore similarities in their shape to avoid repetition. In the presented example, even though Employee and IceCream are different types, they share the same shape: they both contain three fields of the same types: String, Int and Boolean. 

A RANDOM EXAMPLE We start with a practical example to show some of the shapeless concepts that are relevant for automatic derivation of type classes. 

A RANDOM EXAMPLE trait Random[A] extends Function0[A] object Random { def apply[A](implicit r: Random[A]) = r() } Random[Int] // 578073111 Random[Double] // 0.48802405390668835 Random[String] // mbbHlZAam Random[Employee] // Employee(Y7F,1976420522,true) Random[IceCream] // IceCream(CywOmpJd,-335712437,false) The idea is to have a Random typeclass that is capable of producing "random" instances of a given type. 

A RANDOM EXAMPLE implicit val rInt: Random[Int] = () => util.Random.nextInt implicit val rDouble: Random[Double] = () => util.Random.nextDouble implicit val rString: Random[String] = () => util.Random.alphanumeric.take( util.Random.nextInt(10)).mkString implicit val rBoolean: Random[Boolean] = () => util.Random.nextBoolean We can start by implementing Random instances for some base types. The String implementation is obviously incorrect, but the specific implementation is not very relevant for now. We just want to make sure we have a typeclass for some base types. 

A RANDOM EXAMPLE implicit def rEmployee(implicit rs: Random[String], ri: Random[Int], rb: Random[Boolean]): Random[Employee] = () => Employee(rs(), ri(), rb()) implicit def rIceCream(implicit rs: Random[String], ri: Random[Int], rb: Random[Boolean]): Random[IceCream] = () => IceCream(rs(), ri(), rb()) Having implementations of Random for some base types, we can have implementations of Random for more complex types that base themselves on the Random implementations for the types of fields that compose those types. However, we still have two distinct implementations for Employee and IceCream even though they're remarkably similar. Can we do better? 

SHAPELESS TO THE RESCUE! shapeless allows us to do better by taking advantage of the shape of the types and exploiting similarities between shapes. 

HLISTS "hello" :: 13 :: true :: HNil : String :: Int :: Boolean :: HNil The first important concept from shapeless that is very relevant for the purpose of automatic derivation of type classes are HLists. HLists are a generic representation of products, much like Scala's built-in tuples. They're a bit more powerful than Scala's tuples because each size of tuple has an unrelated type and we can't represent 0-length tuples. HLists are similar to Scala's Lists, but, at compile time, we know both their size and the type of each element that composes the list. 

RANDOM HLISTS implicit val rHNil: Random[HNil] = () => HNil implicit def rHList[H, T <: HList]( implicit rh: Random[H], rt: Random[T]): Random[H :: T] = () => rh() :: rt() Knowing what HLists are, we can create an implementation of Random for HLists, provided that we have implementations of Random for the type of each element that the list is composed of. This already provides us with a generic implementation of Random for products. 

GENERIC Generic[Employee] // shapeless.Generic[Employee]{ // type Repr = String :: Int :: Boolean :: shapeless.HNil} = // anon$macro$16$1@3f09ba38 Generic[IceCream] // shapeless.Generic[IceCream]{ // type Repr = String :: Int :: Boolean :: shapeless.HNil} = // anon$macro$12$1@750eba8f trait Generic[T] { type Repr def to(t: T): Repr def from(r: Repr): T } shapeless provides a type class called Generic that allows us to go from an algebraic data type to its generic representation as an HList (and vice-versa). The Generic instance has a type member Repr containing the type of its generic representation. 

GENERIC val hlist = "a" :: 1 :: true :: HNil Generic[Employee].from(hlist) // Employee(a,1,true) Generic[IceCream].from(hlist) // IceCream(a,1,true) Using Generic, we can take advantage of the shape of an ADT and convert back and forth from different ADTs if they have the same Repr. 

RANDOM PRODUCTS implicit def rProduct[T]( implicit g: Generic[T], rg: Random[g.Repr]): Random[T] = () => g.from(rg()) Unfortunately this doesn't compile. Taking this into account, we can derive an implementation of Random for products if we are able to derive an implementation of Random for their generic representation (which we already did for HLists in a previous slide). Unfortunately this doesn't compile because we can't reference a type member of a parameter within the same parameter list the parameter is in, and all the implicits must go in one parameter list. 

RANDOM PRODUCTS object Generic { type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 } } implicit def rProduct[T, Repr]( implicit g: Generic.Aux[T, Repr], rg: Random[Repr]): Random[T] = () => g.from(rg()) Random[Employee] // Employee(ltdui,-443373978,false) case class Foo(a: Employee, b: String) Random[Foo] // Diverging implicit expansion! shapeless solves this problem by using the Aux pattern, which is nowadays a very common technique to expose type members as type parameters. This finally allows us to have a generic implementation of Random for products. Unfortunately, the compiler can run into cycles during implicit search, which can gives us a diverging implicit expansion error. 

LAZY Creates a macro that triggers implicit search for types wrapped in `Lazy` only once. implicit val rHNil: Random[HNil] = () => HNil implicit def rHList[H, T <: HList]( implicit rh: Lazy[Random[H]], rt: Lazy[Random[T]]): Random[H :: T] = () => rh.value() :: rt.value() implicit def rProduct[T, Repr <: HList]( implicit g: Generic.Aux[T, Repr], rg: Lazy[Random[Repr]]): Random[T] = () => g.from(rg.value()) To help us with diverging implicit expansion errors, shapeless provides us with the Lazy macro. The Lazy macro triggers the implicit search for a given implicit and if this search triggers searches for types wrapped in Lazy then these will be done only once and wrapped in a lazy val, which is returned as the corresponding value. 

RANDOM COPRODUCTS sealed trait Receptacle case class Bottle(a: Int) extends Receptacle case class Glass(a: String) extends Receptacle case class Teapot(a: Boolean) extends Receptacle Generic[Receptacle] // shapeless.Generic[Receptacle]{ // type Repr = Bottle :+: Glass :+: Teapot :+: shapeless.CNil} val g = Generic[Receptacle] g.to(Glass("a")) // Inr(Inl(Glass(a))) g.to(Bottle(1)) // Inl(Bottle(1)) g.to(Teapot(true)) // Inr(Inr(Inl(Teapot(true)))) shapeless can also help us with coproducts (i.e. sealed families of case classes). The Generic Repr of coproducts is a Coproduct instead of an HList. Coproduct is defined in terms of Inr (which doesn't have a value and thus defers to the tail) and Inl (which has a value and nothing else, thus guaranteeing that there's only one Inl in a Coproduct). 

RANDOM COPRODUCTS case class CoproductOptions[A, C <: Coproduct]( options: List[() => A] = Nil) implicit def rCNil[A]: CoproductOptions[A, CNil] = CoproductOptions[A, CNil](Nil) implicit def rCP[A, H <: A, T <: Coproduct]( implicit rh: Lazy[Random[H]], rt: Lazy[CoproductOptions[A, T]]): CoproductOptions[A, H :+: T] = CoproductOptions[A, H :+: T](rh.value :: rt.value.options) implicit def rCoproduct[T, Repr <: Coproduct]( implicit g: Generic.Aux[T, Repr], rg: Lazy[CoproductOptions[T, Repr]]): Random[T] = () => { val choices = rg.value.options choices(util.Random.nextInt(choices.length))() } 

RANDOM COPRODUCTS Random[Receptacle] // Bottle(2143179073) Random[Receptacle] // Glass(eMaxESFnD) Random[Receptacle] // Bottle(1915773318) Random[Receptacle] // Glass(4MX5xYi8v) Random[Receptacle] // Teapot(false) 

SPRAY-JSON sealed abstract class JsValue case class JsObject(fields: Map[String, JsValue]) extends JsValue case class JsArray(elements: Vector[JsValue]) extends JsValue case class JsString(value: String) extends JsValue case class JsNumber(value: BigDecimal) extends JsValue sealed trait JsBoolean extends JsValue case object JsTrue extends JsBoolean case object JsFalse extends JsBoolean case object JsNull extends JsValue So far we haven't required the names of the fields of ADTs when deriving typeclasses. In order to show how to use them, we'll be using spray-json as an example. A simplified version of the spray-json AST is defined as this. 

JSONFORMAT trait JsonFormat[T] { def read(json: JsValue): T def write(obj: T): JsValue } We want to use shapeless to derive instances of the JsonFormat type class. 

MORE SHAPELESS 

SINGLETON TYPES "bar".narrow : String("bar") // <: String 42.narrow : Int(42) // <: Int 'foo.narrow : Symbol('foo) // <: Symbol true.narrow : Boolean(true) // <: Boolean 'a ->> "bar" : String with KeyTag[Symbol('a), String] 'b ->> 42 : Int with KeyTag[Symbol('b), Int] 'c ->> true : Boolean with KeyTag[Symbol('c), Boolean] val a = implicitly[Witness[String("foo")]].value : String("foo") field[Symbol('a)]("bar") : FieldType[Symbol('a), String] field[Symbol('b)](42) : FieldType[Symbol('b), Int] field[Symbol('c)](true) : FieldType[Symbol('c), Boolean] shapeless introduces the concept of a singleton type, a construction that allows lifting a constant value to a type. The type of a value that is narrowed is a subtype of the original type, but is refined with a singleton instance of the type. The narrows get erased at runtime, but allow us to work with them at compile time. Singleton types are commonly used to add typelevel keys to a given type, and shapeless provides us with utilites to both add keys and extract keys from a tagged type. 

BACK TO HLISTS ('name ->> "foo") :: ('number ->> 42) :: ('manager ->> true) :: HNil : FieldType[Symbol('name), String] :: FieldType[Symbol('number), Int] :: FieldType[Symbol('manager), Boolean] :: HNil case class Employee(name: String, number: Int, manager: Boolean) Using the concept of FieldType previously introduced, we can extend our generic representation of a product to also include the name of the fields the type is composed of. 

implicit object HNilFormat extends JsonFormat[HNil] { def read(j: JsValue) = HNil def write(n: HNil) = JsObject() } implicit def hListFormat[Key <: Symbol, Value, Remaining <: HList]( implicit key: Witness.Aux[Key], jfh: JsonFormat[Value], jft: JsonFormat[Remaining] ) = new JsonFormat[FieldType[Key, Value] :: Remaining] { def write(hlist: FieldType[Key, Value] :: Remaining) = JsObject(jft.write(hlist.tail).asJsObject.fields + (key.value.name -> jfh.write(hlist.head))) def read(json: JsValue) = { val fields = json.asJsObject.fields val head = jfh.read(fields(key.value.name)) val tail = jft.read(json) field[Key](head) :: tail } } We can also easily derive an implementation of JsonFormat for HLists of FieldTypes. 

val employee = ('name ->> "foo") :: ('number ->> 42) :: ('manager ->> true) :: HNil employee.toJson // {"manager":true,"number":42,"name":"foo"} 

LABELLEDGENERIC LabelledGeneric[Employee] // shapeless.LabelledGeneric[Employee]{ // type Repr = String with KeyTag[Symbol with Tagged[String("name")],String // Int with KeyTag[Symbol with Tagged[String("number")],Int] :: // Boolean with KeyTag[Symbol with Tagged[String("manager")],Bo // = shapeless.LabelledGeneric$$anon$1@5832492c LabelledGeneric[IceCream] // shapeless.LabelledGeneric[Employee]{ // type Repr = String with KeyTag[Symbol with Tagged[String("name")],String // Int with KeyTag[Symbol with Tagged[String("numCherries")],In // Boolean with KeyTag[Symbol with Tagged[String("inCone")],Boo // = shapeless.LabelledGeneric$$anon$1@17a5c45a trait LabelledGeneric[T] { type Repr def to(t: T): Repr def from(r: Repr): T } shapeless provides a type class called LabelledGeneric that allows us to go from an algebraic data type to its generic representation as an HList of FieldTypes (and vice-versa). The LabelledGeneric instance has a type member Repr containing the type of its generic representation. 

LABELLEDGENERIC val hlist = ('name ->> "foo") :: ('number ->> 42) :: ('manager ->> true) :: HNil LabelledGeneric[Employee].from(hlist) // Employee(foo,42,true) LabelledGeneric[IceCream].from(hlist) // Does not compile 

LABELLEDGENERIC implicit def productFormat[T, Repr <: HList]( implicit gen: LabelledGeneric.Aux[T, Repr], sg: JsonFormat[Repr] ): JsonFormat[T] = new JsonFormat[T] { def read(j: JsValue): T = gen.from(sg.read(j)) def write(t: T): JsValue = sg.write(gen.to(t)) } Employee("foo", 42, true).toJson // {"manager":true,"number":42,"name":"foo"} Taking this into account, we can derive an implementation of JsonFormat for products if we are able to derive an implementation of JsonFormat for their generic representation (which we already did for HLists of FieldTypes in a previous slide). We leave the derivation of JsonFormats for coproducts as an exercise for the reader. 

REFERENCES The Type Astronaut's Guide to Shapeless spray-json-shapeless Shapeless for Mortals