$30 off During Our Annual Pro Sale. View Details »

The Compiler is your Friend: ASTs and Code Generation with Quill

The Compiler is your Friend: ASTs and Code Generation with Quill

Slides of my presentation for ScalaCon 2021 (http://www.scalacon.org/)

Quill is a Language Integrated Queries for Scala, which transforms collection-like code into SQL queries in compile-time, without any special mapping, using regular case classes and functions.

In order to transform regular Scala code into queries, Quill uses a mechanism called quotation. With this mechanism, instead of executing code immediately, the code becomes a parse tree that is transformed into an internal Abstract Syntax Tree, or AST. Quill reads the internal AST information, normalizes it, and transforms it into the SQL statement.

In this session, you will learn how Quill uses the compiler to generate safer SQL code. You will hear about how the compiler works, how to generate ASTs, how to manipulate and parse them, how to make inferences, and even perform code optimisations.

Juliano Alves

November 05, 2021
Tweet

More Decks by Juliano Alves

Other Decks in Programming

Transcript

  1. The Compiler is your Friend: ASTs and code generation with

    Quill Juliano Alves @vonjuliano juliano-alves.com
  2. Who am I? • CTO at Broad ~15 years experience

    • The cool ones Scala, Clojure, Elixir • The "vintage" ones Java, C#, Python, Ruby @vonjuliano juliano-alves.com
  3. https://broad.app

  4. None
  5. https://getquill.io http://homepages.inf.ed.ac.uk/slindley/papers/practical-theory-of-linq.pdf

  6. for { c <- cats d <- dogs if (c.name

    == d.nome && c.age > d.idade) } yield { (c.age, c.age - d.age) } quote { }
  7. SELECT c.age, c.age − d.age, FROM cat c, dog d

    WHERE c.name = d.name AND c.age > d.age
  8. Quill: Secret Sauce

  9. Languages

  10. Código noun adjective limpo (Clean code)

  11. context translation rule noun adjective noun adjective Código limpio Clean

    code Código limpo
  12. context translation rule Quill AST quote { query[Fruit].filter(f => true)

    } SELECT f.* FROM Fruit WHERE true SELECT f.* FROM Fruit WHERE (1 = 1)
  13. Code manipulation

  14. Quill: Secret Sauce

  15. Quill Secret Sauce

  16. Abstract syntax trees are data structures widely used in compilers

    to represent the structure of program code ... It often serves as an intermediate representation of the program https://en.wikipedia.org/wiki/Abstract_syntax_tree
  17. https://stackoverflow.com/questions/14790115/where-can -i-learn-about-constructing-asts-for-scala-macros

  18. scala> case class Person(name: String, age: Int) scala> val people

    = List[Person](Person("Juliano", 34)) scala> import reflect.runtime.universe._ scala> showRaw(reify(people).tree)
  19. Apply( TypeApply( Select( Ident(scala.collection.immutable.List), TermName("apply") ), List( Select( Select(Select(Ident($line3.$read), TermName("$iw")),

    TermName("$iw")), TypeName("Person"))) ), List( Apply( Select(Select(Select(Select(Ident($line3.$read), TermName("$iw")), TermName("$iw")), TermName("Person")), TermName("apply")), List(Literal(Constant("Juliano")), Literal(Constant(34))))) )
  20. Overview of Quill AST

  21. A Query represents database/collections actions.

  22. An Operation happens between operands and operators. (A || B

    && C)
  23. An Action is basically an statement.

  24. Ordering defines the order.

  25. A Value indicates the different contexts for values.

  26. A Lift represents lifted values/structures coming from outside the quotation

    scope
  27. Iterable Operation handles collections ops

  28. Option Operation knows EVERYTHING about options

  29. Other elements

  30. Complete-ish Quill AST

  31. Show me the code

  32. scala> case class Person(id: Int, name: String, age: Int) scala>

    case class Address(fk: Option[Int], street: String, number :Int) scala> val q1 = quote { query[Person] } scala> pprint.pprintln(q1.ast) Entity("Person", List())
  33. scala> val q2 = quote { query[Person].filter(_.age > 25) }

    scala> pprint.pprintln(q2.ast) Filter( Entity("Person", List()), Ident("x1"), BinaryOperation( Property(Ident("x1"), "age"), >, Constant(25) ) )
  34. scala> val q3 = quote { | query[Person].filter(_.age > 25).map(_.name)

    | } scala> pprint.pprintln(q3.ast) Map( Filter( Entity("Person", List()), Ident("x1"), BinaryOperation( Property(Ident("x1"), "age"), >, Constant(25)) ), Ident("x2"), Property(Ident("x2"), "name") )
  35. scala> val q4 = quote { | query[Person] | .leftJoin(query[Address])

    | .on((p, a) => a.fk.exists(_ == p.id)) | .map {case (p, a) => (p.name, a.flatMap(_.fk))} | } scala> pprint.pprintln(q4.ast)
  36. Map( Join( LeftJoin, Entity("Person", List()), Entity("Address", List()), Ident("p"), Ident("a"), OptionExists(Property(Ident("a"),

    "fk"), Ident("x1"), BinaryOperation(Ident("x1"), ==, Property(Ident("p"), "id"))) ), Ident("x01"), Tuple( List( Property(Property(Ident("x01"), "_1"), "name"), OptionTableFlatMap(Property(Ident("x01"), "_2"), Ident("x2"), Property(Ident("x2"), "fk")) ) ) )
  37. Tokens and Tokenizers

  38. "A Token is a sequence of characters that can be

    treated as a unit in the grammar of the programming languages" https://www.geeksforgeeks.org/introduction-of-lexical-analysis/ Keywords, identifiers or operators are tokens.
  39. // Statement.scala sealed trait Token sealed trait TagToken extends Token

    case class StringToken(string: String) extends Token { override def toString = string } case class ScalarTagToken(lift: ScalarTag) extends TagToken { override def toString = s"lift(${lift.name})" } case class QuotationTagToken(tag: QuotationTag) extends TagToken { override def toString = s"quoted(${tag.uid()})" }
  40. // Statement.scala case class ScalarLiftToken(lift: ScalarLift) extends Token { override

    def toString = s"lift(${lift.name})" } case class Statement(tokens: List[Token]) extends Token { override def toString = tokens.mkString } case class SetContainsToken(a: Token, op: Token, b: Token) extends Token { override def toString = s"${a.toString} ${op.toString} (${b.toString})" }
  41. // Ast.scala sealed trait Ast { override def toString =

    { import ... implicit def externalTokenizer: Tokenizer[External] = Tokenizer[External](_ => stmt"?") implicit val namingStrategy: NamingStrategy = io.getquill.Literal this.token.toString } }
  42. // StatementInterpolator.scala trait Tokenizer[T] { def token(v: T): Token }

    implicit def listTokenizer[T](implicit tokenize: Tokenizer[T]): Tokenizer[List[T]] = Tokenizer[List[T]] { case list => list.mkStmt() } implicit def stringTokenizer: Tokenizer[String] = Tokenizer[String] { case string => StringToken(string) } // many more implementations
  43. Let's tokenize!

  44. // SqlQuery.scala case class FlattenSqlQuery( from: List[FromContext] = List(), where:

    Option[Ast] = None, groupBy: Option[Ast] = None, orderBy: List[OrderByCriteria] = Nil, limit: Option[Ast] = None, offset: Option[Ast] = None, select: List[SelectValue], distinct: Boolean = false ) extends SqlQuery
  45. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper protected class FlattenSqlQueryTokenizerHelper(q: FlattenSqlQuery) (implicit astTokenizer:

    Tokenizer[Ast], strategy: NamingStrategy) { import q._ def distinctTokenizer = (if (distinct) "DISTINCT " else "").token def withDistinct = select match { case Nil => stmt"$distinctTokenizer*" case _ => stmt"$distinctTokenizer${select.token}" } DISTINCT $t
  46. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper def withFrom = from match {

    case Nil => withDistinct case head :: tail => val t = tail.foldLeft(stmt"${head.token}") { case (a, b: FlatJoinContext) => stmt"$a ${(b: FromContext).token}" case (a, b) => stmt"$a, ${b.token}" } stmt"$withDistinct FROM $t" } DISTINCT $t FROM $t
  47. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper def withWhere = where match {

    case None => withFrom case Some(where) => stmt"$withFrom WHERE ${where.token}" } DISTINCT $t FROM $t WHERE $t
  48. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper def withGroupBy = groupBy match {

    case None => withWhere case Some(groupBy) => stmt"$withWhere GROUP BY ${tokenizeGroupBy(groupBy)}" } DISTINCT $t FROM $t WHERE $t GROUP BY $t
  49. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper def withOrderBy = orderBy match {

    case Nil => withGroupBy case orderBy => stmt"$withGroupBy ${tokenOrderBy(orderBy)}" } protected def tokenOrderBy(criterias: List[OrderByCriteria]) (implicit astTokenizer: Tokenizer[Ast], strategy: NamingStrategy) = stmt"ORDER BY ${criterias.token}" DISTINCT $t FROM $t WHERE $t GROUP BY $t ORDER BY $t
  50. // SqlIdiom.scala def withLimitOffset = limitOffsetToken(withOrderBy).token((limit, offset)) protected def limitOffsetToken(query:

    Statement) (implicit astTokenizer: Tokenizer[Ast], strategy: NamingStrategy) = Tokenizer[(Option[Ast], Option[Ast])] { case (None, None) => query case (Some(limit), None) => stmt"$query LIMIT ${limit.token}" case (Some(limit), Some(offset)) => stmt"$query LIMIT ${limit.token} OFFSET ${offset.token}" case (None, Some(offset)) => stmt"$query OFFSET ${offset.token}" } DISTINCT $t FROM $t WHERE $t GROUP BY $t ORDER BY $t LIMIT $t
  51. // SqlIdiom.scala > FlattenSqlQueryTokenizerHelper def apply = stmt"SELECT $withLimitOffset" SELECT

    DISTINCT $t FROM $t WHERE $t GROUP BY $t ORDER BY $t LIMIT $t
  52. // SqlNormalize.scala private val normalize = (identity[Ast] _) .andThen(DemarcateExternalAliases.apply _)

    .andThen(new FlattenOptionOperation(concatBehavior).apply _) .andThen(new SimplifyNullChecks(equalityBehavior).apply _) .andThen(Normalize.apply _) // Need to do RenameProperties before ExpandJoin which normalizes-out all the tuple indexes // on which RenameProperties relies .andThen(RenameProperties.apply _) .andThen(ExpandDistinct.apply _) .andThen(NestImpureMappedInfix.apply _) .andThen(Normalize.apply _) .andThen(ExpandJoin.apply _) .andThen(ExpandMappedInfix.apply _) .andThen(Normalize.apply _)) def apply(ast: Ast) = normalize(ast)
  53. scala> case class Person(id: Int, name: String, age: Int) scala>

    case class Address(fk: Option[Int], street: String, number :Int) scala> import io.getquill._ scala> val ctx = new SqlMirrorContext(PostgresDialect, Literal) scala> import ctx._
  54. scala> val q1 = quote { query[Person] } scala> pprint.pprintln(q1.ast)

    Entity("Person", List()) scala> pprint.pprintln(ctx.run(q1).string) "SELECT x.id, x.name, x.age FROM Person x"
  55. scala> val q2 = quote { query[Person].filter(_.age > 25) }

    scala> pprint.pprintln(q2.ast) Filter( Entity("Person", List()), Ident("x1"), BinaryOperation( Property(Ident("x1"), "age"), >, Constant(25) ) ) scala> pprint.pprintln(ctx.run(q2).string) "SELECT x1.id, x1.name, x1.age FROM Person x1 WHERE x1.age > 25"
  56. scala> val q3 = quote { | query[Person].filter(_.age > 25).map(_.name)

    | } scala> pprint.pprintln(q3.ast) Map( Filter( Entity("Person", List()), Ident("x1"), BinaryOperation( Property(Ident("x1"), "age"), >, Constant(25)) ), Ident("x2"), Property(Ident("x2"), "name") )
  57. scala> pprint.pprintln(ctx.run(q3).string) "SELECT x1.name FROM Person x1 WHERE x1.age >

    25"
  58. scala> val q4 = quote { | query[Person] | .leftJoin(query[Address])

    | .on((p, a) => a.fk.exists(_ == p.id)) | .map {case (p, a) => (p.name, a.flatMap(_.fk))} | } scala> pprint.pprintln(q4.ast)
  59. Map( Join( LeftJoin, Entity("Person", List()), Entity("Address", List()), Ident("p"), Ident("a"), OptionExists(Property(Ident("a"),

    "fk"), Ident("x1"), BinaryOperation(Ident("x1"), ==, Property(Ident("p"), "id"))) ), Ident("x01"), Tuple( List( Property(Property(Ident("x01"), "_1"), "name"), OptionTableFlatMap(Property(Ident("x01"), "_2"), Ident("x2"), Property(Ident("x2"), "fk")) ) ) )
  60. scala> pprint.pprintln(ctx.run(q4).string) "SELECT p.name, a.fk FROM Person p LEFT JOIN

    Address a ON a.fk = p.id"
  61. Beta Reductions

  62. scala> val pq = quote { | query[Person].map(_.id).filter(_ > 1)

    | } scala> val aq = quote { | query[Address].map(_.fk).filter(_.exists(_ > 1)) | } scala> pprint.pprintln(pq.ast) scala> pprint.pprintln(aq.ast)
  63. Filter( Map(Entity("Person", List()), Ident("x1"), Property(Ident("x1"), "id")), Ident("x2"), BinaryOperation(Ident("x2"), >, Constant(1))

    ) Filter( Map(Entity("Address", List()), Ident("x1"), Property(Ident("x1"), "fk")), Ident("x2"), OptionExists(Ident("x2"), Ident("x3"), BinaryOperation(Ident("x3"), >, Constant(1))) )
  64. scala> pprint.pprintln(ctx.run(pq).string) "SELECT x1.id FROM Person x1 WHERE x1.id >

    1" scala> pprint.pprintln(ctx.run(aq).string) "SELECT x1.fk FROM Address x1 WHERE x1.fk > 1"
  65. https://wiki.haskell.org/Beta_reduction A beta reduction is the process of calculating a

    result from the application of a function to an expression.
  66. // FlattenOptionOperation.scala override def apply(ast: Ast): Ast = ast match

    { // ... case OptionExists(ast, alias, body) => if (containsNonFallthroughElement(body)) { val reduction = BetaReduction(body, alias -> ast) apply((IsNotNullCheck(ast) +&&+ reduction): Ast) } else { uncheckedReduction(ast, alias, body) } // ... }
  67. // BetaReduction.scala override def apply(o: OptionOperation): OptionOperation = o match

    { // ... case OptionMap(a, b, c) => OptionMap(apply(a), b, BetaReduction(replacements - b)(c)) case OptionForall(a, b, c) => OptionForall(apply(a), b, BetaReduction(replacements - b)(c)) case OptionExists(a, b, c) => OptionExists(apply(a), b, BetaReduction(replacements - b)(c)) case other => super.apply(other) }
  68. Optimizations

  69. scala> case class Person(id: Int, name: String, age: Int) scala>

    val q = quote { | (value: String) => | if (value == "drinks") | query[Person].filter(_.age >= 21) | else | query[Person] | } scala> pprint.pprintln(q.ast)
  70. Function( List(Ident("value")), If( BinaryOperation(Ident("value"), ==, Constant("drinks")), Filter( Entity("Person", List()), Ident("x1"),

    BinaryOperation( Property(Ident("x1"), "age"), >=, Constant(21)) ), Entity("Person", List()) ) )
  71. scala> pprint.pprintln(q("drinks").ast) If( BinaryOperation(Constant("drinks"), ==, Constant("drinks")), Filter( Entity("Person", List()), Ident("x1"),

    BinaryOperation( Property(Ident("x1"), "age"), >=, Constant(21)) ), Entity("Person", List()) )
  72. // TriviallyCheckable.scala def unapply(ast: Ast): Option[Ast] = ast match {

    // ... case TriviallyCheckable(one) +==+ TriviallyCheckable(two) if (one == two) => Some(Constant(true)) case TriviallyCheckable(one) +==+ TriviallyCheckable(two) if (one != two) => Some(Constant(false)) // ... } https://github.com/getquill/quill/pull/1693
  73. // BetaReduction.scala override def apply(ast: Ast): Ast = ast match

    { // ... case If(TriviallyTrueCondition(), thenClause, _) if (Messages.reduceTrivials) => apply(thenClause) case If(TriviallyFalseCondition(), _, elseClause) if (Messages.reduceTrivials) => apply(elseClause) // ... } https://github.com/getquill/quill/pull/1693
  74. scala> pprint.pprintln(q("drinks").ast) Filter( Entity("Person", List()), Ident("x1"), BinaryOperation( Property(Ident("x1"), "age"), >=,

    Constant(21)) ) scala> pprint.pprintln(ctx.run(q("drinks")).string) "SELECT x1.id, x1.name, x1.age FROM Person x1 WHERE x1.age >= 21"
  75. scala> pprint.pprintln(q("whatever").ast) Entity("Person", List()) scala> pprint.pprintln(ctx.run(q("whatever")).string) "SELECT x1.id, x1.name, x1.age

    FROM Person x1"
  76. There and back again

  77. enum Ast: def value: AnyRef case Leaf(value: AnyRef) extends Ast

    case Node(value: Map[String, AnyRef]) extends Ast
  78. import java.util.Map as JMap case class Address(street: String, number: Int)

    case class Person(name: String, address: Address) Person("Juliano", Address("Cool St", 12)).toJMap JMap( "name" -> "Juliano", "address" -> JMap( "street" -> "Cool St", "number" -> 12 ) )
  79. import java.util.Map as JMap case class Address(street: String, number: Int)

    case class Person(name: String, address: Address) Person("Juliano", Address("Cool St", 12)).toJson { "name": "Juliano", "address": { "street": "Cool St", "number": 12 } }
  80. Conclusion

  81. The Compiler is your friend Define the domain AST Code

    fitting in the rules Code manipulation Transform and build outputs Normalization Improve how the code works Optimization
  82. https://getquill.io "This is the project you are looking for"

  83. Questions?

  84. The Compiler is your Friend: ASTs and code generation with

    Quill Juliano Alves @vonjuliano juliano-alves.com Thank you!