Metaprogramming in Scala ~ The Past and The Present ~ Takeshi ITOH

Who I am ? ▰ Graduate student ▰ Part-time engineer @ Robotic Biology Institute ▻ 2 years with Scala ▰ Twitter: @TaKeZo_I ▰ GitHub: @iTakeshi 2 / 43

Accelerating Life Science

Scope of this talk ▰ I will talk mainly about the concept underlying metaprogramming in Scala. ▰ I will not provide many tutorial examples to write macros. ▰ I want you to feel the “お気持ち (philosophy)” of Scala metaprogramming system. 4 / 43 主に概念について解説. チュートリアルではない. Scalaメタプロの“お気持ち”.

▰ This talk is based upon my blog post titled “Scalaメタプログラミング今昔物語” 5 / 43 https://itakeshi.hatenablog.com/entry/2017/12/23/002550 Related Blog Post

Outline ▰ Basics of metaprogramming ▰ Metaprogramming history in Scala ▻ “Portability” of macros ▰ Introduction to the next generation 6 / 43 メタプロ基礎→歴史と“可搬性”→これから

Basics of Metaprogramming

What’s Metaprogramming? Metaprogramming is a programming technique in which computer programs have the ability to treat programs as the data. (from https://en.wikipedia.org/wiki/Metaprogramming) 8 / 43 メタプロとは「プログラムをデータとして扱うプロ グラム」 (wikipedia). “

▰ Reflection ▰ Abstract syntax tree (AST) ▰ Reification ▰ Quasiquote ▰ Note that these terms are commonly used in various programming languages. 9 / 43 基本用語 Basic Terms

▰ The ability to inspect and modify the structure of programs. ▰ Provides necessary functions to realize certain subset of “metaprogramming”. ▰ “Runtime” and “compile-time” reflection. 10 / 43 Reflectionはプログラムの構造を解析, 変更する 機能. 実行時orコンパイル時. Reflection

▰ A data structure representing syntactic information of a program. ▰ Different AST definitions can completely capture syntactic structure of identical program, without any information loss. 11 / 43 ASTはプログラムの構文情報を表す木. 異なるAST定義が同等の情報を表せる. Abstract Syntax Tree (AST)

▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #1 12 / 43 If式の例 #1 Abstract Syntax Tree (AST) trait AST { … } case class IfExpr(cond: AST, then: AST, elseOpt: Option[AST]) extends AST val sample = IfExpr(<>, <<1>>, Some(<<0>>)) IfExpr cond str.startsWith("abc") then 1 else 0

▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #2 13 / 43 If式の例 #2 Abstract Syntax Tree (AST) trait AST { … } case class ElseClause(expr: AST) extends AST case class IfExpr(cond: AST, then: AST, elseOpt: Option[ElseClause]) extends AST val sample = IfExpr(<>, <<1>>, Some(ElseClause(<<0>>))) IfExpr cond str.startsWith("abc") then 1 ElseClause 0

▰ Procedure to construct AST from given programs written in texts. ▰ i.e. << … >> operators in the previous slides. 14 / 43 ReificationはテキストのプログラムからASTを構 築する手続き. 前スライドの<< … >>. Reification

▰ Comprehensive notation to handle reification and AST-related manipulations. ▰ In Scala, quasiquote is provided as a kind of StringContext, like q"0 + 1" 15 / 43 準クオートはReificationやASTを便利に取り扱う ための記法. Scalaではq"0 + 1". Quasiquote

Metaprogramming History in Scala

Generation 0 ▰ Until Scala 2.10 was released, there had not been any official metaprogramming tools. ▰ However, there were some tricks. ▻ java.lang.reflect ▻ scalap (ScalaSignature) ▻ They provided severely limited functions in runtime reflection. 17 / 43 Scala 2.10が出る前はjava.reflectかscalap. 実行時リフレクションのごく一部.

18 / 43 Generation 0 Main.scala Main.class scalac java bytecode ScalaSignature java.lang.reflect scalap

▰ scala.reflect is available since Scala 2.10. ▰ Both runtime and compile-time reflection. ▰ Provides a set of API to manipulate internal AST in scalac 19 / 43 2.10からscala.reflectが使える. scalacの内部に 定義されたASTを操作するためのAPI群. Generation 1

20 / 43 Generation 1 Main.scala Main.class scalac ScalaSignature scala.reflect (RT) scalac AST scala.reflect (CT)

▰ scala.reflect deeply depends on the AST definition in the scalac of Scala 2.x. ▰ This means that macros written with scala.reflect are not portable to other Scala implementations which define their original AST. 21 / 43 scala.reflectはScala 2.xのscalacの内部で利用 されるASTの定義に依存. “可搬性”がない. Generation 1: Problem

▰ “Other Scala implementations” include: ▻ Dotty ▻ IDE like intelliJ ▻ causes a lot of warnings because they can’t find macro- expanded Defns. ▰ Who wants to completely rewrite their macros just for an IDE…? 22 / 43 scalac以外の実装: DottyやIDEでは scala.reflectベースのマクロが動かない. Generation 1: Problem

DEMO 23 / 43 Generation 1: Problem

▰ Macro annotation based on scala.meta. ▰ Use scala.meta’s AST definition as the “standard AST”. ▰ Before expanding macro, compiler internal AST are converted to standard AST. 24 / 43 scala.metaベースのmacro annotation. コンパイラASTを標準ASTに変換. Generation 2

25 / 43 Generation 2: Standard AST Main.scala Main.class scalac IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta

26 / 43 Generation 1 (Recap) Main.scala Main.class scalac scalac AST scala.reflect (CT)

DEMO 27 / 43 Generation 2

28 / 43 AST変換に関するコストが大きい: 変換器の実装コスト, 情報ロス, 性能劣化. Generation 2: Problem ▰ Implementing AST converters is a hard task. ▻ Converters must support all possible syntactic elements. ▻ Each compiler requires a tailored converter. ▰ Some information will be lost in conversion. ▰ Conversion incurs performance penalty.

▰ I was one of the top-3 contributers 29 / 43 My Contribution

Introduction to the Next Generation

Generation 3 ▰ scala.macros is under development. ▰ Introducing “standard abstract syntax” to realize portable macros without drawbacks. 31 / 43 scala.macrosが開発中. “標準構文”により可搬 性のトレードオフ問題を解決.

▰ Thin wrapper functions to standardize apparent data structure of AST. ▰ Implemented as “abstract contractors” and “abstract extractors” which let raw compiler AST “act” as standard syntax. ▰ Note: these functions do not convert AST. 32 / 43 標準constractor/extractorでコンパイラASTを 「見かけ上」標準化する. 変換ではない. Generation 3: Standard Syntax

33 / 43 Generation 3: Standard Syntax Main.scala Main.class scalac IDE meta-info intelliJ scala.macros intelliJ AST standard syntax scalac AST standard syntax

34 / 43 Generation 2: Standard AST (Re) Main.scala Main.class scalac IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta

35 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax package object macros { trait Universe { type Term def IfApply(cnd: Term, thn: Term, els: Term): Term def IfUnapply(term: Term): Option[(Term, Term, Term)] } val universe: Universe = ??? type Term = universe.Term object TermIf { def apply(cnd: Term, thn: Term, els: Term) = universe.IfApply(cnd, thn, els) def unapply(term: Term) = universe.IfUnapply(term) } } def invert(term: macros.Term): macros.Term = term match { case macros.TermIf(c, t, e) => macros.TermIf(c, e, t) case stat => stat }

36 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object scalac { trait Tree case class If(cnd: Tree, thn: Tree, els: Tree) extends Tree } package macros { object scalacUniverse extends Universe{ override type Term = scalac.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = scalac.If(cnd, thn, els) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case scalac.If(cnd, thn, els) => Some((cnd, thn, els)) case _ => None } } } val original: scalac.Tree = (scalac)< 0) 1 else 0>> val inverted: scalac.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1

37 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object IDE { trait Tree case class Else(els: Tree) extends Tree case class If(cnd: Tree, thn: Tree, els: Else) extends Tree } package macros { object IDEUniverse extends macros.Universe { override type Term = IDE.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = IDE.If(cnd, thn, IDE.Else(els)) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case IDE.If(cnd, thn, IDE.Else(els)) => Some((cnd, thn, els)) case _ => None } } } val original: IDE.Tree = (IDE)< 0) 1 else 0>> val inverted: IDE.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1

38 / 43 標準構文の利点: 開発コストの低下, マクロの実 行速度改善, AST情報の完全な保持. Generation 3: Advantages ▰ Reduce implementation cost. ▻ wrappers are far easier to implement than converters. ▰ Improve performance of macro code. ▻ no AST conversion, no additional cost. ▰ Retain full information in raw compiler AST. ▻ because scala.macros manipulates raw AST “as-is”.

▰ We still have to implement tailored wrappers for each Scala implementation, don’t we? ▰ YES. But the cost is far smaller than converters. ▻ We don’t have to care about meta information attached to raw AST. ▻ Wrappers are not required to cover all syntactic elements. 39 / 43 標準構文wrapperはScala実装ごとに定義する 必要? → そうだが、コストが段違いに低い. Generation 3: FAQ

40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1 to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr...

40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1 to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... macros.TermIf Generation 3 needs scalac.Term scalac.Term scalac.Term

40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1 to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... scala.meta.Term.{ If, Apply, Select, Name, ApplyInfix, PartialFunction, Throw, Param } scala.meta.{ Lit.{String, Int, Unit}, Template, Pat.Var.Term, Ctor.Ref.Name } Generation 2 needs

41 / 43 Scalaメタプログラミングの歴史は“苦しくない可 搬性”の追求. scala.macrosへの期待. Conclusion ▰ The evolution of metaprogramming system in Scala can be seen as a process to pursue “portability without pain” ▰ New scala.macros is expected to be a good solution towards this goal.

▰ @DavidDudson has started developing scalagen. ▰ It will be a scala.meta-based static code generation tool. ▻ Also it may be the first attempt for an organized code generation tool. ▰ Although this is a private (i.e. not an scalacenter’s official) project, it is worthy of note. 42 / 43 新しい静的コード生成ツール“ScalaGen”にも注 目する価値がある. Appendix: ScalaGen

Thank you! Questions and comments are welcome both in English and Japanese.