Share
Embed
Metaprogramming in Scala ~ The Past and The Present ~
iTakeshi
March 17, 2018
Transcript
45 slides
-
Who I am ? ▰ Graduate student ▰ Part-time engineer
@ Robotic Biology Institute ▻ 2 years with Scala ▰ Twitter: @TaKeZo_I ▰ GitHub: @iTakeshi 2 / 43 -
Scope of this talk ▰ I will talk mainly about
the concept underlying metaprogramming in Scala. ▰ I will not provide many tutorial examples to write macros. ▰ I want you to feel the “お気持ち (philosophy)” of Scala metaprogramming system. 4 / 43 主に概念について解説. チュートリアルではない. Scalaメタプロの“お気持ち”. -
▰ This talk is based upon my blog post titled
“Scalaメタプログラミング今昔物語” 5 / 43 https://itakeshi.hatenablog.com/entry/2017/12/23/002550 Related Blog Post -
Outline ▰ Basics of metaprogramming ▰ Metaprogramming history in Scala
▻ “Portability” of macros ▰ Introduction to the next generation 6 / 43 メタプロ基礎→歴史と“可搬性”→これから -
What’s Metaprogramming? Metaprogramming is a programming technique in which computer
programs have the ability to treat programs as the data. (from https://en.wikipedia.org/wiki/Metaprogramming) 8 / 43 メタプロとは「プログラムをデータとして扱うプロ グラム」 (wikipedia). “ -
▰ Reflection ▰ Abstract syntax tree (AST) ▰ Reification ▰
Quasiquote ▰ Note that these terms are commonly used in various programming languages. 9 / 43 基本用語 Basic Terms -
▰ The ability to inspect and modify the structure of
programs. ▰ Provides necessary functions to realize certain subset of “metaprogramming”. ▰ “Runtime” and “compile-time” reflection. 10 / 43 Reflectionはプログラムの構造を解析, 変更する 機能. 実行時orコンパイル時. Reflection -
▰ A data structure representing syntactic information of a program.
▰ Different AST definitions can completely capture syntactic structure of identical program, without any information loss. 11 / 43 ASTはプログラムの構文情報を表す木. 異なるAST定義が同等の情報を表せる. Abstract Syntax Tree (AST) -
▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #1
12 / 43 If式の例 #1 Abstract Syntax Tree (AST) trait AST { … } case class IfExpr(cond: AST, then: AST, elseOpt: Option[AST]) extends AST val sample = IfExpr(<<str.startsWith("abc")>>, <<1>>, Some(<<0>>)) IfExpr cond str.startsWith("abc") then 1 else 0 -
▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #2
13 / 43 If式の例 #2 Abstract Syntax Tree (AST) trait AST { … } case class ElseClause(expr: AST) extends AST case class IfExpr(cond: AST, then: AST, elseOpt: Option[ElseClause]) extends AST val sample = IfExpr(<<str.startsWith("abc")>>, <<1>>, Some(ElseClause(<<0>>))) IfExpr cond str.startsWith("abc") then 1 ElseClause 0 -
▰ Procedure to construct AST from given programs written in
texts. ▰ i.e. << … >> operators in the previous slides. 14 / 43 ReificationはテキストのプログラムからASTを構 築する手続き. 前スライドの<< … >>. Reification -
▰ Comprehensive notation to handle reification and AST-related manipulations. ▰
In Scala, quasiquote is provided as a kind of StringContext, like q"0 + 1" 15 / 43 準クオートはReificationやASTを便利に取り扱う ための記法. Scalaではq"0 + 1". Quasiquote -
Generation 0 ▰ Until Scala 2.10 was released, there had
not been any official metaprogramming tools. ▰ However, there were some tricks. ▻ java.lang.reflect ▻ scalap (ScalaSignature) ▻ They provided severely limited functions in runtime reflection. 17 / 43 Scala 2.10が出る前はjava.reflectかscalap. 実行時リフレクションのごく一部. -
18 / 43 Generation 0 Main.scala Main.class scalac java bytecode
ScalaSignature java.lang.reflect scalap -
▰ scala.reflect is available since Scala 2.10. ▰ Both runtime
and compile-time reflection. ▰ Provides a set of API to manipulate internal AST in scalac 19 / 43 2.10からscala.reflectが使える. scalacの内部に 定義されたASTを操作するためのAPI群. Generation 1 -
20 / 43 Generation 1 Main.scala Main.class scalac ScalaSignature scala.reflect
(RT) scalac AST scala.reflect (CT) -
▰ scala.reflect deeply depends on the AST definition in the
scalac of Scala 2.x. ▰ This means that macros written with scala.reflect are not portable to other Scala implementations which define their original AST. 21 / 43 scala.reflectはScala 2.xのscalacの内部で利用 されるASTの定義に依存. “可搬性”がない. Generation 1: Problem -
▰ “Other Scala implementations” include: ▻ Dotty ▻ IDE like
intelliJ ▻ causes a lot of warnings because they can’t find macro- expanded Defns. ▰ Who wants to completely rewrite their macros just for an IDE…? 22 / 43 scalac以外の実装: DottyやIDEでは scala.reflectベースのマクロが動かない. Generation 1: Problem -
▰ Macro annotation based on scala.meta. ▰ Use scala.meta’s AST
definition as the “standard AST”. ▰ Before expanding macro, compiler internal AST are converted to standard AST. 24 / 43 scala.metaベースのmacro annotation. コンパイラASTを標準ASTに変換. Generation 2 -
25 / 43 Generation 2: Standard AST Main.scala Main.class scalac
IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta -
26 / 43 Generation 1 (Recap) Main.scala Main.class scalac scalac
AST scala.reflect (CT) -
28 / 43 AST変換に関するコストが大きい: 変換器の実装コスト, 情報ロス, 性能劣化. Generation 2: Problem
▰ Implementing AST converters is a hard task. ▻ Converters must support all possible syntactic elements. ▻ Each compiler requires a tailored converter. ▰ Some information will be lost in conversion. ▰ Conversion incurs performance penalty. -
▰ I was one of the top-3 contributers 29 /
43 My Contribution -
Generation 3 ▰ scala.macros is under development. ▰ Introducing “standard
abstract syntax” to realize portable macros without drawbacks. 31 / 43 scala.macrosが開発中. “標準構文”により可搬 性のトレードオフ問題を解決. -
▰ Thin wrapper functions to standardize apparent data structure of
AST. ▰ Implemented as “abstract contractors” and “abstract extractors” which let raw compiler AST “act” as standard syntax. ▰ Note: these functions do not convert AST. 32 / 43 標準constractor/extractorでコンパイラASTを 「見かけ上」標準化する. 変換ではない. Generation 3: Standard Syntax -
33 / 43 Generation 3: Standard Syntax Main.scala Main.class scalac
IDE meta-info intelliJ scala.macros intelliJ AST standard syntax scalac AST standard syntax -
34 / 43 Generation 2: Standard AST (Re) Main.scala Main.class
scalac IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta -
35 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax package object
macros { trait Universe { type Term def IfApply(cnd: Term, thn: Term, els: Term): Term def IfUnapply(term: Term): Option[(Term, Term, Term)] } val universe: Universe = ??? type Term = universe.Term object TermIf { def apply(cnd: Term, thn: Term, els: Term) = universe.IfApply(cnd, thn, els) def unapply(term: Term) = universe.IfUnapply(term) } } def invert(term: macros.Term): macros.Term = term match { case macros.TermIf(c, t, e) => macros.TermIf(c, e, t) case stat => stat } -
36 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object scalac
{ trait Tree case class If(cnd: Tree, thn: Tree, els: Tree) extends Tree } package macros { object scalacUniverse extends Universe{ override type Term = scalac.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = scalac.If(cnd, thn, els) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case scalac.If(cnd, thn, els) => Some((cnd, thn, els)) case _ => None } } } val original: scalac.Tree = (scalac)<<if (hoge(fuga) > 0) 1 else 0>> val inverted: scalac.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1 -
37 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object IDE
{ trait Tree case class Else(els: Tree) extends Tree case class If(cnd: Tree, thn: Tree, els: Else) extends Tree } package macros { object IDEUniverse extends macros.Universe { override type Term = IDE.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = IDE.If(cnd, thn, IDE.Else(els)) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case IDE.If(cnd, thn, IDE.Else(els)) => Some((cnd, thn, els)) case _ => None } } } val original: IDE.Tree = (IDE)<<if (hoge(fuga) > 0) 1 else 0>> val inverted: IDE.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1 -
38 / 43 標準構文の利点: 開発コストの低下, マクロの実 行速度改善, AST情報の完全な保持. Generation 3:
Advantages ▰ Reduce implementation cost. ▻ wrappers are far easier to implement than converters. ▰ Improve performance of macro code. ▻ no AST conversion, no additional cost. ▰ Retain full information in raw compiler AST. ▻ because scala.macros manipulates raw AST “as-is”. -
▰ We still have to implement tailored wrappers for each
Scala implementation, don’t we? ▰ YES. But the cost is far smaller than converters. ▻ We don’t have to care about meta information attached to raw AST. ▻ Wrappers are not required to cover all syntactic elements. 39 / 43 標準構文wrapperはScala実装ごとに定義する 必要? → そうだが、コストが段違いに低い. Generation 3: FAQ -
40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1
to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... -
40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1
to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... macros.TermIf Generation 3 needs scalac.Term scalac.Term scalac.Term -
40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1
to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... scala.meta.Term.{ If, Apply, Select, Name, ApplyInfix, PartialFunction, Throw, Param } scala.meta.{ Lit.{String, Int, Unit}, Template, Pat.Var.Term, Ctor.Ref.Name } Generation 2 needs -
41 / 43 Scalaメタプログラミングの歴史は“苦しくない可 搬性”の追求. scala.macrosへの期待. Conclusion ▰ The evolution
of metaprogramming system in Scala can be seen as a process to pursue “portability without pain” ▰ New scala.macros is expected to be a good solution towards this goal. -
▰ @DavidDudson has started developing scalagen. ▰ It will be
a scala.meta-based static code generation tool. ▻ Also it may be the first attempt for an organized code generation tool. ▰ Although this is a private (i.e. not an scalacenter’s official) project, it is worthy of note. 42 / 43 新しい静的コード生成ツール“ScalaGen”にも注 目する価値がある. Appendix: ScalaGen