Metaprogramming in Scala ~ The Past and The Present ~

Transcript

1. 1.

   Metaprogramming in Scala ~ The Past and The Present ~

   Takeshi ITOH
2. 2.

   Who I am ? ▰ Graduate student ▰ Part-time engineer

   @ Robotic Biology Institute ▻ 2 years with Scala ▰ Twitter: @TaKeZo_I ▰ GitHub: @iTakeshi 2 / 43
3. 3.

   Accelerating Life Science

4. 4.

   Scope of this talk ▰ I will talk mainly about

   the concept underlying metaprogramming in Scala. ▰ I will not provide many tutorial examples to write macros. ▰ I want you to feel the “お気持ち (philosophy)” of Scala metaprogramming system. 4 / 43 主に概念について解説. チュートリアルではない. Scalaメタプロの“お気持ち”.
5. 5.

   ▰ This talk is based upon my blog post titled

   “Scalaメタプログラミング今昔物語” 5 / 43 https://itakeshi.hatenablog.com/entry/2017/12/23/002550 Related Blog Post
6. 6.

   Outline ▰ Basics of metaprogramming ▰ Metaprogramming history in Scala

   ▻ “Portability” of macros ▰ Introduction to the next generation 6 / 43 メタプロ基礎→歴史と“可搬性”→これから
7. 7.

   Basics of Metaprogramming

8. 8.

   What’s Metaprogramming? Metaprogramming is a programming technique in which computer

   programs have the ability to treat programs as the data. (from https://en.wikipedia.org/wiki/Metaprogramming) 8 / 43 メタプロとは「プログラムをデータとして扱うプロ グラム」 (wikipedia). “
9. 9.

   ▰ Reflection ▰ Abstract syntax tree (AST) ▰ Reification ▰

   Quasiquote ▰ Note that these terms are commonly used in various programming languages. 9 / 43 基本用語 Basic Terms
10. 10.

    ▰ The ability to inspect and modify the structure of

    programs. ▰ Provides necessary functions to realize certain subset of “metaprogramming”. ▰ “Runtime” and “compile-time” reflection. 10 / 43 Reflectionはプログラムの構造を解析, 変更する 機能. 実行時orコンパイル時. Reflection
11. 11.

    ▰ A data structure representing syntactic information of a program.

    ▰ Different AST definitions can completely capture syntactic structure of identical program, without any information loss. 11 / 43 ASTはプログラムの構文情報を表す木. 異なるAST定義が同等の情報を表せる. Abstract Syntax Tree (AST)
12. 12.

    ▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #1

    12 / 43 If式の例 #1 Abstract Syntax Tree (AST) trait AST { … } case class IfExpr(cond: AST, then: AST, elseOpt: Option[AST]) extends AST val sample = IfExpr(<<str.startsWith("abc")>>, <<1>>, Some(<<0>>)) IfExpr cond str.startsWith("abc") then 1 else 0
13. 13.

    ▰ Ex. if (str.startsWith("abc")) 1 else 0 ▰ AST #2

    13 / 43 If式の例 #2 Abstract Syntax Tree (AST) trait AST { … } case class ElseClause(expr: AST) extends AST case class IfExpr(cond: AST, then: AST, elseOpt: Option[ElseClause]) extends AST val sample = IfExpr(<<str.startsWith("abc")>>, <<1>>, Some(ElseClause(<<0>>))) IfExpr cond str.startsWith("abc") then 1 ElseClause 0
14. 14.

    ▰ Procedure to construct AST from given programs written in

    texts. ▰ i.e. << … >> operators in the previous slides. 14 / 43 ReificationはテキストのプログラムからASTを構 築する手続き. 前スライドの<< … >>. Reification
15. 15.

    ▰ Comprehensive notation to handle reification and AST-related manipulations. ▰

    In Scala, quasiquote is provided as a kind of StringContext, like q"0 + 1" 15 / 43 準クオートはReificationやASTを便利に取り扱う ための記法. Scalaではq"0 + 1". Quasiquote
16. 16.

    Metaprogramming History in Scala

17. 17.

    Generation 0 ▰ Until Scala 2.10 was released, there had

    not been any official metaprogramming tools. ▰ However, there were some tricks. ▻ java.lang.reflect ▻ scalap (ScalaSignature) ▻ They provided severely limited functions in runtime reflection. 17 / 43 Scala 2.10が出る前はjava.reflectかscalap. 実行時リフレクションのごく一部.
18. 18.

    18 / 43 Generation 0 Main.scala Main.class scalac java bytecode

    ScalaSignature java.lang.reflect scalap
19. 19.

    ▰ scala.reflect is available since Scala 2.10. ▰ Both runtime

    and compile-time reflection. ▰ Provides a set of API to manipulate internal AST in scalac 19 / 43 2.10からscala.reflectが使える. scalacの内部に 定義されたASTを操作するためのAPI群. Generation 1
20. 20.

    20 / 43 Generation 1 Main.scala Main.class scalac ScalaSignature scala.reflect

    (RT) scalac AST scala.reflect (CT)
21. 21.

    ▰ scala.reflect deeply depends on the AST definition in the

    scalac of Scala 2.x. ▰ This means that macros written with scala.reflect are not portable to other Scala implementations which define their original AST. 21 / 43 scala.reflectはScala 2.xのscalacの内部で利用 されるASTの定義に依存. “可搬性”がない. Generation 1: Problem
22. 22.

    ▰ “Other Scala implementations” include: ▻ Dotty ▻ IDE like

    intelliJ ▻ causes a lot of warnings because they can’t find macro- expanded Defns. ▰ Who wants to completely rewrite their macros just for an IDE…? 22 / 43 scalac以外の実装: DottyやIDEでは scala.reflectベースのマクロが動かない. Generation 1: Problem
23. 23.

    DEMO 23 / 43 Generation 1: Problem

24. 24.

    ▰ Macro annotation based on scala.meta. ▰ Use scala.meta’s AST

    definition as the “standard AST”. ▰ Before expanding macro, compiler internal AST are converted to standard AST. 24 / 43 scala.metaベースのmacro annotation. コンパイラASTを標準ASTに変換. Generation 2
25. 25.

    25 / 43 Generation 2: Standard AST Main.scala Main.class scalac

    IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta
26. 26.

    26 / 43 Generation 1 (Recap) Main.scala Main.class scalac scalac

    AST scala.reflect (CT)
27. 27.

    DEMO 27 / 43 Generation 2

28. 28.

    28 / 43 AST変換に関するコストが大きい: 変換器の実装コスト, 情報ロス, 性能劣化. Generation 2: Problem

    ▰ Implementing AST converters is a hard task. ▻ Converters must support all possible syntactic elements. ▻ Each compiler requires a tailored converter. ▰ Some information will be lost in conversion. ▰ Conversion incurs performance penalty.
29. 29.

    ▰ I was one of the top-3 contributers 29 /

    43 My Contribution
30. 30.

    Introduction to the Next Generation

31. 31.

    Generation 3 ▰ scala.macros is under development. ▰ Introducing “standard

    abstract syntax” to realize portable macros without drawbacks. 31 / 43 scala.macrosが開発中. “標準構文”により可搬 性のトレードオフ問題を解決.
32. 32.

    ▰ Thin wrapper functions to standardize apparent data structure of

    AST. ▰ Implemented as “abstract contractors” and “abstract extractors” which let raw compiler AST “act” as standard syntax. ▰ Note: these functions do not convert AST. 32 / 43 標準constractor/extractorでコンパイラASTを 「見かけ上」標準化する. 変換ではない. Generation 3: Standard Syntax
33. 33.

    33 / 43 Generation 3: Standard Syntax Main.scala Main.class scalac

    IDE meta-info intelliJ scala.macros intelliJ AST standard syntax scalac AST standard syntax
34. 34.

    34 / 43 Generation 2: Standard AST (Re) Main.scala Main.class

    scalac IDE meta-info intelliJ intelliJ AST scalac AST macro annotation standard AST scala.meta
35. 35.

    35 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax package object

    macros { trait Universe { type Term def IfApply(cnd: Term, thn: Term, els: Term): Term def IfUnapply(term: Term): Option[(Term, Term, Term)] } val universe: Universe = ??? type Term = universe.Term object TermIf { def apply(cnd: Term, thn: Term, els: Term) = universe.IfApply(cnd, thn, els) def unapply(term: Term) = universe.IfUnapply(term) } } def invert(term: macros.Term): macros.Term = term match { case macros.TermIf(c, t, e) => macros.TermIf(c, e, t) case stat => stat }
36. 36.

    36 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object scalac

    { trait Tree case class If(cnd: Tree, thn: Tree, els: Tree) extends Tree } package macros { object scalacUniverse extends Universe{ override type Term = scalac.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = scalac.If(cnd, thn, els) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case scalac.If(cnd, thn, els) => Some((cnd, thn, els)) case _ => None } } } val original: scalac.Tree = (scalac)<<if (hoge(fuga) > 0) 1 else 0>> val inverted: scalac.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1
37. 37.

    37 / 43 https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1 Generation 3: Standard Syntax object IDE

    { trait Tree case class Else(els: Tree) extends Tree case class If(cnd: Tree, thn: Tree, els: Else) extends Tree } package macros { object IDEUniverse extends macros.Universe { override type Term = IDE.Tree override def IfApply(cnd: Term, thn: Term, els: Term): Term = IDE.If(cnd, thn, IDE.Else(els)) override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match { case IDE.If(cnd, thn, IDE.Else(els)) => Some((cnd, thn, els)) case _ => None } } } val original: IDE.Tree = (IDE)<<if (hoge(fuga) > 0) 1 else 0>> val inverted: IDE.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1
38. 38.

    38 / 43 標準構文の利点: 開発コストの低下, マクロの実 行速度改善, AST情報の完全な保持. Generation 3:

    Advantages ▰ Reduce implementation cost. ▻ wrappers are far easier to implement than converters. ▰ Improve performance of macro code. ▻ no AST conversion, no additional cost. ▰ Retain full information in raw compiler AST. ▻ because scala.macros manipulates raw AST “as-is”.
39. 39.

    ▰ We still have to implement tailored wrappers for each

    Scala implementation, don’t we? ▰ YES. But the cost is far smaller than converters. ▻ We don’t have to care about meta information attached to raw AST. ▻ Wrappers are not required to cover all syntactic elements. 39 / 43 標準構文wrapperはScala実装ごとに定義する 必要? → そうだが、コストが段違いに低い. Generation 3: FAQ
40. 40.

    40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1

    to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr...
41. 41.

    40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1

    to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... macros.TermIf Generation 3 needs scalac.Term scalac.Term scalac.Term
42. 42.

    40 / 43 invertマクロに必要な標準構文と標準ASTの比較. Generation 3: FAQ if (str.startsWith("abc")) (1

    to 10) foreach { case i if i % 3 == 0 => println(i) case _ => () } else throw new RuntimeException() To invert this if expr... scala.meta.Term.{ If, Apply, Select, Name, ApplyInfix, PartialFunction, Throw, Param } scala.meta.{ Lit.{String, Int, Unit}, Template, Pat.Var.Term, Ctor.Ref.Name } Generation 2 needs
43. 43.

    41 / 43 Scalaメタプログラミングの歴史は“苦しくない可 搬性”の追求. scala.macrosへの期待. Conclusion ▰ The evolution

    of metaprogramming system in Scala can be seen as a process to pursue “portability without pain” ▰ New scala.macros is expected to be a good solution towards this goal.
44. 44.

    ▰ @DavidDudson has started developing scalagen. ▰ It will be

    a scala.meta-based static code generation tool. ▻ Also it may be the first attempt for an organized code generation tool. ▰ Although this is a private (i.e. not an scalacenter’s official) project, it is worthy of note. 42 / 43 新しい静的コード生成ツール“ScalaGen”にも注 目する価値がある. Appendix: ScalaGen
45. 45.

    Thank you! Questions and comments are welcome both in English

    and Japanese.