Metaprogramming in Scala ~ The Past and The Present ~

Transcript

1. Metaprogramming in Scala
   ~ The Past and The Present ~
   Takeshi ITOH
   

   View Slide

2. Who I am ?
   ▰ Graduate student
   ▰ Part-time engineer @ Robotic Biology Institute
   ▻ 2 years with Scala
   ▰ Twitter: @TaKeZo_I
   ▰ GitHub: @iTakeshi
   2 / 43
   

   View Slide

3. Accelerating Life Science
   

   View Slide

4. Scope of this talk
   ▰ I will talk mainly about the concept
   underlying metaprogramming in Scala.
   ▰ I will not provide many tutorial examples to
   write macros.
   ▰ I want you to feel the “お気持ち (philosophy)”
   of Scala metaprogramming system.
   4 / 43
   主に概念について解説. チュートリアルではない.
   Scalaメタプロの“お気持ち”.
   

   View Slide

5. ▰ This talk is based upon my blog post titled
   “Scalaメタプログラミング今昔物語”
   5 / 43
   https://itakeshi.hatenablog.com/entry/2017/12/23/002550
   Related Blog Post
   

   View Slide

6. Outline
   ▰ Basics of metaprogramming
   ▰ Metaprogramming history in Scala
   ▻ “Portability” of macros
   ▰ Introduction to the next generation
   6 / 43
   メタプロ基礎→歴史と“可搬性”→これから
   

   View Slide

7. Basics of
   Metaprogramming
   

   View Slide

8. What’s Metaprogramming?
   Metaprogramming is a programming technique
   in which computer programs have the ability to
   treat programs as the data.
   (from https://en.wikipedia.org/wiki/Metaprogramming)
   8 / 43
   メタプロとは「プログラムをデータとして扱うプロ
   グラム」 (wikipedia).
   “
   

   View Slide

9. ▰ Reflection
   ▰ Abstract syntax tree (AST)
   ▰ Reification
   ▰ Quasiquote
   ▰ Note that these terms are commonly used in
   various programming languages.
   9 / 43
   基本用語
   Basic Terms
   

   View Slide

10. ▰ The ability to inspect and modify the structure
    of programs.
    ▰ Provides necessary functions to realize
    certain subset of “metaprogramming”.
    ▰ “Runtime” and “compile-time” reflection.
    10 / 43
    Reflectionはプログラムの構造を解析, 変更する
    機能. 実行時orコンパイル時.
    Reflection
    

    View Slide

11. ▰ A data structure representing syntactic
    information of a program.
    ▰ Different AST definitions can completely
    capture syntactic structure of identical
    program, without any information loss.
    11 / 43
    ASTはプログラムの構文情報を表す木.
    異なるAST定義が同等の情報を表せる.
    Abstract Syntax Tree (AST)
    

    View Slide

12. ▰ Ex. if (str.startsWith("abc")) 1 else 0
    ▰ AST #1
    12 / 43
    If式の例 #1
    Abstract Syntax Tree (AST)
    trait AST { … }
    case class IfExpr(cond: AST, then: AST, elseOpt: Option[AST]) extends AST
    val sample = IfExpr(<>, <<1>>, Some(<<0>>))
    IfExpr
    cond
    str.startsWith("abc")
    then
    1
    else
    0
    

    View Slide

13. ▰ Ex. if (str.startsWith("abc")) 1 else 0
    ▰ AST #2
    13 / 43
    If式の例 #2
    Abstract Syntax Tree (AST)
    trait AST { … }
    case class ElseClause(expr: AST) extends AST
    case class IfExpr(cond: AST, then: AST, elseOpt: Option[ElseClause]) extends AST
    val sample = IfExpr(<>, <<1>>, Some(ElseClause(<<0>>)))
    IfExpr
    cond
    str.startsWith("abc")
    then
    1
    ElseClause
    0
    

    View Slide

14. ▰ Procedure to construct AST from given
    programs written in texts.
    ▰ i.e. << … >> operators in the previous slides.
    14 / 43
    ReificationはテキストのプログラムからASTを構
    築する手続き. 前スライドの<< … >>.
    Reification
    

    View Slide

15. ▰ Comprehensive notation to handle reification
    and AST-related manipulations.
    ▰ In Scala, quasiquote is provided as a kind of
    StringContext, like q"0 + 1"
    15 / 43
    準クオートはReificationやASTを便利に取り扱う
    ための記法. Scalaではq"0 + 1".
    Quasiquote
    

    View Slide

16. Metaprogramming History
    in Scala
    

    View Slide

17. Generation 0
    ▰ Until Scala 2.10 was released, there had not
    been any official metaprogramming tools.
    ▰ However, there were some tricks.
    ▻ java.lang.reflect
    ▻ scalap (ScalaSignature)
    ▻ They provided severely limited functions in runtime
    reflection.
    17 / 43
    Scala 2.10が出る前はjava.reflectかscalap.
    実行時リフレクションのごく一部.
    

    View Slide

18. 18 / 43
    Generation 0
    Main.scala Main.class
    scalac
    java bytecode
    ScalaSignature
    java.lang.reflect
    scalap
    

    View Slide

19. ▰ scala.reflect is available since Scala 2.10.
    ▰ Both runtime and compile-time reflection.
    ▰ Provides a set of API to manipulate internal
    AST in scalac
    19 / 43
    2.10からscala.reflectが使える. scalacの内部に
    定義されたASTを操作するためのAPI群.
    Generation 1
    

    View Slide

20. 20 / 43
    Generation 1
    Main.scala Main.class
    scalac
    ScalaSignature scala.reflect (RT)
    scalac AST scala.reflect (CT)
    

    View Slide

21. ▰ scala.reflect deeply depends on the AST
    definition in the scalac of Scala 2.x.
    ▰ This means that macros written with
    scala.reflect are not portable to other
    Scala implementations which define their
    original AST.
    21 / 43
    scala.reflectはScala 2.xのscalacの内部で利用
    されるASTの定義に依存. “可搬性”がない.
    Generation 1: Problem
    

    View Slide

22. ▰ “Other Scala implementations” include:
    ▻ Dotty
    ▻ IDE like intelliJ
    ▻ causes a lot of warnings because they can’t find macro-
    expanded Defns.
    ▰ Who wants to completely rewrite their
    macros just for an IDE…?
    22 / 43
    scalac以外の実装: DottyやIDEでは
    scala.reflectベースのマクロが動かない.
    Generation 1: Problem
    

    View Slide

23. DEMO
    23 / 43
    Generation 1: Problem
    

    View Slide

24. ▰ Macro annotation based on scala.meta.
    ▰ Use scala.meta’s AST definition as the
    “standard AST”.
    ▰ Before expanding macro, compiler internal
    AST are converted to standard AST.
    24 / 43
    scala.metaベースのmacro annotation.
    コンパイラASTを標準ASTに変換.
    Generation 2
    

    View Slide

25. 25 / 43
    Generation 2: Standard AST
    Main.scala Main.class
    scalac
    IDE meta-info
    intelliJ
    intelliJ AST
    scalac AST
    macro annotation
    standard AST
    scala.meta
    

    View Slide

26. 26 / 43
    Generation 1 (Recap)
    Main.scala Main.class
    scalac
    scalac AST scala.reflect (CT)
    

    View Slide

27. DEMO
    27 / 43
    Generation 2
    

    View Slide

28. 28 / 43
    AST変換に関するコストが大きい:
    変換器の実装コスト, 情報ロス, 性能劣化.
    Generation 2: Problem
    ▰ Implementing AST converters is a hard task.
    ▻ Converters must support all possible syntactic
    elements.
    ▻ Each compiler requires a tailored converter.
    ▰ Some information will be lost in conversion.
    ▰ Conversion incurs performance penalty.
    

    View Slide

29. ▰ I was one of the top-3 contributers
    29 / 43
    My Contribution
    

    View Slide

30. Introduction to the
    Next Generation
    

    View Slide

31. Generation 3
    ▰ scala.macros is under development.
    ▰ Introducing “standard abstract syntax” to
    realize portable macros without drawbacks.
    31 / 43
    scala.macrosが開発中. “標準構文”により可搬
    性のトレードオフ問題を解決.
    

    View Slide

32. ▰ Thin wrapper functions to standardize
    apparent data structure of AST.
    ▰ Implemented as “abstract contractors” and
    “abstract extractors” which let raw compiler
    AST “act” as standard syntax.
    ▰ Note: these functions do not convert AST.
    32 / 43
    標準constractor/extractorでコンパイラASTを
    「見かけ上」標準化する. 変換ではない.
    Generation 3: Standard Syntax
    

    View Slide

33. 33 / 43
    Generation 3: Standard Syntax
    Main.scala Main.class
    scalac
    IDE meta-info
    intelliJ
    scala.macros
    intelliJ AST standard syntax
    scalac AST standard syntax
    

    View Slide

34. 34 / 43
    Generation 2: Standard AST (Re)
    Main.scala Main.class
    scalac
    IDE meta-info
    intelliJ
    intelliJ AST
    scalac AST
    macro annotation
    standard AST
    scala.meta
    

    View Slide

35. 35 / 43
    https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1
    Generation 3: Standard Syntax
    package object macros {
    trait Universe {
    type Term
    def IfApply(cnd: Term, thn: Term, els: Term): Term
    def IfUnapply(term: Term): Option[(Term, Term, Term)]
    }
    val universe: Universe = ???
    type Term = universe.Term
    object TermIf {
    def apply(cnd: Term, thn: Term, els: Term) = universe.IfApply(cnd, thn, els)
    def unapply(term: Term) = universe.IfUnapply(term)
    }
    }
    def invert(term: macros.Term): macros.Term = term match {
    case macros.TermIf(c, t, e) => macros.TermIf(c, e, t)
    case stat => stat
    }
    

    View Slide

36. 36 / 43
    https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1
    Generation 3: Standard Syntax
    object scalac {
    trait Tree
    case class If(cnd: Tree, thn: Tree, els: Tree) extends Tree
    }
    package macros {
    object scalacUniverse extends Universe{
    override type Term = scalac.Tree
    override def IfApply(cnd: Term, thn: Term, els: Term): Term =
    scalac.If(cnd, thn, els)
    override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match {
    case scalac.If(cnd, thn, els) => Some((cnd, thn, els))
    case _ => None
    }
    }
    }
    val original: scalac.Tree = (scalac)< 0) 1 else 0>>
    val inverted: scalac.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1
    

    View Slide

37. 37 / 43
    https://gist.github.com/iTakeshi/f3aba5bb319693eef301923bc00755e1
    Generation 3: Standard Syntax
    object IDE {
    trait Tree
    case class Else(els: Tree) extends Tree
    case class If(cnd: Tree, thn: Tree, els: Else) extends Tree
    }
    package macros {
    object IDEUniverse extends macros.Universe {
    override type Term = IDE.Tree
    override def IfApply(cnd: Term, thn: Term, els: Term): Term =
    IDE.If(cnd, thn, IDE.Else(els))
    override def IfUnapply(term: Term): Option[(Term, Term, Term)] = term match {
    case IDE.If(cnd, thn, IDE.Else(els)) => Some((cnd, thn, els))
    case _ => None
    }
    }
    }
    val original: IDE.Tree = (IDE)< 0) 1 else 0>>
    val inverted: IDE.Tree = invert(original) // => if (hoge(fuga) > 0) 0 else 1
    

    View Slide

38. 38 / 43
    標準構文の利点: 開発コストの低下, マクロの実
    行速度改善, AST情報の完全な保持.
    Generation 3: Advantages
    ▰ Reduce implementation cost.
    ▻ wrappers are far easier to implement than converters.
    ▰ Improve performance of macro code.
    ▻ no AST conversion, no additional cost.
    ▰ Retain full information in raw compiler AST.
    ▻ because scala.macros manipulates raw AST “as-is”.
    

    View Slide

39. ▰ We still have to implement tailored wrappers
    for each Scala implementation, don’t we?
    ▰ YES. But the cost is far smaller than
    converters.
    ▻ We don’t have to care about meta information
    attached to raw AST.
    ▻ Wrappers are not required to cover all syntactic
    elements.
    39 / 43
    標準構文wrapperはScala実装ごとに定義する
    必要? → そうだが、コストが段違いに低い.
    Generation 3: FAQ
    

    View Slide

40. 40 / 43
    invertマクロに必要な標準構文と標準ASTの比較.
    Generation 3: FAQ
    if (str.startsWith("abc"))
    (1 to 10) foreach {
    case i if i % 3 == 0 => println(i)
    case _ => ()
    }
    else
    throw new RuntimeException()
    To invert this if expr...
    

    View Slide

41. 40 / 43
    invertマクロに必要な標準構文と標準ASTの比較.
    Generation 3: FAQ
    if (str.startsWith("abc"))
    (1 to 10) foreach {
    case i if i % 3 == 0 => println(i)
    case _ => ()
    }
    else
    throw new RuntimeException()
    To invert this if expr...
    macros.TermIf
    Generation 3 needs
    scalac.Term
    scalac.Term
    scalac.Term
    

    View Slide

42. 40 / 43
    invertマクロに必要な標準構文と標準ASTの比較.
    Generation 3: FAQ
    if (str.startsWith("abc"))
    (1 to 10) foreach {
    case i if i % 3 == 0 => println(i)
    case _ => ()
    }
    else
    throw new RuntimeException()
    To invert this if expr...
    scala.meta.Term.{
    If, Apply, Select, Name, ApplyInfix,
    PartialFunction, Throw, Param
    }
    scala.meta.{
    Lit.{String, Int, Unit}, Template,
    Pat.Var.Term, Ctor.Ref.Name
    }
    Generation 2 needs
    

    View Slide

43. 41 / 43
    Scalaメタプログラミングの歴史は“苦しくない可
    搬性”の追求. scala.macrosへの期待.
    Conclusion
    ▰ The evolution of metaprogramming system in
    Scala can be seen as a process to pursue
    “portability without pain”
    ▰ New scala.macros is expected to be a good
    solution towards this goal.
    

    View Slide

44. ▰ @DavidDudson has started developing
    scalagen.
    ▰ It will be a scala.meta-based static code
    generation tool.
    ▻ Also it may be the first attempt for an organized code
    generation tool.
    ▰ Although this is a private (i.e. not an scalacenter’s
    official) project, it is worthy of note.
    42 / 43
    新しい静的コード生成ツール“ScalaGen”にも注
    目する価値がある.
    Appendix: ScalaGen
    

    View Slide

45. Thank you!
    Questions and comments are welcome
    both in English and Japanese.
    

    View Slide