AST meta-programming in Swift

AST meta-programming in Swift

Introduce meta-programming technique in Swift using AST
try! Swift Tokyo 2018

9bf923e39671cde83584e3e926296c13?s=128

Kishikawa Katsumi

March 01, 2018
Tweet

Transcript

  1. 2.

    Agenda • What's AST? • Tools using AST • Kind

    of Swift AST • Let's write a tool yourself
  2. 3.

    What's AST? • Abstract Syntax Tree • Semantic structures form

    a hierarchical tree • Internal representation of a source code for compiler • Handling source code programmatically FunctionDecl func greet ( person : String ) FunctionSignature FunctionParameterList
  3. 8.

    Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation
  4. 9.

    Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery
  5. 10.

    Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery DIKit https://github.com/ishkawa/DIKit
  6. 11.

    Tools using SourceKit's AST • SwiftLint A tool to enforce

    Swift style and conventions. https://github.com/realm/SwiftLint • Jazzy Soulful docs for Swift & Objective-C https://github.com/realm/jazzy • Sourcery Meta-programming for Swift, stop writing boilerplate code. https://github.com/krzysztofzablocki/Sourcery • DIKit A statically typed dependency injector for Swift. https://github.com/ishkawa/DIKit • Taylor Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms. https://github.com/yopeso/Taylor
  7. 12.

    Various Swift's AST • SourceKit • swiftc -dump-parse • swiftc

    -dump-ast • swiftc -emit-syntax • (swiftc -print-ast)
  8. 13.

    SourceKit 1 [ 2 { 3 "type" : "source.lang.swift.syntaxtype.keyword", 4

    "offset" : 0, 5 "length" : 4 6 }, 7 { 8 "type" : "source.lang.swift.syntaxtype.identifier", 9 "offset" : 5, 10 "length" : 5 11 }, 12 { 13 "type" : "source.lang.swift.syntaxtype.identifier", 14 "offset" : 11, 15 "length" : 6 16 }, 17 { 18 "type" : "source.lang.swift.syntaxtype.typeidentifier", 19 "offset" : 19, 20 "length" : 6 21 }, 22 ... ... 62 ] 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 }
  9. 14.

    SourceKitten An adorable little framework and command line tool for

    interacting with SourceKit. https://github.com/jpsim/SourceKitten
  10. 15.

    swiftc -dump-parse 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" (parameter_list (parameter "person" apiName=person)) (result (type_ident (component id='String' bind=none))) (brace_stmt (pattern_binding_decl (pattern_named 'greeting') (sequence_expr type='<null>' (string_literal_expr type='<null>' encoding=utf8 value (unresolved_decl_ref_expr type='<null>' name=+ functio (declref_expr type='<null>' decl=test.(file).greet(per (unresolved_decl_ref_expr type='<null>' name=+ functio (string_literal_expr type='<null>' encoding=utf8 value (var_decl "greeting" type='<null type>' let storage_kind=s (return_stmt (declref_expr type='<null>' decl=test.(file).greet(perso
  11. 16.

    swiftc -dump-ast 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" interface type='(String) -> String' a (parameter_list (parameter "person" apiName=person type='String' interface ty (result (type_ident (component id='String' bind=Swift.(file).String))) (brace_stmt (pattern_binding_decl (pattern_named type='String' 'greeting') (binary_expr type='String' location=test.swift:2:39 range=[ (dot_syntax_call_expr implicit type='(String, String) -> (declref_expr type='(String.Type) -> (String, String) - (type_expr implicit type='String.Type' location=test.sw (tuple_expr implicit type='(String, String)' location=tes (binary_expr type='String' location=test.swift:2:30 ran (dot_syntax_call_expr implicit type='(String, String) (declref_expr type='(String.Type) -> (String, Strin (type_expr implicit type='String.Type' location=tes (tuple_expr implicit type='(String, String)' location (string_literal_expr type='String' location=test.sw (declref_expr type='String' location=test.swift:2:3 (string_literal_expr type='String' location=test.swift: (var_decl "greeting" type='String' interface type='String' ac (return_stmt (declref_expr type='String' location=test.swift:3:12 range=
  12. 17.

    swiftc -emit-syntax 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } 7 { "kind": "FunctionDecl", 8 "layout": [ 9 null, 10 null, 11 { "tokenKind": { 12 "kind": "kw_func" 13 }, 14 "leadingTrivia": [], 15 "trailingTrivia": [ 16 { "kind": "Space", 17 "value": 1 18 } 19 ], 20 "presence": "Present" 21 }, 22 { "tokenKind": { 23 "kind": "identifier", 24 "text": "greet" 25 }, 26 "leadingTrivia": [], 27 "trailingTrivia": [], 28 "presence": "Present" 29 }, 30 null, 31 { "kind": "FunctionSignature", 32 "layout": [ 33 { "kind": "ParameterClause", 34 "layout": [ 35 { "tokenKind": { 36 "kind": "l_paren" 37 }, 38 "leadingTrivia": [], 39 "trailingTrivia": [], 40 "presence": "Present" 41 }, 42 { "kind": "FunctionParameterList",
  13. 18.

    SwiftSyntax • Swift wrapper of libSyntax • Parses libSyntax's AST

    • Offers AST builder API 8 import Foundation 9 import SwiftSyntax 10 11 class TokenVisitor : SyntaxVisitor { 12 var tokens = [Token]() 13 private var line = [Token]() 14 private var contexts = [Context]() 15 16 override func visit(_ node: ImportDeclSy 17 processNode(node) 18 } 19 20 override func visit(_ node: ClassDeclSyn 21 processNode(node) 22 } 23 24 override func visit(_ node: StructDeclSy 25 processNode(node)
  14. 19.

    Comparison Tables Tool Difficulty Type check Source mapping SourceKit SourceKitten

    Easy No Partial -emit-syntax SwiftSyntax Medium No Loss-less -dump-ast - Hard Yes Partial
  15. 21.

    Convert Swift Source code to HTML 1 func greet(person: String)

    -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 1 <!DOCTYPE html> 2 <html lang="en"> ... 10 <boy> 11 <div class="box FunctionDeclSyntax" data-tooltip 12 <span class='keyword'>func</span>&nbsp;<span c 13 <div class="box VariableDeclSyntax" data-toolt 14 <br> 15 &nbsp;&nbsp;&nbsp;&nbsp;<span class='keyword'> 16 <span class='rightBrace'>}</span> 17 </div> 18 <br> 19 <span class='eof'></span> 20 </body> 21 </html>
  16. 22.

    Convert Swift Source code to HTML Swift 4.1 1 import

    SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>")
  17. 23.

    1 import SwiftSyntax 2 3 var html = "" 4

    5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
  18. 24.

    1 import SwiftSyntax 2 3 var html = "" 4

    5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
  19. 25.

    1 <!DOCTYPE html> 2 <html> 3 <body> 4 <span class='funcKeyword'>func</span>

    5 <span class='identifier("greet")'>greet</span> 6 <span class='leftParen'>(</span> 7 <span class='identifier("person")'>person</span> 8 <span class='colon'>:</span> 9 <span class='identifier("String")'>String</span> 10 <span class='rightParen'>)</span> 11 <span class='arrow'>-></span> 12 <span class='identifier("String")'>String</span> 13 <span class='leftBrace'>{</span> 14 <span class='letKeyword'>let</span> 15 <span class='identifier("greeting")'>greeting</span> 16 <span class='equal'>=</span> 17 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 18 <span class='spacedBinaryOperator("+")'>+</span> 19 <span class='identifier("person")'>person</span> 20 <span class='spacedBinaryOperator("+")'>+</span> 21 <span class='stringLiteral("\"!\"")'>"!"</span> 22 <span class='returnKeyword'>return</span> 23 <span class='identifier("greeting")'>greeting</span> 24 <span class='rightBrace'>}</span> 25 <span class='eof'></span> 26 </body> 27 </html>
  20. 26.

    1 <!DOCTYPE html> 2 <html> 3 <link rel="stylesheet" href="css/default.css" type="text/css"

    /> 4 <body> 5 <span class='funcKeyword'>func</span> 6 <span class='identifier("greet")'>greet</span> 7 <span class='leftParen'>(</span> 8 <span class='identifier("person")'>person</span> 9 <span class='colon'>:</span> 10 <span class='identifier("String")'>String</span> 11 <span class='rightParen'>)</span> 12 <span class='arrow'>-></span> 13 <span class='identifier("String")'>String</span> 14 <span class='leftBrace'>{</span> 15 <span class='letKeyword'>let</span> 16 <span class='identifier("greeting")'>greeting</span> 17 <span class='equal'>=</span> 18 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 19 <span class='spacedBinaryOperator("+")'>+</span> 20 <span class='identifier("person")'>person</span> 21 <span class='spacedBinaryOperator("+")'>+</span> 22 <span class='stringLiteral("\"!\"")'>"!"</span> 23 <span class='returnKeyword'>return</span> 24 <span class='identifier("greeting")'>greeting</span> 25 <span class='rightBrace'>}</span> 26 <span class='eof'></span> 27 </body> 28 </html>
  21. 27.
  22. 28.

    5 class TokenVisitor : SyntaxVisitor { ... 7 var types

    = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
  23. 29.

    5 class TokenVisitor : SyntaxVisitor { ... 7 var types

    = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
  24. 30.
  25. 32.

    Aspect Oriented Programming in Swift • Aspect Oriented Programming in

    Swift ‣ Insert logging or tracking code automatically 9 import UIKit 10 import PDFKit 11 12 class BookmarkViewController: ... ... 34 35 override func viewDidLoad( 36 super.viewDidLoad() ... ... 46 } 47 48 override func viewWillAppe super.viewWillAppear(a ... ... 49 50 } print(#function) print(#function)
  26. 33.

    Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  27. 34.

    Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  28. 35.

    Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  29. 36.

    Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  30. 37.

    Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  31. 43.

    SwiftPowerAssert XCTAssert(bar.val == bar.foo.val) | | | | | |

    | 3 | | | 2 | | | Foo(val: 2) | | Bar(foo: main.Foo(val: 2), val: 3) | false Bar(foo: main.Foo(val: 2), val: 3)
  32. 44.

    ... 12 class Tests: XCTestCase { 13 func testMethod() {

    14 let bar = Bar(foo: Foo(val: 2), val: 3) 15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)") .assertBoolean(bar.val == bar.foo.val, op: ==) .record(expression: bar, column: 11).record(expression: bar.val, column: 15) .record(expression: (bar.val == bar.foo.val) as (Bool), column: 19) .record(expression: bar, column: 22).record(expression: bar.foo, column: 26) .record(expression: bar.foo.val, column: 30).render() 18 } 19 } 1 class Tests: XCTestCase { 2 func testMethod() { 3 let bar = Bar(foo: Foo(val: 2), val: 3) 4 XCTAssert(bar.val == bar.foo.val) 5 } 6 }
  33. 46.
  34. 49.

    Wrap up • With AST, we can write code using

    source code information. • Meta-programming with AST can eliminate boilerplate code and give dynamic behaviors to Swift. • Swift has several kinds of AST. • Some ASTs can be used easily with tools like SourceKitten and SwiftSyntax.
  35. 50.

    Resources • Improving Swift Tools with libSyntax by Harlan Haskins

    academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/ • SourceKit and You by JP Simard academy.realm.io/posts/appbuilders-jp-simard-sourcekit/