Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AST meta-programming in Swift

AST meta-programming in Swift

Introduce meta-programming technique in Swift using AST
try! Swift Tokyo 2018

9bf923e39671cde83584e3e926296c13?s=128

Kishikawa Katsumi

March 01, 2018
Tweet

Transcript

  1. AST Meta-Programming Writing a Swift tool using AST

  2. Agenda • What's AST? • Tools using AST • Kind

    of Swift AST • Let's write a tool yourself
  3. What's AST? • Abstract Syntax Tree • Semantic structures form

    a hierarchical tree • Internal representation of a source code for compiler • Handling source code programmatically FunctionDecl func greet ( person : String ) FunctionSignature FunctionParameterList
  4. Handling source code programmatically • Code Analysis

  5. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor

  6. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

  7. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint
  8. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation
  9. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery
  10. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

    SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery DIKit https://github.com/ishkawa/DIKit
  11. Tools using SourceKit's AST • SwiftLint A tool to enforce

    Swift style and conventions. https://github.com/realm/SwiftLint • Jazzy Soulful docs for Swift & Objective-C https://github.com/realm/jazzy • Sourcery Meta-programming for Swift, stop writing boilerplate code. https://github.com/krzysztofzablocki/Sourcery • DIKit A statically typed dependency injector for Swift. https://github.com/ishkawa/DIKit • Taylor Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms. https://github.com/yopeso/Taylor
  12. Various Swift's AST • SourceKit • swiftc -dump-parse • swiftc

    -dump-ast • swiftc -emit-syntax • (swiftc -print-ast)
  13. SourceKit 1 [ 2 { 3 "type" : "source.lang.swift.syntaxtype.keyword", 4

    "offset" : 0, 5 "length" : 4 6 }, 7 { 8 "type" : "source.lang.swift.syntaxtype.identifier", 9 "offset" : 5, 10 "length" : 5 11 }, 12 { 13 "type" : "source.lang.swift.syntaxtype.identifier", 14 "offset" : 11, 15 "length" : 6 16 }, 17 { 18 "type" : "source.lang.swift.syntaxtype.typeidentifier", 19 "offset" : 19, 20 "length" : 6 21 }, 22 ... ... 62 ] 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 }
  14. SourceKitten An adorable little framework and command line tool for

    interacting with SourceKit. https://github.com/jpsim/SourceKitten
  15. swiftc -dump-parse 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" (parameter_list (parameter "person" apiName=person)) (result (type_ident (component id='String' bind=none))) (brace_stmt (pattern_binding_decl (pattern_named 'greeting') (sequence_expr type='<null>' (string_literal_expr type='<null>' encoding=utf8 value (unresolved_decl_ref_expr type='<null>' name=+ functio (declref_expr type='<null>' decl=test.(file).greet(per (unresolved_decl_ref_expr type='<null>' name=+ functio (string_literal_expr type='<null>' encoding=utf8 value (var_decl "greeting" type='<null type>' let storage_kind=s (return_stmt (declref_expr type='<null>' decl=test.(file).greet(perso
  16. swiftc -dump-ast 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" interface type='(String) -> String' a (parameter_list (parameter "person" apiName=person type='String' interface ty (result (type_ident (component id='String' bind=Swift.(file).String))) (brace_stmt (pattern_binding_decl (pattern_named type='String' 'greeting') (binary_expr type='String' location=test.swift:2:39 range=[ (dot_syntax_call_expr implicit type='(String, String) -> (declref_expr type='(String.Type) -> (String, String) - (type_expr implicit type='String.Type' location=test.sw (tuple_expr implicit type='(String, String)' location=tes (binary_expr type='String' location=test.swift:2:30 ran (dot_syntax_call_expr implicit type='(String, String) (declref_expr type='(String.Type) -> (String, Strin (type_expr implicit type='String.Type' location=tes (tuple_expr implicit type='(String, String)' location (string_literal_expr type='String' location=test.sw (declref_expr type='String' location=test.swift:2:3 (string_literal_expr type='String' location=test.swift: (var_decl "greeting" type='String' interface type='String' ac (return_stmt (declref_expr type='String' location=test.swift:3:12 range=
  17. swiftc -emit-syntax 1 func greet(person: String) -> String { 2

    let greeting = "Hello, " + person + "!" 3 return greeting 4 } 7 { "kind": "FunctionDecl", 8 "layout": [ 9 null, 10 null, 11 { "tokenKind": { 12 "kind": "kw_func" 13 }, 14 "leadingTrivia": [], 15 "trailingTrivia": [ 16 { "kind": "Space", 17 "value": 1 18 } 19 ], 20 "presence": "Present" 21 }, 22 { "tokenKind": { 23 "kind": "identifier", 24 "text": "greet" 25 }, 26 "leadingTrivia": [], 27 "trailingTrivia": [], 28 "presence": "Present" 29 }, 30 null, 31 { "kind": "FunctionSignature", 32 "layout": [ 33 { "kind": "ParameterClause", 34 "layout": [ 35 { "tokenKind": { 36 "kind": "l_paren" 37 }, 38 "leadingTrivia": [], 39 "trailingTrivia": [], 40 "presence": "Present" 41 }, 42 { "kind": "FunctionParameterList",
  18. SwiftSyntax • Swift wrapper of libSyntax • Parses libSyntax's AST

    • Offers AST builder API 8 import Foundation 9 import SwiftSyntax 10 11 class TokenVisitor : SyntaxVisitor { 12 var tokens = [Token]() 13 private var line = [Token]() 14 private var contexts = [Context]() 15 16 override func visit(_ node: ImportDeclSy 17 processNode(node) 18 } 19 20 override func visit(_ node: ClassDeclSyn 21 processNode(node) 22 } 23 24 override func visit(_ node: StructDeclSy 25 processNode(node)
  19. Comparison Tables Tool Difficulty Type check Source mapping SourceKit SourceKitten

    Easy No Partial -emit-syntax SwiftSyntax Medium No Loss-less -dump-ast - Hard Yes Partial
  20. Practice makes perfect Let's write a tool

  21. Convert Swift Source code to HTML 1 func greet(person: String)

    -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 1 <!DOCTYPE html> 2 <html lang="en"> ... 10 <boy> 11 <div class="box FunctionDeclSyntax" data-tooltip 12 <span class='keyword'>func</span>&nbsp;<span c 13 <div class="box VariableDeclSyntax" data-toolt 14 <br> 15 &nbsp;&nbsp;&nbsp;&nbsp;<span class='keyword'> 16 <span class='rightBrace'>}</span> 17 </div> 18 <br> 19 <span class='eof'></span> 20 </body> 21 </html>
  22. Convert Swift Source code to HTML Swift 4.1 1 import

    SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>")
  23. 1 import SwiftSyntax 2 3 var html = "" 4

    5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
  24. 1 import SwiftSyntax 2 3 var html = "" 4

    5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
  25. 1 <!DOCTYPE html> 2 <html> 3 <body> 4 <span class='funcKeyword'>func</span>

    5 <span class='identifier("greet")'>greet</span> 6 <span class='leftParen'>(</span> 7 <span class='identifier("person")'>person</span> 8 <span class='colon'>:</span> 9 <span class='identifier("String")'>String</span> 10 <span class='rightParen'>)</span> 11 <span class='arrow'>-></span> 12 <span class='identifier("String")'>String</span> 13 <span class='leftBrace'>{</span> 14 <span class='letKeyword'>let</span> 15 <span class='identifier("greeting")'>greeting</span> 16 <span class='equal'>=</span> 17 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 18 <span class='spacedBinaryOperator("+")'>+</span> 19 <span class='identifier("person")'>person</span> 20 <span class='spacedBinaryOperator("+")'>+</span> 21 <span class='stringLiteral("\"!\"")'>"!"</span> 22 <span class='returnKeyword'>return</span> 23 <span class='identifier("greeting")'>greeting</span> 24 <span class='rightBrace'>}</span> 25 <span class='eof'></span> 26 </body> 27 </html>
  26. 1 <!DOCTYPE html> 2 <html> 3 <link rel="stylesheet" href="css/default.css" type="text/css"

    /> 4 <body> 5 <span class='funcKeyword'>func</span> 6 <span class='identifier("greet")'>greet</span> 7 <span class='leftParen'>(</span> 8 <span class='identifier("person")'>person</span> 9 <span class='colon'>:</span> 10 <span class='identifier("String")'>String</span> 11 <span class='rightParen'>)</span> 12 <span class='arrow'>-></span> 13 <span class='identifier("String")'>String</span> 14 <span class='leftBrace'>{</span> 15 <span class='letKeyword'>let</span> 16 <span class='identifier("greeting")'>greeting</span> 17 <span class='equal'>=</span> 18 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 19 <span class='spacedBinaryOperator("+")'>+</span> 20 <span class='identifier("person")'>person</span> 21 <span class='spacedBinaryOperator("+")'>+</span> 22 <span class='stringLiteral("\"!\"")'>"!"</span> 23 <span class='returnKeyword'>return</span> 24 <span class='identifier("greeting")'>greeting</span> 25 <span class='rightBrace'>}</span> 26 <span class='eof'></span> 27 </body> 28 </html>
  27. None
  28. 5 class TokenVisitor : SyntaxVisitor { ... 7 var types

    = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
  29. 5 class TokenVisitor : SyntaxVisitor { ... 7 var types

    = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
  30. None
  31. Swift AST Explorer swift-ast-explorer.kishikawakatsumi.com

  32. Aspect Oriented Programming in Swift • Aspect Oriented Programming in

    Swift ‣ Insert logging or tracking code automatically 9 import UIKit 10 import PDFKit 11 12 class BookmarkViewController: ... ... 34 35 override func viewDidLoad( 36 super.viewDidLoad() ... ... 46 } 47 48 override func viewWillAppe super.viewWillAppear(a ... ... 49 50 } print(#function) print(#function)
  33. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  34. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  35. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  36. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  37. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {

    24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
  38. Intercept build process Parser TypeChecker IRGen AST SIL IR Source

  39. My Tool Intercept build process SILGen IRGen AST SIL IR

    Parser Source AST
  40. My Tool Intercept build process SILGen IRGen SIL IR Parser

    Source AST
  41. Demo $ ./swiftaop --pattern "^view.+" --advice advice.swift -Xxcodebuild -target BookReader

    \ -sdk iphonesimulator build
  42. SwiftPowerAssert Provide describe assertion message

  43. SwiftPowerAssert XCTAssert(bar.val == bar.foo.val) | | | | | |

    | 3 | | | 2 | | | Foo(val: 2) | | Bar(foo: main.Foo(val: 2), val: 3) | false Bar(foo: main.Foo(val: 2), val: 3)
  44. ... 12 class Tests: XCTestCase { 13 func testMethod() {

    14 let bar = Bar(foo: Foo(val: 2), val: 3) 15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)") .assertBoolean(bar.val == bar.foo.val, op: ==) .record(expression: bar, column: 11).record(expression: bar.val, column: 15) .record(expression: (bar.val == bar.foo.val) as (Bool), column: 19) .record(expression: bar, column: 22).record(expression: bar.foo, column: 26) .record(expression: bar.foo.val, column: 30).render() 18 } 19 } 1 class Tests: XCTestCase { 2 func testMethod() { 3 let bar = Bar(foo: Foo(val: 2), val: 3) 4 XCTAssert(bar.val == bar.foo.val) 5 } 6 }
  45. SwiftPowerAssert Playground swift-power-assert.kishikawakatsumi.com

  46. None
  47. swiftfmt Code formatter for Swift

  48. Swiftfmt Playground swiftfmt.kishikawakatsumi.com

  49. Wrap up • With AST, we can write code using

    source code information. • Meta-programming with AST can eliminate boilerplate code and give dynamic behaviors to Swift. • Swift has several kinds of AST. • Some ASTs can be used easily with tools like SourceKitten and SwiftSyntax.
  50. Resources • Improving Swift Tools with libSyntax by Harlan Haskins

    academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/ • SourceKit and You by JP Simard academy.realm.io/posts/appbuilders-jp-simard-sourcekit/
  51. Resources • SwiftPowerAssert https://github.com/kishikawakatsumi/SwiftPowerAssert/ • Swiftfmt https://github.com/kishikawakatsumi/swiftfmt • Swift AST

    Explorer https://github.com/kishikawakatsumi/swift-ast-explorer