AST meta-programming in Swift

Introduce meta-programming technique in Swift using AST
try! Swift Tokyo 2018

Transcript

1. AST Meta-Programming Writing a Swift tool using AST
2. Agenda • What's AST? • Tools using AST • Kind of Swift AST • Let's write a tool yourself
3. What's AST? • Abstract Syntax Tree • Semantic structures form a hierarchical tree • Internal representation of a source code for compiler • Handling source code programmatically FunctionDecl func greet ( person : String ) FunctionSignature FunctionParameterList
4. Handling source code programmatically • Code Analysis
5. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor
6. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format
7. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint
8. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation
9. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery
10. Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery DIKit https://github.com/ishkawa/DIKit
11. Tools using SourceKit's AST • SwiftLint A tool to enforce Swift style and conventions. https://github.com/realm/SwiftLint • Jazzy Soulful docs for Swift & Objective-C https://github.com/realm/jazzy • Sourcery Meta-programming for Swift, stop writing boilerplate code. https://github.com/krzysztofzablocki/Sourcery • DIKit A statically typed dependency injector for Swift. https://github.com/ishkawa/DIKit • Taylor Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms. https://github.com/yopeso/Taylor
12. Various Swift's AST • SourceKit • swiftc -dump-parse • swiftc -dump-ast • swiftc -emit-syntax • (swiftc -print-ast)
13. SourceKit 1 [ 2 { 3 "type" : "source.lang.swift.syntaxtype.keyword", 4 "offset" : 0, 5 "length" : 4 6 }, 7 { 8 "type" : "source.lang.swift.syntaxtype.identifier", 9 "offset" : 5, 10 "length" : 5 11 }, 12 { 13 "type" : "source.lang.swift.syntaxtype.identifier", 14 "offset" : 11, 15 "length" : 6 16 }, 17 { 18 "type" : "source.lang.swift.syntaxtype.typeidentifier", 19 "offset" : 19, 20 "length" : 6 21 }, 22 ... ... 62 ] 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 }
14. SourceKitten An adorable little framework and command line tool for interacting with SourceKit. https://github.com/jpsim/SourceKitten
15. swiftc -dump-parse 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" (parameter_list (parameter "person" apiName=person)) (result (type_ident (component id='String' bind=none))) (brace_stmt (pattern_binding_decl (pattern_named 'greeting') (sequence_expr type='<null>' (string_literal_expr type='<null>' encoding=utf8 value (unresolved_decl_ref_expr type='<null>' name=+ functio (declref_expr type='<null>' decl=test.(file).greet(per (unresolved_decl_ref_expr type='<null>' name=+ functio (string_literal_expr type='<null>' encoding=utf8 value (var_decl "greeting" type='<null type>' let storage_kind=s (return_stmt (declref_expr type='<null>' decl=test.(file).greet(perso
16. swiftc -dump-ast 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" interface type='(String) -> String' a (parameter_list (parameter "person" apiName=person type='String' interface ty (result (type_ident (component id='String' bind=Swift.(file).String))) (brace_stmt (pattern_binding_decl (pattern_named type='String' 'greeting') (binary_expr type='String' location=test.swift:2:39 range=[ (dot_syntax_call_expr implicit type='(String, String) -> (declref_expr type='(String.Type) -> (String, String) - (type_expr implicit type='String.Type' location=test.sw (tuple_expr implicit type='(String, String)' location=tes (binary_expr type='String' location=test.swift:2:30 ran (dot_syntax_call_expr implicit type='(String, String) (declref_expr type='(String.Type) -> (String, Strin (type_expr implicit type='String.Type' location=tes (tuple_expr implicit type='(String, String)' location (string_literal_expr type='String' location=test.sw (declref_expr type='String' location=test.swift:2:3 (string_literal_expr type='String' location=test.swift: (var_decl "greeting" type='String' interface type='String' ac (return_stmt (declref_expr type='String' location=test.swift:3:12 range=
17. swiftc -emit-syntax 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 7 { "kind": "FunctionDecl", 8 "layout": [ 9 null, 10 null, 11 { "tokenKind": { 12 "kind": "kw_func" 13 }, 14 "leadingTrivia": [], 15 "trailingTrivia": [ 16 { "kind": "Space", 17 "value": 1 18 } 19 ], 20 "presence": "Present" 21 }, 22 { "tokenKind": { 23 "kind": "identifier", 24 "text": "greet" 25 }, 26 "leadingTrivia": [], 27 "trailingTrivia": [], 28 "presence": "Present" 29 }, 30 null, 31 { "kind": "FunctionSignature", 32 "layout": [ 33 { "kind": "ParameterClause", 34 "layout": [ 35 { "tokenKind": { 36 "kind": "l_paren" 37 }, 38 "leadingTrivia": [], 39 "trailingTrivia": [], 40 "presence": "Present" 41 }, 42 { "kind": "FunctionParameterList",
18. SwiftSyntax • Swift wrapper of libSyntax • Parses libSyntax's AST • Offers AST builder API 8 import Foundation 9 import SwiftSyntax 10 11 class TokenVisitor : SyntaxVisitor { 12 var tokens = [Token]() 13 private var line = [Token]() 14 private var contexts = [Context]() 15 16 override func visit(_ node: ImportDeclSy 17 processNode(node) 18 } 19 20 override func visit(_ node: ClassDeclSyn 21 processNode(node) 22 } 23 24 override func visit(_ node: StructDeclSy 25 processNode(node)
19. Comparison Tables Tool Difficulty Type check Source mapping SourceKit SourceKitten Easy No Partial -emit-syntax SwiftSyntax Medium No Loss-less -dump-ast - Hard Yes Partial
20. Practice makes perfect Let's write a tool
21. Convert Swift Source code to HTML 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 1 <!DOCTYPE html> 2 <html lang="en"> ... 10 <boy> 11 <div class="box FunctionDeclSyntax" data-tooltip 12 <span class='keyword'>func</span>&nbsp;<span c 13 <div class="box VariableDeclSyntax" data-toolt 14 <br> 15 &nbsp;&nbsp;&nbsp;&nbsp;<span class='keyword'> 16 <span class='rightBrace'>}</span> 17 </div> 18 <br> 19 <span class='eof'></span> 20 </body> 21 </html>
22. Convert Swift Source code to HTML Swift 4.1 1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>")
23. 1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
24. 1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML
25. 1 <!DOCTYPE html> 2 <html> 3 <body> 4 <span class='funcKeyword'>func</span> 5 <span class='identifier("greet")'>greet</span> 6 <span class='leftParen'>(</span> 7 <span class='identifier("person")'>person</span> 8 <span class='colon'>:</span> 9 <span class='identifier("String")'>String</span> 10 <span class='rightParen'>)</span> 11 <span class='arrow'>-></span> 12 <span class='identifier("String")'>String</span> 13 <span class='leftBrace'>{</span> 14 <span class='letKeyword'>let</span> 15 <span class='identifier("greeting")'>greeting</span> 16 <span class='equal'>=</span> 17 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 18 <span class='spacedBinaryOperator("+")'>+</span> 19 <span class='identifier("person")'>person</span> 20 <span class='spacedBinaryOperator("+")'>+</span> 21 <span class='stringLiteral("\"!\"")'>"!"</span> 22 <span class='returnKeyword'>return</span> 23 <span class='identifier("greeting")'>greeting</span> 24 <span class='rightBrace'>}</span> 25 <span class='eof'></span> 26 </body> 27 </html>
26. 1 <!DOCTYPE html> 2 <html> 3 <link rel="stylesheet" href="css/default.css" type="text/css" /> 4 <body> 5 <span class='funcKeyword'>func</span> 6 <span class='identifier("greet")'>greet</span> 7 <span class='leftParen'>(</span> 8 <span class='identifier("person")'>person</span> 9 <span class='colon'>:</span> 10 <span class='identifier("String")'>String</span> 11 <span class='rightParen'>)</span> 12 <span class='arrow'>-></span> 13 <span class='identifier("String")'>String</span> 14 <span class='leftBrace'>{</span> 15 <span class='letKeyword'>let</span> 16 <span class='identifier("greeting")'>greeting</span> 17 <span class='equal'>=</span> 18 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 19 <span class='spacedBinaryOperator("+")'>+</span> 20 <span class='identifier("person")'>person</span> 21 <span class='spacedBinaryOperator("+")'>+</span> 22 <span class='stringLiteral("\"!\"")'>"!"</span> 23 <span class='returnKeyword'>return</span> 24 <span class='identifier("greeting")'>greeting</span> 25 <span class='rightBrace'>}</span> 26 <span class='eof'></span> 27 </body> 28 </html>
28. 5 class TokenVisitor : SyntaxVisitor { ... 7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
29. 5 class TokenVisitor : SyntaxVisitor { ... 7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36
31. Swift AST Explorer swift-ast-explorer.kishikawakatsumi.com
32. Aspect Oriented Programming in Swift • Aspect Oriented Programming in Swift ‣ Insert logging or tracking code automatically 9 import UIKit 10 import PDFKit 11 12 class BookmarkViewController: ... ... 34 35 override func viewDidLoad( 36 super.viewDidLoad() ... ... 46 } 47 48 override func viewWillAppe super.viewWillAppear(a ... ... 49 50 } print(#function) print(#function)
33. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
34. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
35. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
36. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
37. Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...
38. Intercept build process Parser TypeChecker IRGen AST SIL IR Source
39. My Tool Intercept build process SILGen IRGen AST SIL IR Parser Source AST
40. My Tool Intercept build process SILGen IRGen SIL IR Parser Source AST
41. Demo $ ./swiftaop --pattern "^view.+" --advice advice.swift -Xxcodebuild -target BookReader \ -sdk iphonesimulator build
42. SwiftPowerAssert Provide describe assertion message
43. SwiftPowerAssert XCTAssert(bar.val == bar.foo.val) | | | | | | | 3 | | | 2 | | | Foo(val: 2) | | Bar(foo: main.Foo(val: 2), val: 3) | false Bar(foo: main.Foo(val: 2), val: 3)
44. ... 12 class Tests: XCTestCase { 13 func testMethod() { 14 let bar = Bar(foo: Foo(val: 2), val: 3) 15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)") .assertBoolean(bar.val == bar.foo.val, op: ==) .record(expression: bar, column: 11).record(expression: bar.val, column: 15) .record(expression: (bar.val == bar.foo.val) as (Bool), column: 19) .record(expression: bar, column: 22).record(expression: bar.foo, column: 26) .record(expression: bar.foo.val, column: 30).render() 18 } 19 } 1 class Tests: XCTestCase { 2 func testMethod() { 3 let bar = Bar(foo: Foo(val: 2), val: 3) 4 XCTAssert(bar.val == bar.foo.val) 5 } 6 }
45. SwiftPowerAssert Playground swift-power-assert.kishikawakatsumi.com
47. swiftfmt Code formatter for Swift
48. Swiftfmt Playground swiftfmt.kishikawakatsumi.com
49. Wrap up • With AST, we can write code using source code information. • Meta-programming with AST can eliminate boilerplate code and give dynamic behaviors to Swift. • Swift has several kinds of AST. • Some ASTs can be used easily with tools like SourceKitten and SwiftSyntax.
50. Resources • Improving Swift Tools with libSyntax by Harlan Haskins academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/ • SourceKit and You by JP Simard academy.realm.io/posts/appbuilders-jp-simard-sourcekit/
51. Resources • SwiftPowerAssert https://github.com/kishikawakatsumi/SwiftPowerAssert/ • Swiftfmt https://github.com/kishikawakatsumi/swiftfmt • Swift AST Explorer https://github.com/kishikawakatsumi/swift-ast-explorer