Share
Embed
AST meta-programming in Swift
Introduce meta-programming technique in Swift using AST
try! Swift Tokyo 2018
Kishikawa Katsumi
March 01, 2018
Transcript
51 slides
-
Agenda • What's AST? • Tools using AST • Kind
of Swift AST • Let's write a tool yourself -
What's AST? • Abstract Syntax Tree • Semantic structures form
a hierarchical tree • Internal representation of a source code for compiler • Handling source code programmatically FunctionDecl func greet ( person : String ) FunctionSignature FunctionParameterList -
Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format
SwiftLint https://github.com/realm/SwiftLint -
Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format
SwiftLint https://github.com/realm/SwiftLint • Code generation -
Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format
SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery -
Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format
SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery DIKit https://github.com/ishkawa/DIKit -
Tools using SourceKit's AST • SwiftLint A tool to enforce
Swift style and conventions. https://github.com/realm/SwiftLint • Jazzy Soulful docs for Swift & Objective-C https://github.com/realm/jazzy • Sourcery Meta-programming for Swift, stop writing boilerplate code. https://github.com/krzysztofzablocki/Sourcery • DIKit A statically typed dependency injector for Swift. https://github.com/ishkawa/DIKit • Taylor Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms. https://github.com/yopeso/Taylor -
Various Swift's AST • SourceKit • swiftc -dump-parse • swiftc
-dump-ast • swiftc -emit-syntax • (swiftc -print-ast) -
SourceKit 1 [ 2 { 3 "type" : "source.lang.swift.syntaxtype.keyword", 4
"offset" : 0, 5 "length" : 4 6 }, 7 { 8 "type" : "source.lang.swift.syntaxtype.identifier", 9 "offset" : 5, 10 "length" : 5 11 }, 12 { 13 "type" : "source.lang.swift.syntaxtype.identifier", 14 "offset" : 11, 15 "length" : 6 16 }, 17 { 18 "type" : "source.lang.swift.syntaxtype.typeidentifier", 19 "offset" : 19, 20 "length" : 6 21 }, 22 ... ... 62 ] 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } -
SourceKitten An adorable little framework and command line tool for
interacting with SourceKit. https://github.com/jpsim/SourceKitten -
swiftc -dump-parse 1 func greet(person: String) -> String { 2
let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" (parameter_list (parameter "person" apiName=person)) (result (type_ident (component id='String' bind=none))) (brace_stmt (pattern_binding_decl (pattern_named 'greeting') (sequence_expr type='<null>' (string_literal_expr type='<null>' encoding=utf8 value (unresolved_decl_ref_expr type='<null>' name=+ functio (declref_expr type='<null>' decl=test.(file).greet(per (unresolved_decl_ref_expr type='<null>' name=+ functio (string_literal_expr type='<null>' encoding=utf8 value (var_decl "greeting" type='<null type>' let storage_kind=s (return_stmt (declref_expr type='<null>' decl=test.(file).greet(perso -
swiftc -dump-ast 1 func greet(person: String) -> String { 2
let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" interface type='(String) -> String' a (parameter_list (parameter "person" apiName=person type='String' interface ty (result (type_ident (component id='String' bind=Swift.(file).String))) (brace_stmt (pattern_binding_decl (pattern_named type='String' 'greeting') (binary_expr type='String' location=test.swift:2:39 range=[ (dot_syntax_call_expr implicit type='(String, String) -> (declref_expr type='(String.Type) -> (String, String) - (type_expr implicit type='String.Type' location=test.sw (tuple_expr implicit type='(String, String)' location=tes (binary_expr type='String' location=test.swift:2:30 ran (dot_syntax_call_expr implicit type='(String, String) (declref_expr type='(String.Type) -> (String, Strin (type_expr implicit type='String.Type' location=tes (tuple_expr implicit type='(String, String)' location (string_literal_expr type='String' location=test.sw (declref_expr type='String' location=test.swift:2:3 (string_literal_expr type='String' location=test.swift: (var_decl "greeting" type='String' interface type='String' ac (return_stmt (declref_expr type='String' location=test.swift:3:12 range= -
swiftc -emit-syntax 1 func greet(person: String) -> String { 2
let greeting = "Hello, " + person + "!" 3 return greeting 4 } 7 { "kind": "FunctionDecl", 8 "layout": [ 9 null, 10 null, 11 { "tokenKind": { 12 "kind": "kw_func" 13 }, 14 "leadingTrivia": [], 15 "trailingTrivia": [ 16 { "kind": "Space", 17 "value": 1 18 } 19 ], 20 "presence": "Present" 21 }, 22 { "tokenKind": { 23 "kind": "identifier", 24 "text": "greet" 25 }, 26 "leadingTrivia": [], 27 "trailingTrivia": [], 28 "presence": "Present" 29 }, 30 null, 31 { "kind": "FunctionSignature", 32 "layout": [ 33 { "kind": "ParameterClause", 34 "layout": [ 35 { "tokenKind": { 36 "kind": "l_paren" 37 }, 38 "leadingTrivia": [], 39 "trailingTrivia": [], 40 "presence": "Present" 41 }, 42 { "kind": "FunctionParameterList", -
SwiftSyntax • Swift wrapper of libSyntax • Parses libSyntax's AST
• Offers AST builder API 8 import Foundation 9 import SwiftSyntax 10 11 class TokenVisitor : SyntaxVisitor { 12 var tokens = [Token]() 13 private var line = [Token]() 14 private var contexts = [Context]() 15 16 override func visit(_ node: ImportDeclSy 17 processNode(node) 18 } 19 20 override func visit(_ node: ClassDeclSyn 21 processNode(node) 22 } 23 24 override func visit(_ node: StructDeclSy 25 processNode(node) -
Comparison Tables Tool Difficulty Type check Source mapping SourceKit SourceKitten
Easy No Partial -emit-syntax SwiftSyntax Medium No Loss-less -dump-ast - Hard Yes Partial -
Convert Swift Source code to HTML 1 func greet(person: String)
-> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 1 <!DOCTYPE html> 2 <html lang="en"> ... 10 <boy> 11 <div class="box FunctionDeclSyntax" data-tooltip 12 <span class='keyword'>func</span> <span c 13 <div class="box VariableDeclSyntax" data-toolt 14 <br> 15 <span class='keyword'> 16 <span class='rightBrace'>}</span> 17 </div> 18 <br> 19 <span class='eof'></span> 20 </body> 21 </html> -
Convert Swift Source code to HTML Swift 4.1 1 import
SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") -
1 import SwiftSyntax 2 3 var html = "" 4
5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML -
1 import SwiftSyntax 2 3 var html = "" 4
5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "<span class='\(kind)'>" + token.text + "</span>" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("<!DOCTYPE html><html><body>\(html)</body></html>") 19 Convert Swift Source code to HTML -
1 <!DOCTYPE html> 2 <html> 3 <body> 4 <span class='funcKeyword'>func</span>
5 <span class='identifier("greet")'>greet</span> 6 <span class='leftParen'>(</span> 7 <span class='identifier("person")'>person</span> 8 <span class='colon'>:</span> 9 <span class='identifier("String")'>String</span> 10 <span class='rightParen'>)</span> 11 <span class='arrow'>-></span> 12 <span class='identifier("String")'>String</span> 13 <span class='leftBrace'>{</span> 14 <span class='letKeyword'>let</span> 15 <span class='identifier("greeting")'>greeting</span> 16 <span class='equal'>=</span> 17 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 18 <span class='spacedBinaryOperator("+")'>+</span> 19 <span class='identifier("person")'>person</span> 20 <span class='spacedBinaryOperator("+")'>+</span> 21 <span class='stringLiteral("\"!\"")'>"!"</span> 22 <span class='returnKeyword'>return</span> 23 <span class='identifier("greeting")'>greeting</span> 24 <span class='rightBrace'>}</span> 25 <span class='eof'></span> 26 </body> 27 </html> -
1 <!DOCTYPE html> 2 <html> 3 <link rel="stylesheet" href="css/default.css" type="text/css"
/> 4 <body> 5 <span class='funcKeyword'>func</span> 6 <span class='identifier("greet")'>greet</span> 7 <span class='leftParen'>(</span> 8 <span class='identifier("person")'>person</span> 9 <span class='colon'>:</span> 10 <span class='identifier("String")'>String</span> 11 <span class='rightParen'>)</span> 12 <span class='arrow'>-></span> 13 <span class='identifier("String")'>String</span> 14 <span class='leftBrace'>{</span> 15 <span class='letKeyword'>let</span> 16 <span class='identifier("greeting")'>greeting</span> 17 <span class='equal'>=</span> 18 <span class='stringLiteral("\"Hello, \"")'>"Hello, "</span> 19 <span class='spacedBinaryOperator("+")'>+</span> 20 <span class='identifier("person")'>person</span> 21 <span class='spacedBinaryOperator("+")'>+</span> 22 <span class='stringLiteral("\"!\"")'>"!"</span> 23 <span class='returnKeyword'>return</span> 24 <span class='identifier("greeting")'>greeting</span> 25 <span class='rightBrace'>}</span> 26 <span class='eof'></span> 27 </body> 28 </html> -
5 class TokenVisitor : SyntaxVisitor { ... 7 var types
= [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36 -
5 class TokenVisitor : SyntaxVisitor { ... 7 var types
= [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("<div class=\"box \(type(of: node))\" data-tooltip=\"\(type(of: node))\">") 15 } 16 } 17 } 18 19 override func visit(_ token: TokenSyntax) { ... ... 27 } 28 29 override func visitPost(_ node: Syntax) { 30 for t in types { 31 if type(of: node) == t { 32 list.append("</div>") 33 } 34 } 35 } 36 -
Aspect Oriented Programming in Swift • Aspect Oriented Programming in
Swift ‣ Insert logging or tracking code automatically 9 import UIKit 10 import PDFKit 11 12 class BookmarkViewController: ... ... 34 35 override func viewDidLoad( 36 super.viewDidLoad() ... ... 46 } 47 48 override func viewWillAppe super.viewWillAppear(a ... ... 49 50 } print(#function) print(#function) -
Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ... -
Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ... -
Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ... -
Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ... -
Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ... -
My Tool Intercept build process SILGen IRGen AST SIL IR
Parser Source AST -
Demo $ ./swiftaop --pattern "^view.+" --advice advice.swift -Xxcodebuild -target BookReader
\ -sdk iphonesimulator build -
SwiftPowerAssert XCTAssert(bar.val == bar.foo.val) | | | | | |
| 3 | | | 2 | | | Foo(val: 2) | | Bar(foo: main.Foo(val: 2), val: 3) | false Bar(foo: main.Foo(val: 2), val: 3) -
... 12 class Tests: XCTestCase { 13 func testMethod() {
14 let bar = Bar(foo: Foo(val: 2), val: 3) 15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)") .assertBoolean(bar.val == bar.foo.val, op: ==) .record(expression: bar, column: 11).record(expression: bar.val, column: 15) .record(expression: (bar.val == bar.foo.val) as (Bool), column: 19) .record(expression: bar, column: 22).record(expression: bar.foo, column: 26) .record(expression: bar.foo.val, column: 30).render() 18 } 19 } 1 class Tests: XCTestCase { 2 func testMethod() { 3 let bar = Bar(foo: Foo(val: 2), val: 3) 4 XCTAssert(bar.val == bar.foo.val) 5 } 6 } -
Wrap up • With AST, we can write code using
source code information. • Meta-programming with AST can eliminate boilerplate code and give dynamic behaviors to Swift. • Swift has several kinds of AST. • Some ASTs can be used easily with tools like SourceKitten and SwiftSyntax. -
Resources • Improving Swift Tools with libSyntax by Harlan Haskins
academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/ • SourceKit and You by JP Simard academy.realm.io/posts/appbuilders-jp-simard-sourcekit/ -
Resources • SwiftPowerAssert https://github.com/kishikawakatsumi/SwiftPowerAssert/ • Swiftfmt https://github.com/kishikawakatsumi/swiftfmt • Swift AST
Explorer https://github.com/kishikawakatsumi/swift-ast-explorer