AST Meta-Programming Writing a Swift tool using AST

Agenda • What's AST? • Tools using AST • Kind of Swift AST • Let's write a tool yourself

What's AST? • Abstract Syntax Tree • Semantic structures form a hierarchical tree • Internal representation of a source code for compiler • Handling source code programmatically FunctionDecl func greet ( person : String ) FunctionSignature FunctionParameterList

Handling source code programmatically • Code Analysis

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery

Tools using AST • Code Analysis Taylor https://github.com/yopeso/Taylor • Lint/Format SwiftLint https://github.com/realm/SwiftLint • Code generation Sourcery https://github.com/krzysztofzablocki/Sourcery DIKit https://github.com/ishkawa/DIKit

Tools using SourceKit's AST • SwiftLint A tool to enforce Swift style and conventions. https://github.com/realm/SwiftLint • Jazzy Soulful docs for Swift & Objective-C https://github.com/realm/jazzy • Sourcery Meta-programming for Swift, stop writing boilerplate code. https://github.com/krzysztofzablocki/Sourcery • DIKit A statically typed dependency injector for Swift. https://github.com/ishkawa/DIKit • Taylor Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms. https://github.com/yopeso/Taylor

Various Swift's AST • SourceKit • swiftc -dump-parse • swiftc -dump-ast • swiftc -emit-syntax • (swiftc -print-ast)

SourceKit 1 [ 2 { 3 "type" : "source.lang.swift.syntaxtype.keyword", 4 "offset" : 0, 5 "length" : 4 6 }, 7 { 8 "type" : "source.lang.swift.syntaxtype.identifier", 9 "offset" : 5, 10 "length" : 5 11 }, 12 { 13 "type" : "source.lang.swift.syntaxtype.identifier", 14 "offset" : 11, 15 "length" : 6 16 }, 17 { 18 "type" : "source.lang.swift.syntaxtype.typeidentifier", 19 "offset" : 19, 20 "length" : 6 21 }, 22 ... ... 62 ] 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 }

SourceKitten An adorable little framework and command line tool for interacting with SourceKit. https://github.com/jpsim/SourceKitten

swiftc -dump-parse 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" (parameter_list (parameter "person" apiName=person)) (result (type_ident (component id='String' bind=none))) (brace_stmt (pattern_binding_decl (pattern_named 'greeting') (sequence_expr type='' (string_literal_expr type='' encoding=utf8 value (unresolved_decl_ref_expr type='' name=+ functio (declref_expr type='' decl=test.(file).greet(per (unresolved_decl_ref_expr type='' name=+ functio (string_literal_expr type='' encoding=utf8 value (var_decl "greeting" type='' let storage_kind=s (return_stmt (declref_expr type='' decl=test.(file).greet(perso

swiftc -dump-ast 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } (source_file (func_decl "greet(person:)" interface type='(String) -> String' a (parameter_list (parameter "person" apiName=person type='String' interface ty (result (type_ident (component id='String' bind=Swift.(file).String))) (brace_stmt (pattern_binding_decl (pattern_named type='String' 'greeting') (binary_expr type='String' location=test.swift:2:39 range=[ (dot_syntax_call_expr implicit type='(String, String) -> (declref_expr type='(String.Type) -> (String, String) - (type_expr implicit type='String.Type' location=test.sw (tuple_expr implicit type='(String, String)' location=tes (binary_expr type='String' location=test.swift:2:30 ran (dot_syntax_call_expr implicit type='(String, String) (declref_expr type='(String.Type) -> (String, Strin (type_expr implicit type='String.Type' location=tes (tuple_expr implicit type='(String, String)' location (string_literal_expr type='String' location=test.sw (declref_expr type='String' location=test.swift:2:3 (string_literal_expr type='String' location=test.swift: (var_decl "greeting" type='String' interface type='String' ac (return_stmt (declref_expr type='String' location=test.swift:3:12 range=

swiftc -emit-syntax 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 7 { "kind": "FunctionDecl", 8 "layout": [ 9 null, 10 null, 11 { "tokenKind": { 12 "kind": "kw_func" 13 }, 14 "leadingTrivia": [], 15 "trailingTrivia": [ 16 { "kind": "Space", 17 "value": 1 18 } 19 ], 20 "presence": "Present" 21 }, 22 { "tokenKind": { 23 "kind": "identifier", 24 "text": "greet" 25 }, 26 "leadingTrivia": [], 27 "trailingTrivia": [], 28 "presence": "Present" 29 }, 30 null, 31 { "kind": "FunctionSignature", 32 "layout": [ 33 { "kind": "ParameterClause", 34 "layout": [ 35 { "tokenKind": { 36 "kind": "l_paren" 37 }, 38 "leadingTrivia": [], 39 "trailingTrivia": [], 40 "presence": "Present" 41 }, 42 { "kind": "FunctionParameterList",

SwiftSyntax • Swift wrapper of libSyntax • Parses libSyntax's AST • Offers AST builder API 8 import Foundation 9 import SwiftSyntax 10 11 class TokenVisitor : SyntaxVisitor { 12 var tokens = [Token]() 13 private var line = [Token]() 14 private var contexts = [Context]() 15 16 override func visit(_ node: ImportDeclSy 17 processNode(node) 18 } 19 20 override func visit(_ node: ClassDeclSyn 21 processNode(node) 22 } 23 24 override func visit(_ node: StructDeclSy 25 processNode(node)

Comparison Tables Tool Difficulty Type check Source mapping SourceKit SourceKitten Easy No Partial -emit-syntax SwiftSyntax Medium No Loss-less -dump-ast - Hard Yes Partial

Practice makes perfect Let's write a tool

Convert Swift Source code to HTML 1 func greet(person: String) -> String { 2 let greeting = "Hello, " + person + "!" 3 return greeting 4 } 1 2 ... 10 11 18
19 20 21

Convert Swift Source code to HTML Swift 4.1 1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "" + token.text + "" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("\(html)")

1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "" + token.text + "" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("\(html)") 19 Convert Swift Source code to HTML

1 import SwiftSyntax 2 3 var html = "" 4 5 class TokenVisitor : SyntaxVisitor { 6 override func visit(_ token: TokenSyntax) { 7 let kind = "\(token.tokenKind)" 8 html += "" + token.text + "" 9 } 10 } 11 12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0]) 13 14 let sourceFile = try! SourceFileSyntax.parse(filePath) 15 let visitor = TokenVisitor() 16 visitor.visit(sourceFile) 17 18 print("\(html)") 19 Convert Swift Source code to HTML

1 2 3 4 func 5 greet 6 ( 7 person 8 : 9 String 10 ) 11 -> 12 String 13 { 14 let 15 greeting 16 = 17 "Hello, " 18 + 19 person 20 + 21 "!" 22 return 23 greeting 24 } 25 26 27

1 2 3 4 5 func 6 greet 7 ( 8 person 9 : 10 String 11 ) 12 -> 13 String 14 { 15 let 16 greeting 17 = 18 "Hello, " 19 + 20 person 21 + 22 "!" 23 return 24 greeting 25 } 26 27 28

No content

5 class TokenVisitor : SyntaxVisitor { ... 7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("") 33 } 34 } 35 } 36

5 class TokenVisitor : SyntaxVisitor { ... 7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self 8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe 9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type] 10 11 override func visitPre(_ node: Syntax) { 12 for t in types { 13 if type(of: node) == t { 14 list.append("") 33 } 34 } 35 } 36

No content

Swift AST Explorer swift-ast-explorer.kishikawakatsumi.com

Aspect Oriented Programming in Swift • Aspect Oriented Programming in Swift ‣ Insert logging or tracking code automatically 9 import UIKit 10 import PDFKit 11 12 class BookmarkViewController: ... ... 34 35 override func viewDidLoad( 36 super.viewDidLoad() ... ... 46 } 47 48 override func viewWillAppe super.viewWillAppear(a ... ... 49 50 } print(#function) print(#function)

Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...

Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...

Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...

Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...

Hook method invocation ... 23 class FunctionVisitor : SyntaxRewriter { 24 override func visit(_ node: CodeBlockSyntax) -> Syntax { 25 if let parent = node.parent as? FunctionDeclSyntax { 26 let signature = "\(parent.identifier)\(parent.signature)" 27 let regex = try! NSRegularExpression(pattern: "^view.+", options: []) 28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location: 29 var node = node 30 for item in adviceVisitor.items.reversed() { 31 node = node.withStatements(node.statements.inserting(item, at: 0)) 32 } 33 return node 34 } 35 } 36 return node 37 } 38 } ...

Intercept build process Parser TypeChecker IRGen AST SIL IR Source

My Tool Intercept build process SILGen IRGen AST SIL IR Parser Source AST

My Tool Intercept build process SILGen IRGen SIL IR Parser Source AST

Demo $ ./swiftaop --pattern "^view.+" --advice advice.swift -Xxcodebuild -target BookReader \ -sdk iphonesimulator build

SwiftPowerAssert Provide describe assertion message

SwiftPowerAssert XCTAssert(bar.val == bar.foo.val) | | | | | | | 3 | | | 2 | | | Foo(val: 2) | | Bar(foo: main.Foo(val: 2), val: 3) | false Bar(foo: main.Foo(val: 2), val: 3)

... 12 class Tests: XCTestCase { 13 func testMethod() { 14 let bar = Bar(foo: Foo(val: 2), val: 3) 15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)") .assertBoolean(bar.val == bar.foo.val, op: ==) .record(expression: bar, column: 11).record(expression: bar.val, column: 15) .record(expression: (bar.val == bar.foo.val) as (Bool), column: 19) .record(expression: bar, column: 22).record(expression: bar.foo, column: 26) .record(expression: bar.foo.val, column: 30).render() 18 } 19 } 1 class Tests: XCTestCase { 2 func testMethod() { 3 let bar = Bar(foo: Foo(val: 2), val: 3) 4 XCTAssert(bar.val == bar.foo.val) 5 } 6 }

SwiftPowerAssert Playground swift-power-assert.kishikawakatsumi.com

No content

swiftfmt Code formatter for Swift

Swiftfmt Playground swiftfmt.kishikawakatsumi.com

Wrap up • With AST, we can write code using source code information. • Meta-programming with AST can eliminate boilerplate code and give dynamic behaviors to Swift. • Swift has several kinds of AST. • Some ASTs can be used easily with tools like SourceKitten and SwiftSyntax.

Resources • Improving Swift Tools with libSyntax by Harlan Haskins academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/ • SourceKit and You by JP Simard academy.realm.io/posts/appbuilders-jp-simard-sourcekit/

Resources • SwiftPowerAssert https://github.com/kishikawakatsumi/SwiftPowerAssert/ • Swiftfmt https://github.com/kishikawakatsumi/swiftfmt • Swift AST Explorer https://github.com/kishikawakatsumi/swift-ast-explorer