Slide 1
Slide 1 text
AST Meta-Programming
Writing a Swift tool using AST
Slide 2
Slide 2 text
Agenda
• What's AST?
• Tools using AST
• Kind of Swift AST
• Let's write a tool yourself
Slide 3
Slide 3 text
What's AST?
• Abstract Syntax Tree
• Semantic structures form a hierarchical tree
• Internal representation of a source code for
compiler
• Handling source code programmatically
FunctionDecl
func greet
(
person : String
)
FunctionSignature
FunctionParameterList
Slide 4
Slide 4 text
Handling source code programmatically
• Code Analysis
Slide 5
Slide 5 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
Slide 6
Slide 6 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
• Lint/Format
Slide 7
Slide 7 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
• Lint/Format
SwiftLint
https://github.com/realm/SwiftLint
Slide 8
Slide 8 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
• Lint/Format
SwiftLint
https://github.com/realm/SwiftLint
• Code generation
Slide 9
Slide 9 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
• Lint/Format
SwiftLint
https://github.com/realm/SwiftLint
• Code generation
Sourcery
https://github.com/krzysztofzablocki/Sourcery
Slide 10
Slide 10 text
Tools using AST
• Code Analysis
Taylor
https://github.com/yopeso/Taylor
• Lint/Format
SwiftLint
https://github.com/realm/SwiftLint
• Code generation
Sourcery
https://github.com/krzysztofzablocki/Sourcery
DIKit
https://github.com/ishkawa/DIKit
Slide 11
Slide 11 text
Tools using SourceKit's AST
• SwiftLint
A tool to enforce Swift style and conventions.
https://github.com/realm/SwiftLint
• Jazzy
Soulful docs for Swift & Objective-C
https://github.com/realm/jazzy
• Sourcery
Meta-programming for Swift, stop writing boilerplate code.
https://github.com/krzysztofzablocki/Sourcery
• DIKit
A statically typed dependency injector for Swift.
https://github.com/ishkawa/DIKit
• Taylor
Measure Swift code metrics and get reports in Xcode, Jenkins and other CI platforms.
https://github.com/yopeso/Taylor
Slide 12
Slide 12 text
Various Swift's AST
• SourceKit
• swiftc -dump-parse
• swiftc -dump-ast
• swiftc -emit-syntax
• (swiftc -print-ast)
Slide 13
Slide 13 text
SourceKit
1 [
2 {
3 "type" : "source.lang.swift.syntaxtype.keyword",
4 "offset" : 0,
5 "length" : 4
6 },
7 {
8 "type" : "source.lang.swift.syntaxtype.identifier",
9 "offset" : 5,
10 "length" : 5
11 },
12 {
13 "type" : "source.lang.swift.syntaxtype.identifier",
14 "offset" : 11,
15 "length" : 6
16 },
17 {
18 "type" : "source.lang.swift.syntaxtype.typeidentifier",
19 "offset" : 19,
20 "length" : 6
21 },
22 ...
...
62 ]
1 func greet(person: String) -> String {
2 let greeting = "Hello, " + person + "!"
3 return greeting
4 }
Slide 14
Slide 14 text
SourceKitten
An adorable little framework and command line tool for interacting with SourceKit.
https://github.com/jpsim/SourceKitten
Slide 15
Slide 15 text
swiftc -dump-parse
1 func greet(person: String) -> String {
2 let greeting = "Hello, " + person + "!"
3 return greeting
4 }
(source_file
(func_decl "greet(person:)"
(parameter_list
(parameter "person" apiName=person))
(result
(type_ident
(component id='String' bind=none)))
(brace_stmt
(pattern_binding_decl
(pattern_named 'greeting')
(sequence_expr type=''
(string_literal_expr type='' encoding=utf8 value
(unresolved_decl_ref_expr type='' name=+ functio
(declref_expr type='' decl=test.(file).greet(per
(unresolved_decl_ref_expr type='' name=+ functio
(string_literal_expr type='' encoding=utf8 value
(var_decl "greeting" type='' let storage_kind=s
(return_stmt
(declref_expr type='' decl=test.(file).greet(perso
Slide 16
Slide 16 text
swiftc -dump-ast
1 func greet(person: String) -> String {
2 let greeting = "Hello, " + person + "!"
3 return greeting
4 }
(source_file
(func_decl "greet(person:)" interface type='(String) -> String' a
(parameter_list
(parameter "person" apiName=person type='String' interface ty
(result
(type_ident
(component id='String' bind=Swift.(file).String)))
(brace_stmt
(pattern_binding_decl
(pattern_named type='String' 'greeting')
(binary_expr type='String' location=test.swift:2:39 range=[
(dot_syntax_call_expr implicit type='(String, String) ->
(declref_expr type='(String.Type) -> (String, String) -
(type_expr implicit type='String.Type' location=test.sw
(tuple_expr implicit type='(String, String)' location=tes
(binary_expr type='String' location=test.swift:2:30 ran
(dot_syntax_call_expr implicit type='(String, String)
(declref_expr type='(String.Type) -> (String, Strin
(type_expr implicit type='String.Type' location=tes
(tuple_expr implicit type='(String, String)' location
(string_literal_expr type='String' location=test.sw
(declref_expr type='String' location=test.swift:2:3
(string_literal_expr type='String' location=test.swift:
(var_decl "greeting" type='String' interface type='String' ac
(return_stmt
(declref_expr type='String' location=test.swift:3:12 range=
Slide 17
Slide 17 text
swiftc -emit-syntax
1 func greet(person: String) -> String {
2 let greeting = "Hello, " + person + "!"
3 return greeting
4 }
7 { "kind": "FunctionDecl",
8 "layout": [
9 null,
10 null,
11 { "tokenKind": {
12 "kind": "kw_func"
13 },
14 "leadingTrivia": [],
15 "trailingTrivia": [
16 { "kind": "Space",
17 "value": 1
18 }
19 ],
20 "presence": "Present"
21 },
22 { "tokenKind": {
23 "kind": "identifier",
24 "text": "greet"
25 },
26 "leadingTrivia": [],
27 "trailingTrivia": [],
28 "presence": "Present"
29 },
30 null,
31 { "kind": "FunctionSignature",
32 "layout": [
33 { "kind": "ParameterClause",
34 "layout": [
35 { "tokenKind": {
36 "kind": "l_paren"
37 },
38 "leadingTrivia": [],
39 "trailingTrivia": [],
40 "presence": "Present"
41 },
42 { "kind": "FunctionParameterList",
Slide 18
Slide 18 text
SwiftSyntax
• Swift wrapper of libSyntax
• Parses libSyntax's AST
• Offers AST builder API
8 import Foundation
9 import SwiftSyntax
10
11 class TokenVisitor : SyntaxVisitor {
12 var tokens = [Token]()
13 private var line = [Token]()
14 private var contexts = [Context]()
15
16 override func visit(_ node: ImportDeclSy
17 processNode(node)
18 }
19
20 override func visit(_ node: ClassDeclSyn
21 processNode(node)
22 }
23
24 override func visit(_ node: StructDeclSy
25 processNode(node)
Slide 19
Slide 19 text
Comparison Tables
Tool Difficulty Type check
Source
mapping
SourceKit SourceKitten Easy No Partial
-emit-syntax SwiftSyntax Medium No Loss-less
-dump-ast - Hard Yes Partial
Slide 20
Slide 20 text
Practice makes perfect
Let's write a tool
Slide 21
Slide 21 text
Convert Swift Source code to HTML
1 func greet(person: String) -> String {
2 let greeting = "Hello, " + person + "!"
3 return greeting
4 }
1
2
...
10
11
19 20 21
func
15
16 }
17
18 19 20 21
Slide 22
Slide 22 text
Convert Swift Source code to HTML
Swift 4.1
1 import SwiftSyntax
2
3 var html = ""
4
5 class TokenVisitor : SyntaxVisitor {
6 override func visit(_ token: TokenSyntax) {
7 let kind = "\(token.tokenKind)"
8 html += "" + token.text + ""
9 }
10 }
11
12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0])
13
14 let sourceFile = try! SourceFileSyntax.parse(filePath)
15 let visitor = TokenVisitor()
16 visitor.visit(sourceFile)
17
18 print("\(html)")
Slide 23
Slide 23 text
1 import SwiftSyntax
2
3 var html = ""
4
5 class TokenVisitor : SyntaxVisitor {
6 override func visit(_ token: TokenSyntax) {
7 let kind = "\(token.tokenKind)"
8 html += "" + token.text + ""
9 }
10 }
11
12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0])
13
14 let sourceFile = try! SourceFileSyntax.parse(filePath)
15 let visitor = TokenVisitor()
16 visitor.visit(sourceFile)
17
18 print("\(html)")
19
Convert Swift Source code to HTML
Slide 24
Slide 24 text
1 import SwiftSyntax
2
3 var html = ""
4
5 class TokenVisitor : SyntaxVisitor {
6 override func visit(_ token: TokenSyntax) {
7 let kind = "\(token.tokenKind)"
8 html += "" + token.text + ""
9 }
10 }
11
12 let filePath = URL(fileURLWithPath: CommandLine.arguments[0])
13
14 let sourceFile = try! SourceFileSyntax.parse(filePath)
15 let visitor = TokenVisitor()
16 visitor.visit(sourceFile)
17
18 print("\(html)")
19
Convert Swift Source code to HTML
Slide 25
Slide 25 text
1
2
3
4 func
5 greet
6 (
7 person
8 :
9 String
10 )
11 ->
12 String
13 {
14 let
15 greeting
16 =
17 "Hello, "
18 +
19 person
20 +
21 "!"
22 return
23 greeting
24 }
25
26
27
Slide 26
Slide 26 text
1
2
3
4
5 func
6 greet
7 (
8 person
9 :
10 String
11 )
12 ->
13 String
14 {
15 let
16 greeting
17 =
18 "Hello, "
19 +
20 person
21 +
22 "!"
23 return
24 greeting
25 }
26
27
28
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
5 class TokenVisitor : SyntaxVisitor {
...
7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self
8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe
9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type]
10
11 override func visitPre(_ node: Syntax) {
12 for t in types {
13 if type(of: node) == t {
14 list.append("
")
15 }
16 }
17 }
18
19 override func visit(_ token: TokenSyntax) {
... ...
27 }
28
29 override func visitPost(_ node: Syntax) {
30 for t in types {
31 if type(of: node) == t {
32 list.append("
")
33 }
34 }
35 }
36
Slide 29
Slide 29 text
5 class TokenVisitor : SyntaxVisitor {
...
7 var types = [ImportDeclSyntax.self, StructDeclSyntax.self, ClassDeclSyntax.self, UnknownDeclSyntax.self
8 IfStmtSyntax.self, SwitchStmtSyntax.self, ForInStmtSyntax.self, WhileStmtSyntax.self, Repe
9 DoStmtSyntax.self, CatchClauseSyntax.self, FunctionCallExprSyntax.self] as [Any.Type]
10
11 override func visitPre(_ node: Syntax) {
12 for t in types {
13 if type(of: node) == t {
14 list.append("
")
15 }
16 }
17 }
18
19 override func visit(_ token: TokenSyntax) {
... ...
27 }
28
29 override func visitPost(_ node: Syntax) {
30 for t in types {
31 if type(of: node) == t {
32 list.append("
")
33 }
34 }
35 }
36
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
Swift AST Explorer
swift-ast-explorer.kishikawakatsumi.com
Slide 32
Slide 32 text
Aspect Oriented Programming in Swift
• Aspect Oriented Programming in Swift
‣ Insert logging or tracking code automatically
9 import UIKit
10 import PDFKit
11
12 class BookmarkViewController:
... ...
34
35 override func viewDidLoad(
36 super.viewDidLoad()
... ...
46 }
47
48 override func viewWillAppe
super.viewWillAppear(a
... ...
49
50 }
print(#function)
print(#function)
Slide 33
Slide 33 text
Hook method invocation
...
23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax {
25 if let parent = node.parent as? FunctionDeclSyntax {
26 let signature = "\(parent.identifier)\(parent.signature)"
27 let regex = try! NSRegularExpression(pattern: "^view.+", options: [])
28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location:
29 var node = node
30 for item in adviceVisitor.items.reversed() {
31 node = node.withStatements(node.statements.inserting(item, at: 0))
32 }
33 return node
34 }
35 }
36 return node
37 }
38 }
...
Slide 34
Slide 34 text
Hook method invocation
...
23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax {
25 if let parent = node.parent as? FunctionDeclSyntax {
26 let signature = "\(parent.identifier)\(parent.signature)"
27 let regex = try! NSRegularExpression(pattern: "^view.+", options: [])
28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location:
29 var node = node
30 for item in adviceVisitor.items.reversed() {
31 node = node.withStatements(node.statements.inserting(item, at: 0))
32 }
33 return node
34 }
35 }
36 return node
37 }
38 }
...
Slide 35
Slide 35 text
Hook method invocation
...
23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax {
25 if let parent = node.parent as? FunctionDeclSyntax {
26 let signature = "\(parent.identifier)\(parent.signature)"
27 let regex = try! NSRegularExpression(pattern: "^view.+", options: [])
28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location:
29 var node = node
30 for item in adviceVisitor.items.reversed() {
31 node = node.withStatements(node.statements.inserting(item, at: 0))
32 }
33 return node
34 }
35 }
36 return node
37 }
38 }
...
Slide 36
Slide 36 text
Hook method invocation
...
23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax {
25 if let parent = node.parent as? FunctionDeclSyntax {
26 let signature = "\(parent.identifier)\(parent.signature)"
27 let regex = try! NSRegularExpression(pattern: "^view.+", options: [])
28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location:
29 var node = node
30 for item in adviceVisitor.items.reversed() {
31 node = node.withStatements(node.statements.inserting(item, at: 0))
32 }
33 return node
34 }
35 }
36 return node
37 }
38 }
...
Slide 37
Slide 37 text
Hook method invocation
...
23 class FunctionVisitor : SyntaxRewriter {
24 override func visit(_ node: CodeBlockSyntax) -> Syntax {
25 if let parent = node.parent as? FunctionDeclSyntax {
26 let signature = "\(parent.identifier)\(parent.signature)"
27 let regex = try! NSRegularExpression(pattern: "^view.+", options: [])
28 if let _ = regex.firstMatch(in: signature, options: [], range: NSRange(location:
29 var node = node
30 for item in adviceVisitor.items.reversed() {
31 node = node.withStatements(node.statements.inserting(item, at: 0))
32 }
33 return node
34 }
35 }
36 return node
37 }
38 }
...
Slide 38
Slide 38 text
Intercept build process
Parser TypeChecker IRGen
AST SIL IR
Source
Slide 39
Slide 39 text
My Tool
Intercept build process
SILGen IRGen
AST SIL IR
Parser
Source AST
Slide 40
Slide 40 text
My Tool
Intercept build process
SILGen IRGen
SIL IR
Parser
Source AST
Slide 41
Slide 41 text
Demo
$ ./swiftaop --pattern "^view.+" --advice advice.swift -Xxcodebuild -target BookReader \
-sdk iphonesimulator build
Slide 42
Slide 42 text
SwiftPowerAssert
Provide describe assertion message
Slide 43
Slide 43 text
SwiftPowerAssert
XCTAssert(bar.val == bar.foo.val)
| | | | | |
| 3 | | | 2
| | | Foo(val: 2)
| | Bar(foo: main.Foo(val: 2), val: 3)
| false
Bar(foo: main.Foo(val: 2), val: 3)
Slide 44
Slide 44 text
...
12 class Tests: XCTestCase {
13 func testMethod() {
14 let bar = Bar(foo: Foo(val: 2), val: 3)
15 __ValueRecorder(assertion: "XCTAssert(bar.val == bar.foo.val)")
.assertBoolean(bar.val == bar.foo.val, op: ==)
.record(expression: bar, column: 11).record(expression: bar.val, column: 15)
.record(expression: (bar.val == bar.foo.val) as (Bool), column: 19)
.record(expression: bar, column: 22).record(expression: bar.foo, column: 26)
.record(expression: bar.foo.val, column: 30).render()
18 }
19 }
1 class Tests: XCTestCase {
2 func testMethod() {
3 let bar = Bar(foo: Foo(val: 2), val: 3)
4 XCTAssert(bar.val == bar.foo.val)
5 }
6 }
Slide 45
Slide 45 text
SwiftPowerAssert Playground
swift-power-assert.kishikawakatsumi.com
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
swiftfmt
Code formatter for Swift
Slide 48
Slide 48 text
Swiftfmt Playground
swiftfmt.kishikawakatsumi.com
Slide 49
Slide 49 text
Wrap up
• With AST, we can write code using source code information.
• Meta-programming with AST can eliminate boilerplate code and give
dynamic behaviors to Swift.
• Swift has several kinds of AST.
• Some ASTs can be used easily with tools like SourceKitten and
SwiftSyntax.
Slide 50
Slide 50 text
Resources
• Improving Swift Tools with libSyntax
by Harlan Haskins
academy.realm.io/posts/improving-swift-tools-with-libsyntax-try-swift-haskin-2017/
• SourceKit and You
by JP Simard
academy.realm.io/posts/appbuilders-jp-simard-sourcekit/
Slide 51
Slide 51 text
Resources
• SwiftPowerAssert
https://github.com/kishikawakatsumi/SwiftPowerAssert/
• Swiftfmt
https://github.com/kishikawakatsumi/swiftfmt
• Swift AST Explorer
https://github.com/kishikawakatsumi/swift-ast-explorer