Quick Overview of SwiftParser 1 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Swift Compiler 2 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Swift Compiler 3 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Parsers → libParse in C++ for Swift → SwiftSyntax in Swift for Swift Macros 4 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

libParse 5 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftSyntax + ASTGen 6 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Swift Macros Swift Macros implementations will be executed in a sandbox 7 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Sandbox for Swift Macros 8 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftSyntax 9 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftSyntax SwiftSyntax is a set of Swift libraries for parsing, inspecting, generating, and transforming Swift source code. 10 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftSyntax 11 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Mainline SwiftSyntax → SwiftParser for natively parsing source code → SwiftOperators for folding SwiftSyntax trees containing operators → SwiftSyntaxBuilder for generating Swift code with a result builder-style interface → SwiftSyntaxMacros for providing syntactic macro 12 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftParser The SwiftParser framework implements a parser that accepts Swift source text as input and produces a SwiftSyntax syntax tree. 13 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SwiftParser 14 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Parser.parse import SwiftParser import SwiftSyntax let sourceText = """ let a = 1 """ // Parse the source code in sourceText into a syntax tree let sourceFile: SourceFileSyntax = Parser.parse(source: sourceText) // Visualize the complete syntax tree. dump(sourceFile) 15 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Source text let a = 1 16 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Syntax tree // let a = 1 SourceFile CodeBlockItemList CodeBlockItem VariableDecl let PatternBindingList PatternBinding IdentifierPattern a InitializerClause = IntegerLiteralExpr 1 17 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Lexer 18 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Token // let a = 1 [ Lexer.Lexeme(tokenKind: .keyword, ...), Lexer.Lexeme(tokenKind: .identifier, ...), Lexer.Lexeme(tokenKind: .equal, ...), Lexer.Lexeme(tokenKind: .integerLiteral, ...), ] 19 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

TokenKind(RawTokenKind) public enum TokenKind: Hashable { case arrow case atSign case backslash case backtick case binaryOperator(String) case colon case comma ... } 20 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Keyword public enum Keyword: UInt8, Hashable { case accesses case actor case addressWithNativeOwner case addressWithOwner case any case `Any` case `as` ... 21 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

How to lex number 22 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Lexing Basics → peek(at:) → is(offset:, at), is(offset:, notAt) → advance(matching:), advance(if:), advance(while:) 23 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 24 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 25 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 26 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 27 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 28 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 29 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 30 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 31 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Parser 32 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Syntax tree // let a = 1 SourceFile CodeBlockItemList CodeBlockItem VariableDecl let PatternBindingList PatternBinding IdentifierPattern a InitializerClause = IntegerLiteralExpr 1 33 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SyntaxNodes → SyntaxDeclNodes → SyntaxExprNodes → SyntaxNodes → SyntaxPatternNodes → SyntaxStmtNodes → SyntaxTypeNodes 34 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

SyntaxKind public enum SyntaxKind: CaseIterable { case token case accessorBlock case accessorDeclList case accessorDecl case accessorEffectSpecifiers case accessorParameters case actorDecl case arrayElementList ... } 35 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

How to parse integer literal 36 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Parsing Basics → peek(isAt:) → at(anyIn:), at(prefix:) → eat → consume(if:), consume(ifAnyIn:), consume(ifPrefix:, as) 37 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 38 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 39 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 40 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 41 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)

Recap → Much readable than C++ → Swift Macros → Performance comparison with libParse 42 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)