SwiftSyntax: for Good and Evil (inc. Presenter Notes)

SWIFTSYNTAX FOR GOOD AND EVIL Hi, my name is Nick
and my talk today is titled Swift Syntax: For good and evil Oh I'm sorry, that's a typo

SWIFTSYNTAX FOR GOOD AND EVIL EVAL It should be Swift
Syntax: For good and Eval

Q. WHAT IS SWIFTSYNTAX? So what is SwiftSyntax?

A. A SET OF SWIFT LIBRARIES FOR PARSING, INSPECTING, GENERATING, AND
TRANSFORMING SWIFT SOURCE CODE1 1 https://github.com/swiftlang/swift-syntax The SwiftSyntax github page describes it as "A set of Swift libraries for parsing, inspecting, generating, and transforming Swift source code" Why a "set" of libraries?

SWIFTSYNTAX MODULES // From swift-syntax Package.swift .library(name: "SwiftBasicFormat", targets: ["SwiftBasicFormat"]),
.library(name: "SwiftCompilerPlugin", targets: ["SwiftCompilerPlugin"]), .library(name: "SwiftDiagnostics", targets: ["SwiftDiagnostics"]), .library(name: "SwiftIDEUtils", targets: ["SwiftIDEUtils"]), .library(name: "SwiftOperators", targets: ["SwiftOperators"]), .library(name: "SwiftParser", targets: ["SwiftParser"]), .library(name: "SwiftParserDiagnostics", targets: ["SwiftParserDiagnostics"]), .library(name: "SwiftRefactor", targets: ["SwiftRefactor"]), .library(name: "SwiftSyntax", targets: ["SwiftSyntax"]), .library(name: "SwiftSyntaxBuilder", targets: ["SwiftSyntaxBuilder"]), .library(name: "SwiftSyntaxMacros", targets: ["SwiftSyntaxMacros"]), .library(name: "SwiftSyntaxMacroExpansion", targets: ["SwiftSyntaxMacroExpansion"]), .library(name: "SwiftSyntaxMacrosTestSupport", targets: ["SwiftSyntaxMacrosTestSupport"]), .library(name: "SwiftSyntaxMacrosGenericTestSupport", targets: ["SwiftSyntaxMacrosGenericTestSupport"]), .library(name: "_SwiftCompilerPluginMessageHandling", targets: ["SwiftCompilerPluginMessageHandling"]), .library(name: "_SwiftLibraryPluginProvider", targets: ["SwiftLibraryPluginProvider"]), Well swiftsyntax isn't just one library; The SwiftSyntax package actually contains sixteen separate modules - not including tests and internal helpers

SWIFTSYNTAX MODULES // From swift-syntax Package.swift .library(name: "SwiftBasicFormat", targets: ["SwiftBasicFormat"]),
.library(name: "SwiftCompilerPlugin", targets: ["SwiftCompilerPlugin"]), .library(name: "SwiftDiagnostics", targets: ["SwiftDiagnostics"]), .library(name: "SwiftIDEUtils", targets: ["SwiftIDEUtils"]), .library(name: "SwiftOperators", targets: ["SwiftOperators"]), .library(name: "SwiftParser", targets: ["SwiftParser"]), .library(name: "SwiftParserDiagnostics", targets: ["SwiftParserDiagnostics"]), .library(name: "SwiftRefactor", targets: ["SwiftRefactor"]), .library(name: "SwiftSyntax", targets: ["SwiftSyntax"]), .library(name: "SwiftSyntaxBuilder", targets: ["SwiftSyntaxBuilder"]), .library(name: "SwiftSyntaxMacros", targets: ["SwiftSyntaxMacros"]), .library(name: "SwiftSyntaxMacroExpansion", targets: ["SwiftSyntaxMacroExpansion"]), .library(name: "SwiftSyntaxMacrosTestSupport", targets: ["SwiftSyntaxMacrosTestSupport"]), .library(name: "SwiftSyntaxMacrosGenericTestSupport", targets: ["SwiftSyntaxMacrosGenericTestSupport"]), .library(name: "_SwiftCompilerPluginMessageHandling", targets: ["SwiftCompilerPluginMessageHandling"]), .library(name: "_SwiftLibraryPluginProvider", targets: ["SwiftLibraryPluginProvider"]), I won't attempt to cover them all, but these are the ones we'll be looking at today

Q. HOW DOES IT WORK? So how does SwiftSyntax actually
work

THE SYNTAX TREE At the core of SwiftSyntax is a
data structure called the "syntax tree" The tree consists of nodes, each representing part of a Swift program

THE SYNTAX TREE Tokens Declarations Expressions Statements Source File The
root of the syntax tree is a source file

THE SYNTAX TREE Tokens Declarations Expressions Statements Source File From
the root grow branches that represent statements...

THE SYNTAX TREE Tokens Declarations Expressions Statements Source File expressions...

THE SYNTAX TREE Tokens Declarations Expressions Statements Source File and
declarations

THE SYNTAX TREE Tokens Declarations Expressions Statements Source File The
leaves of the tree are tokens; indivisible source code elements such as keywords, identifiers and literals that have no children of their own You might have heard of an Abstract Syntax Tree - an AST This is a CST - a Concrete Syntax Tree In an AST, only the actual statements, expressions and so on are stored In a concrete tree, everything from the original source code is preserved, including comments and whitespace

SYNTAX PROTOCOL All nodes in the SwiftSyntax tree conform to
SyntaxProtocol

SYNTAX PROTOCOL Tokens Declarations Expressions Statements Source File Every distinct
syntax element has its own struct type

SYNTAX PROTOCOL struct TokenSyntax: SyntaxProtocol Declarations Expressions Statements Source File
Tokens are represented by a TokenSyntax

SYNTAX PROTOCOL struct TokenSyntax: SyntaxProtocol struct FunctionDeclSyntax: DeclSyntaxProtocol Expressions Statements
Source File Function declarations are represented by FunctionDeclSyntax... (why "Function" in full but "Decl" for declaration? Your guess is as good as mine!)

SYNTAX PROTOCOL struct TokenSyntax: SyntaxProtocol struct FunctionDeclSyntax: DeclSyntaxProtocol struct TupleExprSyntax:
ExprSyntaxProtocol Statements Source File and so on...

ExprSyntaxProtocol struct ForStmtSyntax: StmtSyntaxProtcol Source File and so on...

ExprSyntaxProtocol struct ForStmtSyntax: StmtSyntaxProtcol struct SourceFileSyntax: SyntaxProtocol ... all the way down to SourceFileSyntax

Q. WHAT IS IT GOOD FOR? What is SwiftSyntax useful
for?

USE CASES ‣Language Features → Conditional compilation, macros SwiftSyntax is
already used for several things within the Swift compiler itself, such as evaluating conditional compilation directives, and Swift Macros of course

USE CASES ‣Language Features → Conditional compilation, macros ‣IDE Tools
→ Swift migration tool, refactoring, fixits and formatting It's also used within Xcode - for the Swift migrator and refactoring tools, fixits for erros and warnings, and for code formatting

→ Swift migration tool, refactoring, fixits and formatting ‣Linting → Detecting style violations and enforcing best practices And beyond that there are many other useful applications, such as linting...

→ Swift migration tool, refactoring, fixits and formatting ‣Linting → Detecting style violations and enforcing best practices ‣Static Analysis → Detecting dead code, deprecated APIs, etc. static analysis...

→ Swift migration tool, refactoring, fixits and formatting ‣Linting → Detecting style violations and enforcing best practices ‣Static Analysis → Detecting dead code, deprecated APIs, etc. ‣Code generation → Creating and modifying source code safely Or code generation But there are many great tools for all of these things already So what if we tried something a bit different?

EVAL Most scripting languages have some form of eval() function
- Javascript, Ruby, Python, PHP all have this concept Eval is a function that takes a string of source code and runs it Languages that support runtime code evaluation make it easy to write apps that are scriptable, or can be extended with plugins to customize their behavior or support new workflows

SWIFT EVAL Swift is my favorite language, and so I've
often wished there was an eval function for Swift But swift is a compiled language, and the Swift compiler is a very complex, monolithic piece of software It's not a dependency that you can easily include in a mac or iOS app, especially not on the App Store But now we have SwiftSyntax to take care of the parsing side, all we'd need to do is the actual evaluation part - how hard could that be? Let's find out!

EVALUATING EXPRESSIONS let source = "1 + 2 * 3"
We'll start with just a simple mathematical expression No functions, variables or loops - just numbers and operators

EVALUATING EXPRESSIONS import SwiftParser let source = "1 + 2
* 3" Before we can evaluate the expression, we'll need to parse it... So let's import the SwiftParser module...

* 3" let sourceSyntax = Parser.parse(source: source) debugPrint(sourceSyntax) Next we'll call Parser.parse() to generate the syntax tree and use debugPrint to print it out

* 3" let sourceSyntax = Parser.parse(source: source) debugPrint(sourceSyntax) SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: SequenceExprSyntax │ ╰─elements: ExprListSyntax │ ├─[0]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─[1]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ├─[2]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─[3]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─[4]: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile The result is what we see on the right There's a lot going on here but for now let's focus on the SequenceExprSyntax node

* 3" let sourceSyntax = Parser.parse(source: source) debugPrint(sourceSyntax) SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: SequenceExprSyntax │ ╰─elements: ExprListSyntax │ ├─[0]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─[1]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ├─[2]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─[3]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─[4]: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile SequenceExprSyntax is how expressions are represented in SwiftSyntax - basically it's a flat list of operators and operands And here we have our first hurdle:

PROBLEM: OPERATOR SOUP A problem I like to call "Operator
soup"

ORDER OF OPERATIONS 1 + 2 * 3 As Sophie
expertly explained to us on Wednesday, mathematical expressions can't just be evaluated from left to right Some operators have higher precedence than others

ORDER OF OPERATIONS 1 +(2 * 3) Multiplication takes precedence
over addition, for example, so it must be evaluated first

CUSTOM OPERATORS // Percentage operator postfix operator % postfix func
% (value: Double) -> Double { return value / 100 } But in Swift, operators are defined in the language itself, and programs can define new operators

CUSTOM OPERATORS // Exponent operator infix operator ^ : MultiplicationPrecedence
func ^ (base: Double, exponent: Double) -> Double { return pow(base, exponent) } You can even redefine built-in operators, and alter their precedence or associativity Because of this SwiftSyntax cannot know the precedence of a given operator at parsing time It might not even know what a given operator is at all until it has parsed the declaration for it, which could be in a different file, or even another module

SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: SequenceExprSyntax │
╰─elements: ExprListSyntax │ ├─[0]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─[1]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ├─[2]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─[3]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─[4]: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile For this reason, SwiftSyntax doesn't attempt to resolve operator precedence at parsing time, it just produces this flat expression list We can't directly evaluate an expression in this form - we'll need to figure out the precedences and then extract the expression into a new data structure that we can traverse in the correct order

SOLUTION: SWIFTOPERATORS But we don't actually need to do that,
because Apple already did it for us

SWIFTOPERATORS import SwiftParser import SwiftOperators let source = "1 +
2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(sourceSyntax) SwiftSyntax has a module called SwiftOperators that is specifically designed to solve this problem

2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(sourceSyntax) We'll start by importing the SwiftOperators module

2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(sourceSyntax) The OperatorTable type keeps track of known operators and their precedences SwiftOperators can populate the OperatorTable by parsing operator declarations in your Swift source files...

2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(sourceSyntax) But OperatorTable has a convenient standardOperators default instance that comes primed with all the built-in operators from the swift stdlib, which should be sufficient for our purposes

2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(foldedSyntax) Finally, we'll call operatorTable.foldAll () on the syntax tree

SWIFTOPERATORS import SwiftSyntax import SwiftParser import SwiftOperators let source =
"1 + 2 * 3" let sourceSyntax = Parser.parse(source: source) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) debugPrint(foldedSyntax) SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile Here is the result on the right

SWIFTOPERATORS SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: SequenceExprSyntax
→ │ ╰─elements: ExprListSyntax │ ├─[0]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─[1]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ├─[2]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─[3]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─[4]: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile And if we compare it with the original tree on the left...

SWIFTOPERATORS SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: SequenceExprSyntax
→ │ ╰─elements: ExprListSyntax │ ├─[0]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─[1]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ├─[2]: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─[3]: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─[4]: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile You can see that the SequenceExpression node has been replaced by nested InfixOperator nodes

INTERPRETER OK! With that solved, we are ready to write
an interpreter

INTERPRETER class Interpreter { private var result: Double = .nan
func evaluate( _ syntax: SyntaxProtocol) -> Double { result = .nan // TODO return result } } Unlike a compiler which turns source code into executable code, an interpreter is a program that executes the source code directly

func evaluate( _ syntax: SyntaxProtocol) -> Double { result = .nan // TODO return result } } The main interface for our Interpreter is the evaluate function

func evaluate( _ syntax: SyntaxProtocol ) -> Double { result = .nan // TODO return result } } SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile This takes a source tree - such as the one the right here - and evaluates it, before returning the result

func evaluate( _ syntax: SyntaxProtocol ) -> Double { result = .nan // TODO return result } } SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile To do that, we'll need to write a function that recursively visits each node in the tree and then dispatch to other functions to perform the appropriate logic

SYNTAX VISITOR But we don't actually need to do that,

SYNTAX VISITOR open class SyntaxVisitor SyntaxVisitor is a utility class
that traverses a syntax tree, optionally calling a user- defined function for each node it encounters

SYNTAX VISITOR open class SyntaxVisitor { public func walk(_ node:
some SyntaxProtocol) } // Usage let syntaxTree = Parser.parse(source: source) let visitor = MySyntaxVisitor(viewMode: .fixedUp) visitor.walk(syntaxTree) The main entry point for SyntaxVisitor is the walk function, which accepts a syntax node``

SYNTAX VISITOR open class SyntaxVisitor { public func walk(_ node:
some SyntaxProtocol) open func visit(_ node: AccessorBlockSyntax) open func visit(_ node: AccessorDeclListSyntax) open func visit(_ node: AccessorEffectSpeci...) open func visit(_ node: AccessorParametersSyntax) open func visit(_ node: ActorDeclSyntax) ... SyntaxVisitor is an open base class, with stub implementations for every type of node it can visit To provide our own functionality, we need subclass it and override the visit methods for the syntax types we are interested in

INTERPRETER import SwiftSyntax class Interpreter { private var result: Double
= .nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan // TODO return result } } So to implement the Interpreter, we'll first import the SwiftSyntax module

INTERPRETER import SwiftSyntax class Interpreter: SyntaxVisitor { private var result:
Double = .nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan // TODO return result } } Then set the superclass to SyntaxVisitor

INTERPRETER import SwiftSyntax class Interpreter: SyntaxVisitor { private var result:
Double = .nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } } And in evaluate we'll call the inherited walk function to traverse the input

import SwiftSyntax class Interpreter: SyntaxVisitor { private var result: Double
= .nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } } To actually perform the evaluation, we need override a couple of visit functions from SyntaxVisitor...

import SwiftSyntax class Interpreter: SyntaxVisitor { private var result: Double
= .nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } } First, the visit function for IntegerLiteralExprSynt ax This just parses the integer literal value and stores it in result

import SwiftSyntax class Interpreter: SyntaxVisitor { ... override func visit(_
node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch node.operator.trimmedDescription { case "+": result = lhs + rhs case "-": result = lhs - rhs case "*": result = lhs * rhs case "/": result = lhs / rhs default: break } return .skipChildren } } Next, slightly more complex, we'll add a visit function for InfixOperatorExprSyntax This calls evaluate() recursively for the left and right operand expressions and stores the results in local variables, then switches on the operator token to apply the appropriate logic

REPL That's the basic implementation for the Interpreter done To
test it, we'll create a program called a REPL (Read Eval Print Loop)

import SwiftParser import SwiftOperators print("Type expressions to see the result.
Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Here's the code

Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } The REPL's main loop reads an expression input by the user

Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Parses and folds the source tree...

Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Then evaluates the result, and prints it back to the console

Let's see it in action

We'll start with something simple...

Can't get much more simple than that

Success!

Now an expression with multiple terms...

Good - the order of operations is correct

Division next...

What about dividing by zero?

OK, great!

Now let's try a fractional value

Hmm, that's odd

What if we make a deliberate syntax error?

OK... but I kind of expected a parser error of
some kind?

What about an unknown operator...?

Oops, looks like we crashed

private class ExpressionEvaluator: SyntaxVisitor { var result: Double = .nan
... override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch node.operator.trimmedDescription { case "+": result = lhs + rhs case "-": result = lhs - rhs case "*": result = lhs * rhs case "/": result = lhs / rhs default: break } return .skipChildren } } Type expressions to see the result. Type 'exit' to quit. > 1 + 2 = 3.0 > 2 + 3 * 4 = 14.0 > 1 / 2 = 0.5 > 1 / 0 = inf > 1 + 2.5 = nan > 3 + * 4 = nan > 3 ++ 4 Swift/ErrorType.swift:253: Fatal error: Error raised at top level: SwiftOperators.OperatorError.missingOperator( "++", referencedFrom: binaryOperator("++")) To figure out what happened, let's take another look at the Interpreter

... override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch node.operator.trimmedDescription { case "+": result = lhs + rhs case "-": result = lhs - rhs case "*": result = lhs * rhs case "/": result = lhs / rhs default: break } return .skipChildren } } Type expressions to see the result. Type 'exit' to quit. > 1 + 2 = 3.0 > 2 + 3 * 4 = 14.0 > 1 / 2 = 0.5 > 1 / 0 = inf > 1 + 2.5 = nan > 3 + * 4 = nan > 3 ++ 4 Swift/ErrorType.swift:253: Fatal error: Error raised at top level: SwiftOperators.OperatorError.missingOperator( "++", referencedFrom: binaryOperator("++")) The first bug was with fractional input values

... override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch node.operator.trimmedDescription { case "+": result = lhs + rhs case "-": result = lhs - rhs case "*": result = lhs * rhs case "/": result = lhs / rhs default: break } return .skipChildren } } Type expressions to see the result. Type 'exit' to quit. > 1 + 2 = 3.0 > 2 + 3 * 4 = 14.0 > 1 / 2 = 0.5 > 1 / 0 = inf > 1 + 2.5 = nan > 3 + * 4 = nan > 3 ++ 4 Swift/ErrorType.swift:253: Fatal error: Error raised at top level: SwiftOperators.OperatorError.missingOperator( "++", referencedFrom: binaryOperator("++")) Ah, there's the problem - we added a visit function for integer literals, but not float literals That's why it just returned the default nan value from the evaluate function

... override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) ... return .skipChildren } } Type expressions to see the result. Type 'exit' to quit. > 1 + 2 = 3.0 > 2 + 3 * 4 = 14.0 > 1 / 2 = 0.5 > 1 / 0 = inf > 1 + 2.5 = nan > 3 + * 4 = nan > 3 ++ 4 Swift/ErrorType.swift:253: Fatal error: Error raised at top level: SwiftOperators.OperatorError.missingOperator( "++", referencedFrom: binaryOperator("++")) Easily fixed - we'll add a visit function for FloatLiteralExprSynta x

... override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) ... return .skipChildren } } Type expressions to see the result. Type 'exit' to quit. > 1 + 2 = 3.0 > 2 + 3 * 4 = 14.0 > 1 / 2 = 0.5 > 1 / 0 = inf > 1 + 2.5 = nan > 3 + * 4 = nan > 3 ++ 4 Swift/ErrorType.swift:253: Fatal error: Error raised at top level: SwiftOperators.OperatorError.missingOperator( "++", referencedFrom: binaryOperator("++")) Next, the deliberate syntax error I wasn't expecting this to produce a meaningful result - but why didn't we get some sort of parsing error?

import SwiftSyntax import SwiftParser import SwiftOperators print("Type expressions to see
the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Let's examine the REPL code again...

import SwiftSyntax import SwiftParser import SwiftOperators print("Type expressions to see
the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Here's where we parse the input... But Parser.parse() doesn't throw or appear to return any errors So how does SwiftSyntax handle invalid input?

ERROR RECOVERY UnexpectedNodesSyntax // For input that couldn't be parsed
MissingSyntax // For missing nodes MissingStmtSyntax MissingDeclSyntax MissingTypeSyntax MissingExprSyntax etc... extension TokenSyntax { var presence: SourcePresence // present or missing } There are several reasons why a source file might include unexpected syntax - it could be a programmer error, but it could also just be new syntax that this version of the parser isn't aware of yet Even if it is a typo, we expect a modern compiler to report all the issues it encounters in a single pass, not just bail out after the first error

MissingSyntax // For missing nodes MissingStmtSyntax MissingDeclSyntax MissingTypeSyntax MissingExprSyntax etc... extension TokenSyntax { var presence: SourcePresence // present or missing } Instead of discarding unrecognized input, SwiftSyntax preserves and encapsulates it so it can continue to work with the rest of the code UnexpectedNodesSyntax is used when the parser finds some code that it doesn't understand

MissingSyntax // For missing nodes MissingStmtSyntax MissingDeclSyntax MissingTypeSyntax MissingExprSyntax etc... extension TokenSyntax { var presence: SourcePresence // present or missing } There is also a family of syntax node types related to missing code from the input These act as placeholders for gaps in the source tree so that the parser can continue to construct the rest of the tree without being blocked

MissingSyntax // For missing nodes MissingStmtSyntax MissingDeclSyntax MissingTypeSyntax MissingExprSyntax etc... extension TokenSyntax { var presence: SourcePresence // present or missing } For even more granularity, every token has a presence property indicating whether it was actually found in the input or whether it was inserted by the parser to fill a gap It's great that SwiftSyntax is able to recover from errors, but working with broken input can be dangerous and lead to more serious errors later on So how can we detect and report these errors instead of just ignoring them?

VIEWMODE open class SyntaxVisitor { public init(viewMode: SyntaxTreeViewMode) { ...
} } public enum SyntaxTreeViewMode: Sendable { /// Visit only the nodes present in the original source code case sourceAccurate /// Missing nodes will be visited, unexpected nodes will be skipped case fixedUp /// Both missing and unexpected nodes will be traversed case all } SyntaxVisitor can traverse these missing or unexpected nodes just like any other syntax

} } public enum SyntaxTreeViewMode: Sendable { /// Visit only the nodes present in the original source code case sourceAccurate /// Missing nodes will be visited, unexpected nodes will be skipped case fixedUp /// Both missing and unexpected nodes will be traversed case all } We glossed over it before, but the SyntaxVisitor initializer has a viewMode parameter that controls how invalid syntax should be handled

} } public enum SyntaxTreeViewMode: Sendable { /// Visit only the nodes present in the original source code case sourceAccurate /// Missing nodes will be visited, unexpected nodes will be skipped case fixedUp /// Both missing and unexpected nodes will be traversed case all } You can choose between sourceAccurate, fixedUp or all We're currently using fixedUp for the Interpreter if we switched to all, we could add visit functions for the missing or malformed syntax nodes and produce an exhaustive list of errors

PARSER DIAGNOSTICS But we don't actually need to do that,

Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } The ParserDiagnosticsGene rator class provides a simple way to get a list of errors and warnings from a syntax tree

import SwiftParser import SwiftOperators import SwiftParserDiagnostics print("Type expressions to see
the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } To use it we first need to import the SwiftParserDiagnostic s module...

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Then to get the parsing errors we run ParserDiagnosticsGene rator.diagnostics(for :) on the syntax tree

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } This returns an array of Diagnostics, which we can print out directly

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Finally, we'll bail out if there are any errors rather than continuing with bad input

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Lets see how the output looks now

Good - that's a lot more helpful than "nan" Note
that the error even includes the line number and character position!

So what about that crash when we used a custom
operator?

Is that fixed now that we're logging syntax errors and
bailing out?

Nope. From the crash message, it seems like an error
is being thrown at the top level of the app There's also a pretty strong clue here that the error relates to the SwiftOperators library

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } Let's take another look at the REPL code...

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } The operatorTable.foldAll() call is the only line that is marked with try and so it's the only line that could actually be throwing an error. But why is it throwing?

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParserDiagnosticsGenerator.diagnostics(for: sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } We initialized the OperatorTable with all the operators from the stdlib, but ++ isn't a standard operator in Swift The SwiftSyntax parser is happy to ignore undefined operators, but OperatorTable can't fold an expression if it doesn't recognize the operator, so instead of failing silently it throws an error

OPERATOR ERROR extension OperatorTable { public func foldAll( _ node:
some SyntaxProtocol, errorHandler: OperatorErrorHandler = { throw $0 } ) rethrows -> Syntax } extension OperatorError { /// Produce a diagnostic for a given operator-precedence error. public var asDiagnostic: Diagnostic } The OperatorTable.foldAll() method accepts an optional errorHandler callback The default behavior is to throw, but we can provide our own implementation to handle the error more gracefully

OPERATOR ERROR extension OperatorTable { public func foldAll( _ node:
some SyntaxProtocol, errorHandler: OperatorErrorHandler = { throw $0 } ) rethrows -> Syntax } extension OperatorError { /// Produce a diagnostic for a given operator-precedence error. public var asDiagnostic: Diagnostic } The callback receives an OperatorError, when what we actually want is a diagnostic But OperatorError has a convenient asDiagnostic getter we can use to convert

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) let diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = try operatorTable.foldAll(sourceSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } So we'll replace the throwing foldAll() call...

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } ...with one that appends operator errors to the existing diagnostics array

Lets try the REPL again

Much better! Let's try some more expressions

Oh Well that isn't right By now you can probably
guess what's happened We never added a visitor function for prefix operators, so the unary minus is just being ignored But this time it's not even returning nan to indicate something went wrong - it's just silently skipping the negation step and giving the wrong result We really need a reliable way for the Interpreter to detect when it encounters any syntax that it doesn't support yet Overriding every single visit function exposed by SyntaxVisitor just to log a diagnostic doesn't sound like much fun (there are almost 300 of them!)

SYNTAX ANY VISITOR But we don't actually need to do
that, because Apple already did it for us

SYNTAX ANY VISITOR /// A SyntaxVisitor that can visit the
nodes as generic Syntax values. /// /// This subclass of SyntaxVisitor is slower than the type-specific visitation /// of SyntaxVisitor. Use SyntaxAnyVisitor if the visitAny(_) function /// would be useful to have, otherwise inherit from SyntaxVisitor. /// open class SyntaxAnyVisitor: SyntaxVisitor { open func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { return .visitChildren } ... } SyntaxAnyVisitor is a subclass of SyntaxVisitor that adds a visitAny fallback method. visitAny is called for any syntax type that we haven't added a more specific visit function override for (at a slight cost to performance)

INTERPRETER class Interpreter: SyntaxVisitor { private var result: Double =
.nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } ... } Let's modify Interpreter to inherit from SyntaxAnyVisitor

INTERPRETER class Interpreter: SyntaxAnyVisitor { private var result: Double =
.nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } ... }

.nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } ... } We don't need to touch any of our existing visit functions...

.nan func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { result = .nan return .visitChildren } } we'll just add a new override for visitAny() Ideally we'd log an error of some kind, but for now we'll just return .nan so we can verify that it works...

Let's try that problematic expression again...

ok, that's good - it's hitting our fallback and not
just failing silently anymore

Call me suspicious though, but ...

Yep - it looks like we broke everything The problem
with using nan to signal errors is that it doesn't give us much debug info to work with Can we make use of SwiftSyntax Diagnostics here somehow? Maybe raise our own Diagnostic errors?

CUSTOM DIAGNOSTICS Let's find out!

DIAGNOSTIC public struct Diagnostic: CustomDebugStringConvertible, Sendable { public init( node:
some SyntaxProtocol, position: AbsolutePosition? = nil, message: DiagnosticMessage, highlights: [Syntax]? = nil, notes: [Note] = [], fixIts: [FixIt] = [] ) } The Diagnostic type is just a struct, defined in the SwiftDiagnostics module

some SyntaxProtocol, position: AbsolutePosition? = nil, message: DiagnosticMessage, highlights: [Syntax]? = nil, notes: [Note] = [], fixIts: [FixIt] = [] ) } The initializer has a bunch of parameters, but the only ones we need to think about are node: the syntax node that spawned the diagnostic...

some SyntaxProtocol, position: AbsolutePosition? = nil, message: DiagnosticMessage, highlights: [Syntax]? = nil, notes: [Note] = [], fixIts: [FixIt] = [] ) } And message: which is a type conforming to the DiagnosticMessage protocol

DIAGNOSTIC public protocol DiagnosticMessage: Sendable { /// The diagnostic message
that should be displayed in the client. var message: String { get } var diagnosticID: MessageID { get } var severity: DiagnosticSeverity { get } } The DiagnosticMessage protocol is the extension point for creating our own diagnostic messages SwiftSyntax doesn't provide a concrete DiagnosticMessage type for general use, so we'll need to define our own

import SwiftDiagnostics class Interpreter: SyntaxAnyVisitor { private var result: Double
= .nan ... override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { result = .nan return .visitChildren } } We'll start by importing the SwiftDiagnostics module (not to be confused with the SwiftParserDiagnostics module we used earlier)

import SwiftDiagnostics struct Message: DiagnosticMessage { var message: String var
severity: DiagnosticSeverity = .error var diagnosticID: MessageID { .init(domain: "Interpreter", id: message) } } class Interpreter: SyntaxAnyVisitor { private var result: Double = .nan ... override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { result = .nan return .visitChildren } } Next we'll define a Message struct that conforms to DiagnosticMessage Note: This doesn't have to be a struct - it could be an enum instead if we wanted to define a fixed set of errors

severity: DiagnosticSeverity = .error var diagnosticID: MessageID { .init(domain: "Interpreter", id: message) } } extension SyntaxProtocol { func error(_ message: String) -> Diagnostic { .init(node: self, message: Message(message: message)) } } class Interpreter: SyntaxAnyVisitor { private var result: Double = .nan ... override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { result = .nan return .visitChildren } } To save some typing, we'll also add a convenience method to SyntaxProtol...

severity: DiagnosticSeverity = .error var diagnosticID: MessageID { .init(domain: "Interpreter", id: message) } } extension SyntaxProtocol { func error(_ message: String) -> Diagnostic { .init(node: self, message: Message(message: message)) } } class Interpreter: SyntaxAnyVisitor { private var result: Double = .nan var diagnostics: [Diagnostic] = [] ... override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { result = .nan return .visitChildren } } Finally, we'll add a diagnostics property to Interpreter...

severity: DiagnosticSeverity = .error var diagnosticID: MessageID { .init(domain: "Interpreter", id: message) } } extension SyntaxProtocol { func error(_ message: String) -> Diagnostic { .init(node: self, message: Message(message: message)) } } class Interpreter: SyntaxAnyVisitor { private var result: Double = .nan var diagnostics: [Diagnostic] = [] ... override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax: '\(node.kind)'")) return .visitChildren } } And replace the nan result in visitAny() with a diagnostic

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } let result = interpreter.evaluate(foldedSyntax) print("= \(result)") } We'll need to update the REPL logic again...

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } print("= \(result)") } First we'll move the evaluation up to before we print the diagnostics...

the result. Type 'exit' to quit.") let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += result.diagnostics result.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } print("= \(result)") } then append the interpreter diagnostics to the array, and clear them so they don't accumulate between iterations

Let's see if that helps us get a better sense
of what's going wrong with the evaluation

OK, that's a lot of info - and rather more
errors that I was expecting? But we can see in the middle there - we have the unhandled prefixOperatorExpr

Let's try the simple expression again

OK, so that's why it wasn't working... But what are
all these unhandled syntax nodes?

SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │
├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile If you recall the syntax tree from earlier...

SourceFileSyntax ├─statements: CodeBlockItemListSyntax │ ╰─[0]: CodeBlockItemSyntax │ ╰─item: InfixOperatorExprSyntax │
├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("1") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("+") │ ╰─rightOperand: InfixOperatorExprSyntax │ ├─leftOperand: IntegerLiteralExprSyntax │ │ ╰─literal: integerLiteral("2") │ ├─operator: BinaryOperatorExprSyntax │ │ ╰─operator: binaryOperator("*") │ ╰─rightOperand: IntegerLiteralExprSyntax │ ╰─literal: integerLiteral("3") ╰─endOfFileToken: endOfFile The SwiftSyntax Parser always returns a SourceFileSyntax node Even a simple single-line expression is packaged as a statement inside a codeblock inside a SourceFileSyntax node, terminated by an endOfFileToken

class Interpreter: SyntaxAnyVisitor { ... override func visit(_ node: IntegerLiteralExprSyntax)
-> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax: '\(node.kind)'")) return .visitChildren } } Previously we were implicitly ignoring these container nodes by just not overriding the visit functions for them, and it was fine. but now, any nodes we don't have overrides for will raise a diagnostic in visitAll()

class Interpreter: SyntaxAnyVisitor { ... override func visit(_ node: IntegerLiteralExprSyntax)
-> SyntaxVisitorContinueKind { ... } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { ... } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax: '\(node.kind)'")) return .visitChildren } // MARK: Ignored override open func visit(_ token: SourceFileSyntax) -> SyntaxVisitorContinueKind { return .visitChildren } override open func visit(_ token: CodeBlockItemListSyntax) -> SyntaxVisitorContinueKind { return .visitChildren } override open func visit(_ token: CodeBlockItemSyntax) -> SyntaxVisitorContinueKind { return .visitChildren } override open func visit(_ token: TokenSyntax) -> SyntaxVisitorContinueKind { return .visitChildren } } So we will have to explicitly ignore them, by adding no-op visit functions

Now, with any luck...

Yes! Only the problematic syntax is being flagged

And simple expressions are back to working!

But what about something like this?

Or this?

It seems we are still a long way from a
fully-fledged swift interpreter

TYPES & VARIABLES It's time to upgrade our simple expression
evaluator

enum Variant: Equatable { case undefined case number(Double) case boolean(Bool)
case string(String) } extension Variant: CustomStringConvertible { var description: String { switch self { case .undefined: "undefined" case let .number(value): "\(value)" case let .boolean(value): "\(value)" case let .string(value): value } } } class Interpreter: SyntaxAnyVisitor { private var result: Double = .nan var diagnostics: [Diagnostic] = [] func evaluate(_ syntax: SyntaxProtocol) -> Double { result = .nan walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = Double(node.literal.text) ?? .nan return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch node.operator.trimmedDescription { case "+": result = lhs + rhs case "-": result = lhs - rhs case "*": result = lhs * rhs case "/": result = lhs / rhs default: break } return .skipChildren } ... } To support strings and boolean values we'll introduce an enum type called Variant

case string(String) } extension Variant: CustomStringConvertible { var description: String { switch self { case .undefined: "undefined" case let .number(value): "\(value)" case let .boolean(value): "\(value)" case let .string(value): value } } } class Interpreter: SyntaxAnyVisitor { private var result: Variant = .undefined var diagnostics: [Diagnostic] = [] func evaluate(_ syntax: SyntaxProtocol) -> Variant { result = .undefined walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch (lhs, rhs, node.operator.trimmedDescription) { case let (.number(l), .number(r), "+"): result = .number(l + r) case let (.number(l), .number(r), "-"): result = .number(l - r) case let (.number(l), .number(r), "*"): result = .number(l * r) case let (.number(l), .number(r), "/"): result = .number(l / r) default: diagnostics.append(node.error("invalid expression: \(node.trimmed)")) } return .skipChildren } ... } This replaces the Double values in the Interpreter

case string(String) } extension Variant: CustomStringConvertible { var description: String { switch self { case .undefined: "undefined" case let .number(value): "\(value)" case let .boolean(value): "\(value)" case let .string(value): value } } } class Interpreter: SyntaxAnyVisitor { private var result: Variant = .undefined private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] func evaluate(_ syntax: SyntaxProtocol) -> Variant { result = .undefined walk(syntax) return result } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: InfixOperatorExprSyntax) -> SyntaxVisitorContinueKind { let lhs = evaluate(node.leftOperand) let rhs = evaluate(node.rightOperand) switch (lhs, rhs, node.operator.trimmedDescription) { case let (.number(l), .number(r), "+"): result = .number(l + r) case let (.number(l), .number(r), "-"): result = .number(l - r) case let (.number(l), .number(r), "*"): result = .number(l * r) case let (.number(l), .number(r), "/"): result = .number(l / r) default: diagnostics.append(node.error("invalid expression: \(node.trimmed)")) } return .skipChildren } ... } We'll add a dictionary property to store named variables...

case string(String) } extension Variant: CustomStringConvertible { var description: String { switch self { case .undefined: "undefined" case let .number(value): "\(value)" case let .boolean(value): "\(value)" case let .string(value): value } } } class Interpreter: SyntaxAnyVisitor { private var result: Variant = .undefined private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] func evaluate(_ syntax: SyntaxProtocol) -> Variant { ... } override func visit(_ node: DeclReferenceExprSyntax) -> SyntaxVisitorContinueKind { if let value = variables[node.baseName.text] { result = value } else { result = .undefined diagnostics.append(node.error("undefined variable '\(node.trimmed)'")) } return .skipChildren } override func visit(_ node: IntegerLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: FloatLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .number(Double(node.literal.text) ?? .nan) return .skipChildren } override func visit(_ node: BooleanLiteralExprSyntax) -> SyntaxVisitorContinueKind { result = .boolean(node.literal.tokenKind == .keyword(.true)) return .skipChildren } ... } Then a visit function for DeclReferenceExprSynta x - that' a reference to a previously declared identifier, such as a function, type or variable

case string(String) } extension Variant: CustomStringConvertible { var description: String { switch self { case .undefined: "undefined" case let .number(value): "\(value)" case let .boolean(value): "\(value)" case let .string(value): value } } } class Interpreter: SyntaxAnyVisitor { private var result: Variant = .undefined private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] func evaluate(_ syntax: SyntaxProtocol) -> Variant { ... } override func visit(_ node: DeclReferenceExprSyntax) -> SyntaxVisitorContinueKind { if let value = variables[node.baseName.text] { result = value } else { result = .undefined diagnostics.append(node.error("undefined variable '\(node.trimmed)'")) } return .skipChildren } override func visit(_ node: VariableDeclSyntax) -> SyntaxVisitorContinueKind { for binding in node.bindings { switch binding.pattern.as(PatternSyntaxEnum.self) { case let .identifierPattern(identifier): let value = binding.initializer.map { evaluate($0.value) } variables[identifier.identifier.text] = value ?? .undefined default: diagnostics.append( node.error("unsupported syntax '\(binding.pattern.trimmed)'") ) } } return .skipChildren } ... } And for VariableDeclSyntax, which is the syntax for declaring variables

SKIPPING AHEAD... OK, we're making progress, but what about functions,
loops, conditional statements...? All these things can be implemented using the techniques we've already covered, but since we're short on time I'm going to skip ahead. If you're interested in the implementation details, you can check out the examples links at the end

OK, so let's take stock...

We' try some variables...

Now an expression...

OK, that all seems right

How about a deliberate type error...

And what about an unknown variable?

Perfect! We've seen a lot of trivial test examples -
now let's try some real code

This is a fibonacci function - it returns the nth
element in the fibonacci sequence It uses recursion, conditional statements, comparison operators...

(Sorry it's all on one line, but the REPL doesn't
support multiline statements just yet)

Let's try it with a couple of values

Nice But something's bothering me You see those "Int" parameter
and return types in the fibonacci function declaration? We've specified them, but we don't check them at all

in fact we can define a function like this

with made up types

and it makes no difference at all our interpreter doesn't
have any concept of types - only values but it seems wrong to allow type annotations if we're just going to ignore them, so what if we just.. rejected them instead? we already reject other valid swift syntax that we don't support yet, let's do the same for types

class Interpreter: SyntaxAnyVisitor { private var result: Variant = .undefined
private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] ... override func visit(_ node: VariableDeclSyntax) -> SyntaxVisitorContinueKind { for binding in node.bindings { switch binding.pattern.as(PatternSyntaxEnum.self) { case let .identifierPattern(identifier): let value = binding.initializer.map { evaluate($0.value) } variables[identifier.identifier.text] = value ?? .undefined default: diagnostics.append(node.error("unsupported syntax '\(binding.pattern.kind)'")) } } return .skipChildren } override func visit(_ node: FunctionDeclSyntax) -> SyntaxVisitorContinueKind { variables[node.name.text] = .function(node) return .skipChildren } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.kind)'")) return .visitChildren } ... } You might think we could just add a visitor function for type annotations to the interpreter, but that won't work

private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] ... override func visit(_ node: VariableDeclSyntax) -> SyntaxVisitorContinueKind { for binding in node.bindings { switch binding.pattern.as(PatternSyntaxEnum.self) { case let .identifierPattern(identifier): let value = binding.initializer.map { evaluate($0.value) } variables[identifier.identifier.text] = value ?? .undefined default: diagnostics.append(node.error("unsupported syntax '\(binding.pattern.kind)'")) } } return .skipChildren } override func visit(_ node: FunctionDeclSyntax) -> SyntaxVisitorContinueKind { variables[node.name.text] = .function(node) return .skipChildren } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.kind)'")) return .visitChildren } ... } if it did, the visitAny function would already be catching them

private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] ... override func visit(_ node: VariableDeclSyntax) -> SyntaxVisitorContinueKind { for binding in node.bindings { switch binding.pattern.as(PatternSyntaxEnum.self) { case let .identifierPattern(identifier): let value = binding.initializer.map { evaluate($0.value) } variables[identifier.identifier.text] = value ?? .undefined default: diagnostics.append(node.error("unsupported syntax '\(binding.pattern.kind)'")) } } return .skipChildren } override func visit(_ node: FunctionDeclSyntax) -> SyntaxVisitorContinueKind { variables[node.name.text] = .function(node) return .skipChildren } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.kind)'")) return .visitChildren } ... } The reason it doesn't work is that the type annotations are child nodes of their respective variable or function declarations...

private var variables: [String: Variant] = [:] var diagnostics: [Diagnostic] = [] ... override func visit(_ node: VariableDeclSyntax) -> SyntaxVisitorContinueKind { for binding in node.bindings { switch binding.pattern.as(PatternSyntaxEnum.self) { case let .identifierPattern(identifier): let value = binding.initializer.map { evaluate($0.value) } variables[identifier.identifier.text] = value ?? .undefined default: diagnostics.append(node.error("unsupported syntax '\(binding.pattern.kind)'")) } } return .skipChildren } override func visit(_ node: FunctionDeclSyntax) -> SyntaxVisitorContinueKind { variables[node.name.text] = .function(node) return .skipChildren } override func visitAny(_ node: Syntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.kind)'")) return .visitChildren } ... } ... but the visit functions for those nodes return .skipChildren because they already contain all the logic needed to process those nodes, so the visitor never traverses those type annotations

PREPROCESSOR What we'll do instead is create a separate syntax
visitor that operates as a pre-processing step prior to running the interpreter

class Preprocessor: SyntaxVisitor { private var diagnostics: [Diagnostic] = []
static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } The Preprocessor follows the same pattern as ParserDiagnosticsGenerator...

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } It has a static diagnostics(for:) function that creates an instance and invokes the walk method to generate the diagnostics

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } And we've added visitor functions for variable type annotations...

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } Function return types...

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } And, function parameters The parameters case is a little more complex than the others because the type signature for a function parameter is not a syntax node in it's own right; the type is part of the parameter node, and we still need to preserve the other parameter attributes such as its name

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } so instead of rejecting the entire parameter node, we inspect its properties to see if it has an explicit type. The easiest way is just to check for the presence of a colon

print("Type expressions to see the result. Type 'exit' to quit.")
let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } Next We'll need to update the REPL to call the Preprocessor and capture its diagnostics

let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } We call the preprocessor before the ParserDiagnosticsGenerator, for reasons I'll explain later

let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = Preprocessor.diagnostics(for: sourceSyntax) diagnostics += ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } So let's test it...

If we try to use explicit an explicit type now,
we should get an error...

OK, good

Now let's try with a function...

Great! it's working

Let's try a variable declaration without a type

And finally let's try our fibonnacci function again, but this
time without parameter or return types

Er... what? OK, I see what's happening. It's not the
preprocessor raising this error, it's SwiftParserDiagnostics The real swift parser doesn't allow function parameters without types, so ParserDiagnosticsGenerator raises an error But the preprocessor won't accept functions with types So how can we satisfy them both? Well, what if we modify the preprocessor to actually insert the missing types before we run the regular Swift Parser diagnostics generator?

SYNTAX REWRITER SyntaxVisitor isn't able to modify the syntax tree
while traversing it For that we will need to use another SwiftSyntax utility class called SyntaxRewriter

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } So let's change the base class of our Preprocessor from SyntaxVisitor to SyntaxRewriter

class Preprocessor: SyntaxRewriter { private var diagnostics: [Diagnostic] = []
static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } }

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: ReturnClauseSyntax) -> SyntaxVisitorContinueKind { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return .skipChildren } override func visit(_ node: FunctionParameterSyntax) -> SyntaxVisitorContinueKind { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return .skipChildren } } Next we need to update the visit functions SyntaxRewriter follows a similar pattern to SyntaxVisitor but unlike SyntaxAnyVisitor, it's not a subclass of it and the visit method signatures are slightly different

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node } } Instead of SyntaxVisitorContinueKind, each visit function returns a copy of its input node

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node } } In most cases we don't actually need to modify the source at all, because variable and function return types are optional anyway in regular swift But function parameters must have types to be valid syntax, so we'll need to update the visitor function for FunctionParameterSyntax to "repair" the typeless parameters that we are now mandating

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node .with(\.colon.presence, .present) } } First we'll mark the colon as present, so that ParserDiagnosticsGene rator won't complain that it was missing

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node .with(\.colon.presence, .present) .with(\.type, TypeSyntax( IdentifierTypeSyntax(name: .identifier("Any"), trailingTrivia: .space) )) } } Then we need to insert a parameter type the type itself doesn't matter - we just need to put something there to make the syntax valid I've used "Any" but we could have put "Foobar"

static func diagnostics(for syntax: SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor(viewMode: .sourceAccurate) processor.walk(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node .with(\.colon.presence, .present) .with(\.type, TypeSyntax( IdentifierTypeSyntax(name: .identifier("Any"), trailingTrivia: .space) )) } } Finally, we'll replace the diagnostics(for:) entry point with a new "process" method

static func process(_ syntax: inout SourceFileSyntax) -> [Diagnostic] { let processor = Preprocessor() syntax = processor.visit(syntax) return processor.diagnostics } override func visit(_ node: TypeAnnotationSyntax) -> TypeAnnotationSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: ReturnClauseSyntax) -> ReturnClauseSyntax { diagnostics.append(node.error("unsupported syntax '\(node.trimmed)'")) return node } override func visit(_ node: FunctionParameterSyntax) -> FunctionParameterSyntax { if node.colon.presence == .present { diagnostics.append(node.error("unsupported syntax ': \(node.type.trimmed)'")) } return node .with(\.colon.presence, .present) .with(\.type, TypeSyntax( IdentifierTypeSyntax(name: .identifier("Any"), trailingTrivia: .space) )) } } This new method accepts the SourceFileSyntax as an inout parameter so that it can modify it as well as returning diagnostics Note that SyntaxRewriter doesn't have a walk method. We just call vistit directly

let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = Preprocessor.diagnostics(for: sourceSyntax) diagnostics += ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } Because we've changed the API for the Preprocessor, we'll also need to make a small adjustment to the REPL code

let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } let sourceSyntax = Parser.parse(source: input) var diagnostics = Preprocessor.diagnostics(for: sourceSyntax) diagnostics += ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } The sourceSyntax now needs to be mutable...

let interpreter = Interpreter(viewMode: .fixedUp) while true { print("> ", terminator: "") guard let input = readLine(), input.lowercased() != "exit" else { break } var sourceSyntax = Parser.parse(source: input) var diagnostics = Preprocessor.process(&sourceSyntax) diagnostics += ParseDiagnosticsGenerator.diagnostics(for: sourceSyntax) let operatorTable = OperatorTable.standardOperators let foldedSyntax = operatorTable.foldAll(sourceSyntax) { error in diagnostics.append(error.asDiagnostic) } let result = interpreter.evaluate(foldedSyntax) diagnostics += interpreter.diagnostics interpreter.diagnostics.removeAll() for diagnostic in diagnostics { print(diagnostic) } if diagnostics.contains(where: { $0.diagMessage.severity == .error }) { continue } if result != .undefined { print("= \(result)") } } and passed by reference to the process function

All done! We should now be free to write swift
unencumbered by types...

Let's try the fibonacci function again...

Success! No warnings

Let's try another function

That also works

Oops, I got a bit excited and mistyped the name
At least the error checking works though! but wait a second... "Undefined is not a function..." where have I heard that before

Something has gone very wrong

SWIFT EVAL I set out to make an eval function
for Swift

SWIFT EVAL But this isn't Swift

STRONG EVAL Swift is strongly typed

STATIC EVAL Swift has static safety checks

FAST EVAL Swift is fast

SWIFT SYNTAX It might be Swift Syntax

But I think I just reinvented JavaScript?!

EVAL EVIL Maybe "Evil" wasn't a typo after all!

SWIFTSYNTAX FOR GOOD AND EVIL We're over time so I'll
have to leave it there I can only apologise for this terrible crime against Swift...

THANK YOU HTTPS://GITHUB.COM/NICKLOCKWOOD/SWIFTSYNTAXTALK If you're interested in learning more you
can find the code for everything I've shown today here on my github page Thank you!

SwiftSyntax: for Good and Evil (inc. Presenter ...

SwiftSyntax: for Good and Evil (inc. Presenter Notes)

More Decks by Nick Lockwood

Other Decks in Technology

Featured

Transcript