SwiftSyntax Direct Parser (as of swift-DEVELOPMENT-SNAPSHOT-2019-04-10-a) 2019/04/19 Waiwai Swiftc Yasuhiro Inami / @inamiy

Swift 4.2-5.0: SwiftcRunner

// Swift 4.2-5.0 (swiftc runner) public enum SyntaxTreeParser { public static func parse(_ url: URL, swiftcURL: URL? = nil) throws -> SourceFileSyntax { let swiftcRunner = try SwiftcRunner(sourceFile: url, swiftcURL: swiftcURL) let result = try swiftcRunner.invoke() let deserializer = SyntaxTreeDeserializer() return try deserializer.deserialize(result.stdoutData, serializationFormat: .json) } } struct SwiftcRunner { func invoke() throws -> ProcessResult { var arguments = ["-frontend", "-emit-syntax"] arguments.append(...) return run(swiftcURL, arguments: arguments) } }

xcrun swiftc -frontend -emit-syntax Foo.swift

C++ AST ↓ (JSON) ↓ Swift AST

No content

Swift 5.0-dev: SwiftLang (SourceKit (XPC)+ ByteTree)

// Swift 5.0-dev (SourceKit + ByteTree, not in Xcode 10.2 toolchain) import SwiftLang extension SwiftLang { fileprivate static func parse( _ content: SourceFile, into format: SyntaxTreeFormat ) throws -> Tree { let Service = SourceKitdService() let Request = SourceKitdRequest(uid: .request_EditorOpen) Request.addParameter(.key_SyntaxTreeSerializationFormat, value: format.kind) let Resp = Service.sendSyn(request: Request) let CloseReq = SourceKitdRequest(uid: .request_EditorClose) let CloseResp = Service.sendSyn(request: CloseReq) return try format.makeTree(Resp.value) } }

No content

apple/swift/docs/ByteTree.md [libSyntax] Add a binary serialization format for syntax trees by ahoppen · Pull Request #18497 · apple/swift

No content

Swift 5.1: libSwiftSyntaxParser (Direct & incremental parser)

// Swift 5.1 (direct parser, not in Xcode 10.2 toolchain) import _InternalSwiftSyntaxParser // renamed from SwiftSyntaxParser public enum SyntaxParser { private static func parseRaw( // called from `SyntaxParser.parse` _ source: String, _ parseTransition: IncrementalParseTransition?, _ filenameForDiagnostics: String, _ diagnosticEngine: DiagnosticEngine? ) -> RawSyntax { let c_parser = swiftparse_parser_create() ... // See next slide for more details let c_top = swiftparse_parse_string(c_parser, source) return RawSyntax.moveFromOpaque(c_top)! } }

let c_parser = swiftparse_parser_create() defer { swiftparse_parser_dispose(c_parser) } // Transfer `RawSyntax` ownership to C parser. swiftparse_parser_set_node_handler(c_parser, nodeHandler); // For incremental parsing. swiftparse_parser_set_node_lookup(c_parser, nodeLookup); defer { diagnosticEngine.diagnose(Diagnostic(pendingDiag, pendingNotes)) } swiftparse_parser_set_diagnostic_handler(c_parser, diagHandler) let c_top = swiftparse_parse_string(c_parser, source) // run parser return RawSyntax.moveFromOpaque(c_top)! // get ownership back

How SwiftSyntax access to lib/Parse 1. SyntaxParser.parse() // SwiftSyntax 2. swiftparse_parse_string()                     // _InternalSwiftSyntaxParser1 3. SynParser::parse() // libSwiftSyntaxParser 4. ParserUnit::parse() // lib/Parse 5. Parser::parseTopLevel() // lib/Parse 1 Renamed from libSwiftSyntaxParser in tools/libSwiftSyntaxParser/CMakeLists.txt

How SwiftSyntax interacts raw data with lib/Parse

[Parse/Syntax] Refactoring to decouple the parser from syntax tree creation by akyrtzi · Pull Request #21368 · apple/swift Parser (C++ class) -> SyntaxParsingContext -> RootContextData -> ParsedRawSyntaxRecorder -> std::shared_ptr SyntaxParseActions defines interface between the parser and a receiver of raw syntax nodes.

/// Interface between the parser and a receiver of raw syntax nodes. class SyntaxParseActions { virtual OpaqueSyntaxNode recordToken(tok tokenKind, ...) = 0; /// Record a missing token. \c loc can be invalid or an approximate location /// of where the token would be if not missing. virtual OpaqueSyntaxNode recordMissingToken(tok tokenKind, SourceLoc loc) = 0; /// The provided \c elements are an exact layout appropriate for the syntax /// \c kind. Missing optional elements are represented with a null /// OpaqueSyntaxNode object. virtual OpaqueSyntaxNode recordRawSyntax(syntax::SyntaxKind kind, ...) = 0; /// Used for incremental re-parsing. virtual std::pair lookupNode(size_t lexerOffset, syntax::SyntaxKind kind) { return std::make_pair(0, nullptr); } };

CLibParseActions // libSwiftSyntaxParser.cpp class CLibParseActions : public SyntaxParseActions { // Uses registered node_handler // set via `swiftparse_parser_set_node_handler` ... } CLibParseActions is a concrete subclass of SyntaxParseActions to bridge C++ 㲗 C 㲗 Swift.

Speeding up SwiftSyntax by using the parser directly - Swift Forums apple/swift#21762 apple/swift-syntax#59

My Experiments • Add `RunSwift5Command` (SwiftLang + ByteTree, experiment) by inamiy · Pull Request #13 · inamiy/ SwiftRewriter • Speeds up 4.5x faster than swiftc + JSON • Update toolchain to `PR-21772-166` & use direct-parser by inamiy · Pull Request #14 · inamiy/SwiftRewriter • Speeds up 3x-6x faster than above SourceKit + ByteTree

Thanks! Yasuhiro Inami @inamiy