SwiftSyntax Direct Parser (Swift 5.1)

Transcript

1. SwiftSyntax Direct Parser (as of swift-DEVELOPMENT-SNAPSHOT-2019-04-10-a) 2019/04/19 Waiwai Swiftc Yasuhiro

   Inami / @inamiy

2. Swift 4.2-5.0: SwiftcRunner

3. // Swift 4.2-5.0 (swiftc runner) public enum SyntaxTreeParser { public

   static func parse(_ url: URL, swiftcURL: URL? = nil) throws -> SourceFileSyntax { let swiftcRunner = try SwiftcRunner(sourceFile: url, swiftcURL: swiftcURL) let result = try swiftcRunner.invoke() let deserializer = SyntaxTreeDeserializer() return try deserializer.deserialize(result.stdoutData, serializationFormat: .json) } } struct SwiftcRunner { func invoke() throws -> ProcessResult { var arguments = ["-frontend", "-emit-syntax"] arguments.append(...) return run(swiftcURL, arguments: arguments) } }

4. xcrun swiftc -frontend -emit-syntax Foo.swift

5. C++ AST ↓ (JSON) ↓ Swift AST

6. None

7. Swift 5.0-dev: SwiftLang (SourceKit (XPC)+ ByteTree)

8. // Swift 5.0-dev (SourceKit + ByteTree, not in Xcode 10.2

   toolchain) import SwiftLang extension SwiftLang { fileprivate static func parse<Tree>( _ content: SourceFile, into format: SyntaxTreeFormat<Tree> ) throws -> Tree { let Service = SourceKitdService() let Request = SourceKitdRequest(uid: .request_EditorOpen) Request.addParameter(.key_SyntaxTreeSerializationFormat, value: format.kind) let Resp = Service.sendSyn(request: Request) let CloseReq = SourceKitdRequest(uid: .request_EditorClose) let CloseResp = Service.sendSyn(request: CloseReq) return try format.makeTree(Resp.value) } }
9. None

10. apple/swift/docs/ByteTree.md [libSyntax] Add a binary serialization format for syntax trees

    by ahoppen · Pull Request #18497 · apple/swift
11. None

12. Swift 5.1: libSwiftSyntaxParser (Direct & incremental parser)

13. // Swift 5.1 (direct parser, not in Xcode 10.2 toolchain)

    import _InternalSwiftSyntaxParser // renamed from SwiftSyntaxParser public enum SyntaxParser { private static func parseRaw( // called from `SyntaxParser.parse` _ source: String, _ parseTransition: IncrementalParseTransition?, _ filenameForDiagnostics: String, _ diagnosticEngine: DiagnosticEngine? ) -> RawSyntax { let c_parser = swiftparse_parser_create() ... // See next slide for more details let c_top = swiftparse_parse_string(c_parser, source) return RawSyntax.moveFromOpaque(c_top)! } }

14. let c_parser = swiftparse_parser_create() defer { swiftparse_parser_dispose(c_parser) } // Transfer

    `RawSyntax` ownership to C parser. swiftparse_parser_set_node_handler(c_parser, nodeHandler); // For incremental parsing. swiftparse_parser_set_node_lookup(c_parser, nodeLookup); defer { diagnosticEngine.diagnose(Diagnostic(pendingDiag, pendingNotes)) } swiftparse_parser_set_diagnostic_handler(c_parser, diagHandler) let c_top = swiftparse_parse_string(c_parser, source) // run parser return RawSyntax.moveFromOpaque(c_top)! // get ownership back

15. How SwiftSyntax access to lib/Parse 1. SyntaxParser.parse() // SwiftSyntax 2.

    swiftparse_parse_string()                     // _InternalSwiftSyntaxParser1 3. SynParser::parse() // libSwiftSyntaxParser 4. ParserUnit::parse() // lib/Parse 5. Parser::parseTopLevel() // lib/Parse 1 Renamed from libSwiftSyntaxParser in tools/libSwiftSyntaxParser/CMakeLists.txt

16. How SwiftSyntax interacts raw data with lib/Parse

17. [Parse/Syntax] Refactoring to decouple the parser from syntax tree creation

    by akyrtzi · Pull Request #21368 · apple/swift Parser (C++ class) -> SyntaxParsingContext -> RootContextData -> ParsedRawSyntaxRecorder -> std::shared_ptr<SyntaxParseActions> SyntaxParseActions defines interface between the parser and a receiver of raw syntax nodes.

18. /// Interface between the parser and a receiver of raw

    syntax nodes. class SyntaxParseActions { virtual OpaqueSyntaxNode recordToken(tok tokenKind, ...) = 0; /// Record a missing token. \c loc can be invalid or an approximate location /// of where the token would be if not missing. virtual OpaqueSyntaxNode recordMissingToken(tok tokenKind, SourceLoc loc) = 0; /// The provided \c elements are an exact layout appropriate for the syntax /// \c kind. Missing optional elements are represented with a null /// OpaqueSyntaxNode object. virtual OpaqueSyntaxNode recordRawSyntax(syntax::SyntaxKind kind, ...) = 0; /// Used for incremental re-parsing. virtual std::pair<size_t, OpaqueSyntaxNode> lookupNode(size_t lexerOffset, syntax::SyntaxKind kind) { return std::make_pair(0, nullptr); } };

19. CLibParseActions // libSwiftSyntaxParser.cpp class CLibParseActions : public SyntaxParseActions { //

    Uses registered node_handler // set via `swiftparse_parser_set_node_handler` ... } CLibParseActions is a concrete subclass of SyntaxParseActions to bridge C++ 㲗 C 㲗 Swift.

20. Speeding up SwiftSyntax by using the parser directly - Swift

    Forums apple/swift#21762 apple/swift-syntax#59

21. My Experiments • Add `RunSwift5Command` (SwiftLang + ByteTree, experiment) by

    inamiy · Pull Request #13 · inamiy/ SwiftRewriter • Speeds up 4.5x faster than swiftc + JSON • Update toolchain to `PR-21772-166` & use direct-parser by inamiy · Pull Request #14 · inamiy/SwiftRewriter • Speeds up 3x-6x faster than above SourceKit + ByteTree

22. Thanks! Yasuhiro Inami @inamiy