Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SwiftSyntax Direct Parser (Swift 5.1)

SwiftSyntax Direct Parser (Swift 5.1)

Waiwai Swiftc #10 (Apr 19, 2019)
https://iosdiscord.connpass.com/event/123573/

Yasuhiro Inami

April 19, 2019
Tweet

More Decks by Yasuhiro Inami

Other Decks in Programming

Transcript

  1. SwiftSyntax
    Direct Parser
    (as of swift-DEVELOPMENT-SNAPSHOT-2019-04-10-a)
    2019/04/19 Waiwai Swiftc
    Yasuhiro Inami / @inamiy

    View Slide

  2. Swift 4.2-5.0:
    SwiftcRunner

    View Slide

  3. // Swift 4.2-5.0 (swiftc runner)
    public enum SyntaxTreeParser {
    public static func parse(_ url: URL, swiftcURL: URL? = nil) throws
    -> SourceFileSyntax {
    let swiftcRunner = try SwiftcRunner(sourceFile: url, swiftcURL: swiftcURL)
    let result = try swiftcRunner.invoke()
    let deserializer = SyntaxTreeDeserializer()
    return try deserializer.deserialize(result.stdoutData,
    serializationFormat: .json)
    }
    }
    struct SwiftcRunner {
    func invoke() throws -> ProcessResult {
    var arguments = ["-frontend", "-emit-syntax"]
    arguments.append(...)
    return run(swiftcURL, arguments: arguments)
    }
    }

    View Slide

  4. xcrun swiftc
    -frontend
    -emit-syntax
    Foo.swift

    View Slide

  5. C++ AST

    (JSON)

    Swift AST

    View Slide

  6. View Slide

  7. Swift 5.0-dev:
    SwiftLang
    (SourceKit (XPC)+ ByteTree)

    View Slide

  8. // Swift 5.0-dev (SourceKit + ByteTree, not in Xcode 10.2 toolchain)
    import SwiftLang
    extension SwiftLang {
    fileprivate static func parse(
    _ content: SourceFile, into format: SyntaxTreeFormat
    ) throws -> Tree {
    let Service = SourceKitdService()
    let Request = SourceKitdRequest(uid: .request_EditorOpen)
    Request.addParameter(.key_SyntaxTreeSerializationFormat, value: format.kind)
    let Resp = Service.sendSyn(request: Request)
    let CloseReq = SourceKitdRequest(uid: .request_EditorClose)
    let CloseResp = Service.sendSyn(request: CloseReq)
    return try format.makeTree(Resp.value)
    }
    }

    View Slide

  9. View Slide

  10. apple/swift/docs/ByteTree.md
    [libSyntax] Add a binary
    serialization format for syntax
    trees by ahoppen · Pull Request
    #18497 · apple/swift

    View Slide

  11. View Slide

  12. Swift 5.1:
    libSwiftSyntaxParser
    (Direct & incremental parser)

    View Slide

  13. // Swift 5.1 (direct parser, not in Xcode 10.2 toolchain)
    import _InternalSwiftSyntaxParser // renamed from SwiftSyntaxParser
    public enum SyntaxParser {
    private static func parseRaw( // called from `SyntaxParser.parse`
    _ source: String,
    _ parseTransition: IncrementalParseTransition?,
    _ filenameForDiagnostics: String,
    _ diagnosticEngine: DiagnosticEngine?
    ) -> RawSyntax {
    let c_parser = swiftparse_parser_create()
    ... // See next slide for more details
    let c_top = swiftparse_parse_string(c_parser, source)
    return RawSyntax.moveFromOpaque(c_top)!
    }
    }

    View Slide

  14. let c_parser = swiftparse_parser_create()
    defer {
    swiftparse_parser_dispose(c_parser)
    }
    // Transfer `RawSyntax` ownership to C parser.
    swiftparse_parser_set_node_handler(c_parser, nodeHandler);
    // For incremental parsing.
    swiftparse_parser_set_node_lookup(c_parser, nodeLookup);
    defer {
    diagnosticEngine.diagnose(Diagnostic(pendingDiag, pendingNotes))
    }
    swiftparse_parser_set_diagnostic_handler(c_parser, diagHandler)
    let c_top = swiftparse_parse_string(c_parser, source) // run parser
    return RawSyntax.moveFromOpaque(c_top)! // get ownership back

    View Slide

  15. How SwiftSyntax access to lib/Parse
    1. SyntaxParser.parse() // SwiftSyntax
    2. swiftparse_parse_string()
                        // _InternalSwiftSyntaxParser1
    3. SynParser::parse() // libSwiftSyntaxParser
    4. ParserUnit::parse() // lib/Parse
    5. Parser::parseTopLevel() // lib/Parse
    1 Renamed from libSwiftSyntaxParser in tools/libSwiftSyntaxParser/CMakeLists.txt

    View Slide

  16. How SwiftSyntax interacts
    raw data with lib/Parse

    View Slide

  17. [Parse/Syntax] Refactoring to decouple the parser
    from syntax tree creation by akyrtzi · Pull Request
    #21368 · apple/swift
    Parser (C++ class)
    -> SyntaxParsingContext
    -> RootContextData
    -> ParsedRawSyntaxRecorder
    -> std::shared_ptr
    SyntaxParseActions defines interface between the parser
    and a receiver of raw syntax nodes.

    View Slide

  18. /// Interface between the parser and a receiver of raw syntax nodes.
    class SyntaxParseActions {
    virtual OpaqueSyntaxNode recordToken(tok tokenKind, ...) = 0;
    /// Record a missing token. \c loc can be invalid or an approximate location
    /// of where the token would be if not missing.
    virtual OpaqueSyntaxNode recordMissingToken(tok tokenKind, SourceLoc loc) = 0;
    /// The provided \c elements are an exact layout appropriate for the syntax
    /// \c kind. Missing optional elements are represented with a null
    /// OpaqueSyntaxNode object.
    virtual OpaqueSyntaxNode recordRawSyntax(syntax::SyntaxKind kind, ...) = 0;
    /// Used for incremental re-parsing.
    virtual std::pair
    lookupNode(size_t lexerOffset, syntax::SyntaxKind kind) {
    return std::make_pair(0, nullptr);
    }
    };

    View Slide

  19. CLibParseActions
    // libSwiftSyntaxParser.cpp
    class CLibParseActions : public SyntaxParseActions {
    // Uses registered node_handler
    // set via `swiftparse_parser_set_node_handler`
    ...
    }
    CLibParseActions is a concrete subclass of
    SyntaxParseActions to bridge C++ 㲗 C 㲗 Swift.

    View Slide

  20. Speeding up SwiftSyntax by
    using the parser directly
    - Swift Forums
    apple/swift#21762
    apple/swift-syntax#59

    View Slide

  21. My Experiments
    • Add `RunSwift5Command` (SwiftLang + ByteTree,
    experiment) by inamiy · Pull Request #13 · inamiy/
    SwiftRewriter
    • Speeds up 4.5x faster than swiftc + JSON
    • Update toolchain to `PR-21772-166` & use direct-parser by
    inamiy · Pull Request #14 · inamiy/SwiftRewriter
    • Speeds up 3x-6x faster than above SourceKit + ByteTree

    View Slide

  22. Thanks!
    Yasuhiro Inami
    @inamiy

    View Slide