SwiftSyntax Direct Parser (Swift 5.1)

Waiwai Swiftc #10 (Apr 19, 2019)
https://iosdiscord.connpass.com/event/123573/

Transcript

1. SwiftSyntax
   Direct Parser
   (as of swift-DEVELOPMENT-SNAPSHOT-2019-04-10-a)
   2019/04/19 Waiwai Swiftc
   Yasuhiro Inami / @inamiy
   

   View Slide

2. Swift 4.2-5.0:
   SwiftcRunner
   

   View Slide

3. // Swift 4.2-5.0 (swiftc runner)
   public enum SyntaxTreeParser {
   public static func parse(_ url: URL, swiftcURL: URL? = nil) throws
   -> SourceFileSyntax {
   let swiftcRunner = try SwiftcRunner(sourceFile: url, swiftcURL: swiftcURL)
   let result = try swiftcRunner.invoke()
   let deserializer = SyntaxTreeDeserializer()
   return try deserializer.deserialize(result.stdoutData,
   serializationFormat: .json)
   }
   }
   struct SwiftcRunner {
   func invoke() throws -> ProcessResult {
   var arguments = ["-frontend", "-emit-syntax"]
   arguments.append(...)
   return run(swiftcURL, arguments: arguments)
   }
   }
   

   View Slide

4. xcrun swiftc
   -frontend
   -emit-syntax
   Foo.swift
   

   View Slide

5. C++ AST
   ↓
   (JSON)
   ↓
   Swift AST
   

   View Slide

6. View Slide

7. Swift 5.0-dev:
   SwiftLang
   (SourceKit (XPC)+ ByteTree)
   

   View Slide

8. // Swift 5.0-dev (SourceKit + ByteTree, not in Xcode 10.2 toolchain)
   import SwiftLang
   extension SwiftLang {
   fileprivate static func parse(
   _ content: SourceFile, into format: SyntaxTreeFormat
   ) throws -> Tree {
   let Service = SourceKitdService()
   let Request = SourceKitdRequest(uid: .request_EditorOpen)
   Request.addParameter(.key_SyntaxTreeSerializationFormat, value: format.kind)
   let Resp = Service.sendSyn(request: Request)
   let CloseReq = SourceKitdRequest(uid: .request_EditorClose)
   let CloseResp = Service.sendSyn(request: CloseReq)
   return try format.makeTree(Resp.value)
   }
   }
   

   View Slide

9. View Slide

10. apple/swift/docs/ByteTree.md
    [libSyntax] Add a binary
    serialization format for syntax
    trees by ahoppen · Pull Request
    #18497 · apple/swift
    

    View Slide

11. View Slide

12. Swift 5.1:
    libSwiftSyntaxParser
    (Direct & incremental parser)
    

    View Slide

13. // Swift 5.1 (direct parser, not in Xcode 10.2 toolchain)
    import _InternalSwiftSyntaxParser // renamed from SwiftSyntaxParser
    public enum SyntaxParser {
    private static func parseRaw( // called from `SyntaxParser.parse`
    _ source: String,
    _ parseTransition: IncrementalParseTransition?,
    _ filenameForDiagnostics: String,
    _ diagnosticEngine: DiagnosticEngine?
    ) -> RawSyntax {
    let c_parser = swiftparse_parser_create()
    ... // See next slide for more details
    let c_top = swiftparse_parse_string(c_parser, source)
    return RawSyntax.moveFromOpaque(c_top)!
    }
    }
    

    View Slide

14. let c_parser = swiftparse_parser_create()
    defer {
    swiftparse_parser_dispose(c_parser)
    }
    // Transfer `RawSyntax` ownership to C parser.
    swiftparse_parser_set_node_handler(c_parser, nodeHandler);
    // For incremental parsing.
    swiftparse_parser_set_node_lookup(c_parser, nodeLookup);
    defer {
    diagnosticEngine.diagnose(Diagnostic(pendingDiag, pendingNotes))
    }
    swiftparse_parser_set_diagnostic_handler(c_parser, diagHandler)
    let c_top = swiftparse_parse_string(c_parser, source) // run parser
    return RawSyntax.moveFromOpaque(c_top)! // get ownership back
    

    View Slide

15. How SwiftSyntax access to lib/Parse
    1. SyntaxParser.parse() // SwiftSyntax
    2. swiftparse_parse_string()
                        // _InternalSwiftSyntaxParser1
    3. SynParser::parse() // libSwiftSyntaxParser
    4. ParserUnit::parse() // lib/Parse
    5. Parser::parseTopLevel() // lib/Parse
    1 Renamed from libSwiftSyntaxParser in tools/libSwiftSyntaxParser/CMakeLists.txt
    

    View Slide

16. How SwiftSyntax interacts
    raw data with lib/Parse
    

    View Slide

17. [Parse/Syntax] Refactoring to decouple the parser
    from syntax tree creation by akyrtzi · Pull Request
    #21368 · apple/swift
    Parser (C++ class)
    -> SyntaxParsingContext
    -> RootContextData
    -> ParsedRawSyntaxRecorder
    -> std::shared_ptr
    SyntaxParseActions defines interface between the parser
    and a receiver of raw syntax nodes.
    

    View Slide

18. /// Interface between the parser and a receiver of raw syntax nodes.
    class SyntaxParseActions {
    virtual OpaqueSyntaxNode recordToken(tok tokenKind, ...) = 0;
    /// Record a missing token. \c loc can be invalid or an approximate location
    /// of where the token would be if not missing.
    virtual OpaqueSyntaxNode recordMissingToken(tok tokenKind, SourceLoc loc) = 0;
    /// The provided \c elements are an exact layout appropriate for the syntax
    /// \c kind. Missing optional elements are represented with a null
    /// OpaqueSyntaxNode object.
    virtual OpaqueSyntaxNode recordRawSyntax(syntax::SyntaxKind kind, ...) = 0;
    /// Used for incremental re-parsing.
    virtual std::pair
    lookupNode(size_t lexerOffset, syntax::SyntaxKind kind) {
    return std::make_pair(0, nullptr);
    }
    };
    

    View Slide

19. CLibParseActions
    // libSwiftSyntaxParser.cpp
    class CLibParseActions : public SyntaxParseActions {
    // Uses registered node_handler
    // set via `swiftparse_parser_set_node_handler`
    ...
    }
    CLibParseActions is a concrete subclass of
    SyntaxParseActions to bridge C++ 㲗 C 㲗 Swift.
    

    View Slide

20. Speeding up SwiftSyntax by
    using the parser directly
    - Swift Forums
    apple/swift#21762
    apple/swift-syntax#59
    

    View Slide

21. My Experiments
    • Add `RunSwift5Command` (SwiftLang + ByteTree,
    experiment) by inamiy · Pull Request #13 · inamiy/
    SwiftRewriter
    • Speeds up 4.5x faster than swiftc + JSON
    • Update toolchain to `PR-21772-166` & use direct-parser by
    inamiy · Pull Request #14 · inamiy/SwiftRewriter
    • Speeds up 3x-6x faster than above SourceKit + ByteTree
    

    View Slide

22. Thanks!
    Yasuhiro Inami
    @inamiy
    

    View Slide