IDE Development with Ruby

Euruko 2021

Transcript

1. IDE Development with Ruby
   Soutaro Matsumoto

   @soutaro
   Euruko 2021
   

   View Slide

2. Soutaro Matsumoto
   • Working for Square from Tokyo
   
   • A Ruby core committer working for RBS
   
   • Develops a static type checker Steep
   
   • @soutaro (GitHub, Twitter)
   

   View Slide

3. $ steep check
   # Type checking files:
   ............................................................................F................F.F.F.F....
   ....................................................
   lib/rbs/definition_builder/ancestor_builder.rb:324:39: [error] Cannot pass a value of type
   `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName`
   │ ::RBS::AST::Members::Include <: ::RBS::TypeName
   │ ::RBS::AST::Members::Base <: ::RBS::TypeName
   │ ::Object <: ::RBS::TypeName
   │ ::BasicObject <: ::RBS::TypeName
   │
   │ Diagnostic ID: Ruby::ArgumentTypeMismatch
   │
   └ NoMixinFoundError.check!(member, env: env, member: member)
   ~~~~~~
   lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive
   │ Diagnostic ID: Ruby::ElseOnExhaustiveCase
   │
   └ raise
   ~~~~~
   ...
   

   View Slide

4. IDE Powered by Steep
   • Install Steep and VSCode extension
   
   • Diagnostics reporting, completion, navigations, hover
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. IDE Features
   • Text editor
   
   • Build system
   
   • Debugger
   
   • Test runner
   
   • UI builder
   

   View Slide

9. IDE Features
   • Text editor
   
   • Syntax highlight
   
   • Folding
   
   • Diagnostics reporting
   
   • Hover
   
   • Navigations
   
   • Completion
   
   • Refactoring
   

   View Slide

10. Syntax Highlighting/Folding
    (vscode-ruby)
    

    View Slide

11. Diagnostics reporting
    (Steep)
    

    View Slide

12. Hover
    (Steep)
    

    View Slide

13. Navigations
    (Steep)
    

    View Slide

14. Completion
    (Steep)
    

    View Slide

15. Refactoring
    (RubyMine)
    

    View Slide

16. IDE Features
    • IDE features widely depend on the knowledge of the language
    
    • Syntax highlighting and folding requires the grammar
    
    • Diagnostics reporting, navigation, refactoring, ... are built on top of
    program analyses including type checking
    
    • (Build system and debugger depend on the language runtime)
    

    View Slide

17. Program Analyses Levels
    Text analyses Syntactic analyses Semantic analyses
    • The input is sequence of characters
    
    • "Line is too long"
    line = io.gets()
    • The input is a syntax tree of Ruby
    program
    
    • Syntax highlighting, folding, basic
    linter features
    line = io.gets() line = io.gets()
    • Analyze based on the everything of
    program
    
    • Type checking (error detection),
    navigations, completion, refactoring,
    ...
    Assignment
    Method call
    Local variable
    ::IO
    String | nil (::IO#gets)
    

    View Slide

18. IDE Development
    • IDE products need own program analyzers, on top of different APIs, in
    different languages
    
    • For RubyMine (Java)
    
    • For Visual Studio Code (TypeScript)
    
    • For Emacs (elisp)
    
    • For VIM (vimscript)
    
    • The analyzers provide essentially the same set of features
    IDE 1
    UI
    Analyzer 1
    IDE 2
    UI
    Analyzer 2
    IDE 3
    UI
    Analyzer 3
    

    View Slide

19. Extracting the analyzers?
    • Extracts language specific IDE features
    
    • Share the implementation among IDE frontends
    

    View Slide

20. Language Server Protocol (LSP)
    • Supported by VSCode and many text editors
    
    • We can implement the server in Ruby!
    A Language Server is meant to provide the language-specific
    smarts and communicate with development tools over a
    protocol that enables inter-process communication.
    https://microsoft.github.io/language-server-protocol/
    Microsoft, 2016
    

    View Slide

21. IDE Development (revised)
    • Implement language specific features based on LSP
    
    • IDEs use the implementation
    VSCode Emacs VIM
    Analyzer
    LSP
    LSP
    LSP
    

    View Slide

22. Language Servers for Ruby
    • Steep
    
    • Solargraph
    
    • Sorbet (C++)
    
    • vscode-ruby has it's own language server (TypeScript)
    

    View Slide

23. Steep Architecture
    IDE Frontend Language Server Type Checker
    Steep
    LSP
    • IDE frontend starts Steep process
    
    • The communication is on pipe (stdio in Steep)
    

    View Slide

24. You type in the editor
    line = io.getsa
    

    View Slide

25. You type in the editor
    line = io.getsa
    DidChangeTextDocument Notification
    {
    "method": "textDocument/didChange",
    "params":
    {
    "textDocument":
    {
    "uri": "/home/soutaro/src/foo/bar.rb",
    "version": 12
    }
    ,
    "contentChanges": [
    {
    "range":
    {
    "start":
    {
    line: 0, character: 14
    }
    ,
    "end":
    {
    line: 0, character: 15
    }
    }
    ,
    "text": "a"
    }
    ]
    }
    }
    

    View Slide

26. You type in the editor
    line = io.getsa
    DidChangeTextDocument Notification
    {
    "method": "textDocument/didChange",
    "params":
    {
    "textDocument":
    {
    "uri": "/home/soutaro/src/foo/bar.rb",
    "version": 12
    }
    ,
    "contentChanges": [
    {
    "range":
    {
    "start":
    {
    line: 0, character: 14
    }
    ,
    "end":
    {
    line: 0, character: 15
    }
    }
    ,
    "text": "a"
    }
    ]
    }
    }
    Steep updates the source code and
    starts type checking
    

    View Slide

27. Steep detects a type error
    

    View Slide

28. PublishDiagnostics Notification
    {
    "method": "textDocument/publishDiagnostics",
    "params":
    {
    "uri": "/home/soutaro/src/foo/bar.rb",
    "diagnostics": [
    {
    "range":
    {
    "start":
    {
    "line": 0, "character": 10
    }
    ,
    "end":
    {
    "line": 0, "character": 15
    }
    }
    ,
    "message": "Type `::IO` does not have method `getsa`"
    }
    ]
    }
    }
    Steep detects a type error
    

    View Slide

29. Editor shows an error
    line = io.getsa
    PublishDiagnostics Notification
    {
    "method": "textDocument/publishDiagnostics",
    "params":
    {
    "uri": "/home/soutaro/src/foo/bar.rb",
    "diagnostics": [
    {
    "range":
    {
    "start":
    {
    "line": 0, "character": 10
    }
    ,
    "end":
    {
    "line": 0, "character": 15
    }
    }
    ,
    "message": "Type `::IO` does not have method `getsa`"
    }
    ]
    }
    }
    Steep detects a type error
    

    View Slide

30. LSP events
    From IDE frontend to server From server to IDE frontend
    • textDocument/didChange
    
    • textDocument/didOpen
    
    • textDocument/didSave
    
    • textDocument/completion
    
    • textDocument/definition
    
    • textDocument/rename
    
    • workspace/symbol
    
    • textDocument/formatting
    
    • textDocument/foldingRange
    
    • textDocument/semanticTokens
    • textDocument/publishDiagnostics
    
    • window/showMessage
    
    • $/progress
    

    View Slide

31. Simplified Source Code
    while
    event = client.receive()
    case
    event[:method]
    when
    "textDocument/didChange"
    checker.update_source_code(event[:params])
    checker.type_check
    do
    |error|
    client.send_diagnostics(error)
    end
    when
    "textDocument/completion"
    ...
    end
    end
    

    View Slide

32. Simplified Source Code
    while
    event = client.receive()
    case
    event[:method]
    when
    "textDocument/didChange"
    checker.update_source_code(event[:params])
    checker.type_check
    do
    |error|
    client.send_diagnostics(error)
    end
    when
    "textDocument/completion"
    ...
    end
    end
    Main loop
    

    View Slide

33. Simplified Source Code
    while
    event = client.receive()
    case
    event[:method]
    when
    "textDocument/didChange"
    checker.update_source_code(event[:params])
    checker.type_check
    do
    |error|
    client.send_diagnostics(error)
    end
    when
    "textDocument/completion"
    ...
    end
    end
    Main loop
    When the event is
    didChange notification
    

    View Slide

34. Simplified Source Code
    while
    event = client.receive()
    case
    event[:method]
    when
    "textDocument/didChange"
    checker.update_source_code(event[:params])
    checker.type_check
    do
    |error|
    client.send_diagnostics(error)
    end
    when
    "textDocument/completion"
    ...
    end
    end
    Main loop
    When the event is
    didChange notification
    Updates the source code
    

    View Slide

35. Simplified Source Code
    while
    event = client.receive()
    case
    event[:method]
    when
    "textDocument/didChange"
    checker.update_source_code(event[:params])
    checker.type_check
    do
    |error|
    client.send_diagnostics(error)
    end
    when
    "textDocument/completion"
    ...
    end
    end
    Main loop
    When the event is
    didChange notification
    Updates the source code
    Type checks and reports the
    detected errors
    

    View Slide

36. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    

    View Slide

37. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    

    View Slide

38. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    Start type checking
    

    View Slide

39. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    Start type checking
    

    View Slide

40. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 13 },
    end: { line: 0, character: 14 }
    },
    text: "s"
    }
    ]
    }
    }
    Start type checking
    

    View Slide

41. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 13 },
    end: { line: 0, character: 14 }
    },
    text: "s"
    }
    ]
    }
    }
    Start type checking
    Still type checking...
    

    View Slide

42. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    line = io.getsa
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 13 },
    end: { line: 0, character: 14 }
    },
    text: "s"
    }
    ]
    }
    }
    Start type checking
    Still type checking...
    

    View Slide

43. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    line = io.getsa
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 13 },
    end: { line: 0, character: 14 }
    },
    text: "s"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    Start type checking
    Still type checking...
    

    View Slide

44. Problem
    • Running type checking on every single key hit blocks user's interaction
    line = io.get
    line = io.gets
    line = io.getsa
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 12 },
    end: { line: 0, character: 13 }
    },
    text: "t"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    start: { line: 0, character: 13 },
    end: { line: 0, character: 14 }
    },
    text: "s"
    }
    ]
    }
    }
    {
    method: "textDocument/didChange",
    params: {
    textDocument: {
    uri: "/home/soutaro/src/foo/bar.rb",
    version: 12
    },
    contentChanges: [
    {
    range: {
    Start type checking
    Still type checking...
    Still type checking...
    

    View Slide

45. Responsiveness
    • Returning response quickly to user's interaction
    
    • Tricks to make Steep analysis more responsive
    
    • Incremental type checking
    
    • Open files first
    
    • Drop unrelated code
    

    View Slide

46. Incremental Type Checking
    • Type checking Ruby code affected by the change
    
    • Finishes much faster (when the project is big)
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    class User
    attr_reader :name
    line = io.gets
    1 changed file requires type checking 99 unchanged files can be skipped
    

    View Slide

47. The fast path
    • When you change a Ruby file, it type checks only the file
    
    • Fast enough: Type checking a Ruby file takes = 500ms~1s
    
    • When you change RBS files, it runs full type checking
    
    • Slow: Type checking all code may take minutes
    
    • (Implementing incremental RBS validation improved, but still needs
    type checking all Ruby files)
    

    View Slide

48. Open Files First
    • Open files have priority in the type checking queue for quick feedbacks
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    class Conference
    attr_reader :talks
    class User
    attr_reader name: String
    attr_reader email: String
    attr_reader twitter: String
    def twitter_url: () -> String
    end
    user = User.load(payload)
    url = user.twitter_url
    "Twitter"
    Ruby code opened RBS file you edit Not open files
    

    View Slide

49. Open Files First
    • Open files have priority in the type checking queue for quick feedbacks
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    class Conference
    attr_reader :talks
    class User
    attr_reader name: String
    attr_reader email: String
    attr_reader twitter: String
    def twitter_url: () -> String
    end
    user = User.load(payload)
    url = user.twitter_url
    "Twitter"
    Ruby code opened RBS file you edit Not open files
    

    View Slide

50. Open Files First
    • Open files have priority in the type checking queue for quick feedbacks
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    line = io.get
    class Conference
    attr_reader :talks
    class User
    attr_reader name: String
    attr_reader email: String
    attr_reader twitter: String
    def twitter_url: () -> String
    end
    user = User.load(payload)
    url = user.twitter_url
    "Twitter"
    Ruby code opened RBS file you edit Not open files
    

    View Slide

51. Drop unrelated code
    • Completion needs more responsiveness (<= 300ms)
    
    • Users wait for completion candidates when they use completion
    
    • Drop the unrelated method definitions to make the source code shorter
    def
    to_namespace
    namespace.append(
    self
    .name)
    end
    def alias
    ?
    kin
    end
    def
    absolute!
    self
    .
    class
    .new(namespace: namespace.absolute!, name: name)
    end
    def
    absolute?
    

    View Slide

52. Drop unrelated code
    • Completion needs more responsiveness (<= 300ms)
    
    • Users wait for completion candidates when they use completion
    
    • Drop the unrelated method definitions to make the source code shorter
    def
    to_namespace
    namespace.append(
    self
    .name)
    end
    def alias
    ?
    kin
    end
    def
    absolute!
    self
    .
    class
    .new(namespace: namespace.absolute!, name: name)
    end
    def
    absolute?
    def to_namespace
    namespace.append(self.name)
    end
    def alias
    ?
    kin
    end
    def absolute!
    self.class.new(namespace: namespace.absolute!, name: name)
    end
    def absolute?
    namespace.absolute?
    

    View Slide

53. Steep Architecture
    IDE Frontend Language Server & Type Checker
    Steep
    LSP
    • Many tricks for responsiveness in the type checker implementation
    
    • No clear boundary between language server and the type checker
    

    View Slide

54. IDE Development with Ruby
    • IDE features need advanced program analyses like type checking
    
    • LSP allows development of IDE in any language
    
    • Steep is a static type checker implemented in Ruby with LSP support
    
    • Responsiveness is the key requirement
    
    • Shared some tricks to make Steep responsive
    

    View Slide

55. • Language Server Protocol - https://microsoft.github.io/language-server-
    protocol/
    
    • Steep - https://github.com/soutaro/steep
    
    • Solargraph - https://solargraph.org
    
    • Sorbet - https://sorbet.org
    
    • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode-
    ruby
    
    • LanguageServer::Protocol Gem - https://github.com/mtsmfm/
    language_server-protocol-ruby
    

    View Slide