$30 off During Our Annual Pro Sale. View Details »

IDE Development with Ruby

IDE Development with Ruby

Euruko 2021

Soutaro Matsumoto

May 29, 2021
Tweet

More Decks by Soutaro Matsumoto

Other Decks in Programming

Transcript

  1. IDE Development with Ruby Soutaro Matsumoto
 @soutaro Euruko 2021

  2. Soutaro Matsumoto • Working for Square from Tokyo • A

    Ruby core committer working for RBS • Develops a static type checker Steep • @soutaro (GitHub, Twitter)
  3. $ steep check # Type checking files: ............................................................................F................F.F.F.F.... .................................................... lib/rbs/definition_builder/ancestor_builder.rb:324:39:

    [error] Cannot pass a value of type `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName` │ ::RBS::AST::Members::Include <: ::RBS::TypeName │ ::RBS::AST::Members::Base <: ::RBS::TypeName │ ::Object <: ::RBS::TypeName │ ::BasicObject <: ::RBS::TypeName │ │ Diagnostic ID: Ruby::ArgumentTypeMismatch │ └ NoMixinFoundError.check!(member, env: env, member: member) ~~~~~~ lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive │ Diagnostic ID: Ruby::ElseOnExhaustiveCase │ └ raise ~~~~~ ...
  4. IDE Powered by Steep • Install Steep and VSCode extension

    • Diagnostics reporting, completion, navigations, hover
  5. None
  6. None
  7. None
  8. IDE Features • Text editor • Build system • Debugger

    • Test runner • UI builder
  9. IDE Features • Text editor • Syntax highlight • Folding

    • Diagnostics reporting • Hover • Navigations • Completion • Refactoring
  10. Syntax Highlighting/Folding (vscode-ruby)

  11. Diagnostics reporting (Steep)

  12. Hover (Steep)

  13. Navigations (Steep)

  14. Completion (Steep)

  15. Refactoring (RubyMine)

  16. IDE Features • IDE features widely depend on the knowledge

    of the language • Syntax highlighting and folding requires the grammar • Diagnostics reporting, navigation, refactoring, ... are built on top of program analyses including type checking • (Build system and debugger depend on the language runtime)
  17. Program Analyses Levels Text analyses Syntactic analyses Semantic analyses •

    The input is sequence of characters • "Line is too long" line = io.gets() • The input is a syntax tree of Ruby program • Syntax highlighting, folding, basic linter features line = io.gets() line = io.gets() • Analyze based on the everything of program • Type checking (error detection), navigations, completion, refactoring, ... Assignment Method call Local variable ::IO String | nil (::IO#gets)
  18. IDE Development • IDE products need own program analyzers, on

    top of different APIs, in different languages • For RubyMine (Java) • For Visual Studio Code (TypeScript) • For Emacs (elisp) • For VIM (vimscript) • The analyzers provide essentially the same set of features IDE 1 UI Analyzer 1 IDE 2 UI Analyzer 2 IDE 3 UI Analyzer 3
  19. Extracting the analyzers? • Extracts language specific IDE features •

    Share the implementation among IDE frontends
  20. Language Server Protocol (LSP) • Supported by VSCode and many

    text editors • We can implement the server in Ruby! A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. https://microsoft.github.io/language-server-protocol/ Microsoft, 2016
  21. IDE Development (revised) • Implement language specific features based on

    LSP • IDEs use the implementation VSCode Emacs VIM Analyzer LSP LSP LSP
  22. Language Servers for Ruby • Steep • Solargraph • Sorbet

    (C++) • vscode-ruby has it's own language server (TypeScript)
  23. Steep Architecture IDE Frontend Language Server Type Checker Steep LSP

    • IDE frontend starts Steep process • The communication is on pipe (stdio in Steep)
  24. You type in the editor line = io.getsa

  25. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } }
  26. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } } Steep updates the source code and starts type checking
  27. Steep detects a type error

  28. PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics":

    [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  29. Editor shows an error line = io.getsa PublishDiagnostics Notification {

    "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  30. LSP events From IDE frontend to server From server to

    IDE frontend • textDocument/didChange • textDocument/didOpen • textDocument/didSave • textDocument/completion • textDocument/definition • textDocument/rename • workspace/symbol • textDocument/formatting • textDocument/foldingRange • textDocument/semanticTokens • textDocument/publishDiagnostics • window/showMessage • $/progress
  31. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end
  32. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop
  33. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification
  34. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code
  35. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code Type checks and reports the detected errors
  36. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get
  37. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } }
  38. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  39. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  40. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking
  41. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  42. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  43. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking...
  44. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking... Still type checking...
  45. Responsiveness • Returning response quickly to user's interaction • Tricks

    to make Steep analysis more responsive • Incremental type checking • Open files first • Drop unrelated code
  46. Incremental Type Checking • Type checking Ruby code affected by

    the change • Finishes much faster (when the project is big) line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class User attr_reader :name line = io.gets 1 changed file requires type checking 99 unchanged files can be skipped
  47. The fast path • When you change a Ruby file,

    it type checks only the file • Fast enough: Type checking a Ruby file takes = 500ms~1s • When you change RBS files, it runs full type checking • Slow: Type checking all code may take minutes • (Implementing incremental RBS validation improved, but still needs type checking all Ruby files)
  48. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  49. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  50. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  51. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute?
  52. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute? def to_namespace namespace.append(self.name) end def alias ? kin end def absolute! self.class.new(namespace: namespace.absolute!, name: name) end def absolute? namespace.absolute?
  53. Steep Architecture IDE Frontend Language Server & Type Checker Steep

    LSP • Many tricks for responsiveness in the type checker implementation • No clear boundary between language server and the type checker
  54. IDE Development with Ruby • IDE features need advanced program

    analyses like type checking • LSP allows development of IDE in any language • Steep is a static type checker implemented in Ruby with LSP support • Responsiveness is the key requirement • Shared some tricks to make Steep responsive
  55. • Language Server Protocol - https://microsoft.github.io/language-server- protocol/ • Steep -

    https://github.com/soutaro/steep • Solargraph - https://solargraph.org • Sorbet - https://sorbet.org • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode- ruby • LanguageServer::Protocol Gem - https://github.com/mtsmfm/ language_server-protocol-ruby