Save 37% off PRO during our Black Friday Sale! »

IDE Development with Ruby

IDE Development with Ruby

Euruko 2021

1fab9d01b25e99522f3dfd01e3d4cb51?s=128

Soutaro Matsumoto

May 29, 2021
Tweet

Transcript

  1. IDE Development with Ruby Soutaro Matsumoto
 @soutaro Euruko 2021

  2. Soutaro Matsumoto • Working for Square from Tokyo • A

    Ruby core committer working for RBS • Develops a static type checker Steep • @soutaro (GitHub, Twitter)
  3. $ steep check # Type checking files: ............................................................................F................F.F.F.F.... .................................................... lib/rbs/definition_builder/ancestor_builder.rb:324:39:

    [error] Cannot pass a value of type `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName` │ ::RBS::AST::Members::Include <: ::RBS::TypeName │ ::RBS::AST::Members::Base <: ::RBS::TypeName │ ::Object <: ::RBS::TypeName │ ::BasicObject <: ::RBS::TypeName │ │ Diagnostic ID: Ruby::ArgumentTypeMismatch │ └ NoMixinFoundError.check!(member, env: env, member: member) ~~~~~~ lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive │ Diagnostic ID: Ruby::ElseOnExhaustiveCase │ └ raise ~~~~~ ...
  4. IDE Powered by Steep • Install Steep and VSCode extension

    • Diagnostics reporting, completion, navigations, hover
  5. None
  6. None
  7. None
  8. IDE Features • Text editor • Build system • Debugger

    • Test runner • UI builder
  9. IDE Features • Text editor • Syntax highlight • Folding

    • Diagnostics reporting • Hover • Navigations • Completion • Refactoring
  10. Syntax Highlighting/Folding (vscode-ruby)

  11. Diagnostics reporting (Steep)

  12. Hover (Steep)

  13. Navigations (Steep)

  14. Completion (Steep)

  15. Refactoring (RubyMine)

  16. IDE Features • IDE features widely depend on the knowledge

    of the language • Syntax highlighting and folding requires the grammar • Diagnostics reporting, navigation, refactoring, ... are built on top of program analyses including type checking • (Build system and debugger depend on the language runtime)
  17. Program Analyses Levels Text analyses Syntactic analyses Semantic analyses •

    The input is sequence of characters • "Line is too long" line = io.gets() • The input is a syntax tree of Ruby program • Syntax highlighting, folding, basic linter features line = io.gets() line = io.gets() • Analyze based on the everything of program • Type checking (error detection), navigations, completion, refactoring, ... Assignment Method call Local variable ::IO String | nil (::IO#gets)
  18. IDE Development • IDE products need own program analyzers, on

    top of different APIs, in different languages • For RubyMine (Java) • For Visual Studio Code (TypeScript) • For Emacs (elisp) • For VIM (vimscript) • The analyzers provide essentially the same set of features IDE 1 UI Analyzer 1 IDE 2 UI Analyzer 2 IDE 3 UI Analyzer 3
  19. Extracting the analyzers? • Extracts language specific IDE features •

    Share the implementation among IDE frontends
  20. Language Server Protocol (LSP) • Supported by VSCode and many

    text editors • We can implement the server in Ruby! A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. https://microsoft.github.io/language-server-protocol/ Microsoft, 2016
  21. IDE Development (revised) • Implement language specific features based on

    LSP • IDEs use the implementation VSCode Emacs VIM Analyzer LSP LSP LSP
  22. Language Servers for Ruby • Steep • Solargraph • Sorbet

    (C++) • vscode-ruby has it's own language server (TypeScript)
  23. Steep Architecture IDE Frontend Language Server Type Checker Steep LSP

    • IDE frontend starts Steep process • The communication is on pipe (stdio in Steep)
  24. You type in the editor line = io.getsa

  25. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } }
  26. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } } Steep updates the source code and starts type checking
  27. Steep detects a type error

  28. PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics":

    [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  29. Editor shows an error line = io.getsa PublishDiagnostics Notification {

    "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  30. LSP events From IDE frontend to server From server to

    IDE frontend • textDocument/didChange • textDocument/didOpen • textDocument/didSave • textDocument/completion • textDocument/definition • textDocument/rename • workspace/symbol • textDocument/formatting • textDocument/foldingRange • textDocument/semanticTokens • textDocument/publishDiagnostics • window/showMessage • $/progress
  31. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end
  32. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop
  33. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification
  34. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code
  35. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code Type checks and reports the detected errors
  36. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get
  37. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } }
  38. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  39. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  40. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking
  41. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  42. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  43. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking...
  44. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking... Still type checking...
  45. Responsiveness • Returning response quickly to user's interaction • Tricks

    to make Steep analysis more responsive • Incremental type checking • Open files first • Drop unrelated code
  46. Incremental Type Checking • Type checking Ruby code affected by

    the change • Finishes much faster (when the project is big) line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class User attr_reader :name line = io.gets 1 changed file requires type checking 99 unchanged files can be skipped
  47. The fast path • When you change a Ruby file,

    it type checks only the file • Fast enough: Type checking a Ruby file takes = 500ms~1s • When you change RBS files, it runs full type checking • Slow: Type checking all code may take minutes • (Implementing incremental RBS validation improved, but still needs type checking all Ruby files)
  48. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  49. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  50. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  51. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute?
  52. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute? def to_namespace namespace.append(self.name) end def alias ? kin end def absolute! self.class.new(namespace: namespace.absolute!, name: name) end def absolute? namespace.absolute?
  53. Steep Architecture IDE Frontend Language Server & Type Checker Steep

    LSP • Many tricks for responsiveness in the type checker implementation • No clear boundary between language server and the type checker
  54. IDE Development with Ruby • IDE features need advanced program

    analyses like type checking • LSP allows development of IDE in any language • Steep is a static type checker implemented in Ruby with LSP support • Responsiveness is the key requirement • Shared some tricks to make Steep responsive
  55. • Language Server Protocol - https://microsoft.github.io/language-server- protocol/ • Steep -

    https://github.com/soutaro/steep • Solargraph - https://solargraph.org • Sorbet - https://sorbet.org • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode- ruby • LanguageServer::Protocol Gem - https://github.com/mtsmfm/ language_server-protocol-ruby