IDE Development with Ruby

Transcript

1. IDE Development with Ruby Soutaro Matsumoto
 @soutaro Euruko 2021

2. Soutaro Matsumoto • Working for Square from Tokyo • A

   Ruby core committer working for RBS • Develops a static type checker Steep • @soutaro (GitHub, Twitter)

3. $ steep check # Type checking files: ............................................................................F................F.F.F.F.... .................................................... lib/rbs/definition_builder/ancestor_builder.rb:324:39:

   [error] Cannot pass a value of type `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName` │ ::RBS::AST::Members::Include <: ::RBS::TypeName │ ::RBS::AST::Members::Base <: ::RBS::TypeName │ ::Object <: ::RBS::TypeName │ ::BasicObject <: ::RBS::TypeName │ │ Diagnostic ID: Ruby::ArgumentTypeMismatch │ └ NoMixinFoundError.check!(member, env: env, member: member) ~~~~~~ lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive │ Diagnostic ID: Ruby::ElseOnExhaustiveCase │ └ raise ~~~~~ ...

4. IDE Powered by Steep • Install Steep and VSCode extension

   • Diagnostics reporting, completion, navigations, hover
5. None
6. None
7. None

8. IDE Features • Text editor • Build system • Debugger

   • Test runner • UI builder

9. IDE Features • Text editor • Syntax highlight • Folding

   • Diagnostics reporting • Hover • Navigations • Completion • Refactoring

10. Syntax Highlighting/Folding (vscode-ruby)

11. Diagnostics reporting (Steep)

12. Hover (Steep)

13. Navigations (Steep)

14. Completion (Steep)

15. Refactoring (RubyMine)

16. IDE Features • IDE features widely depend on the knowledge

    of the language • Syntax highlighting and folding requires the grammar • Diagnostics reporting, navigation, refactoring, ... are built on top of program analyses including type checking • (Build system and debugger depend on the language runtime)

17. Program Analyses Levels Text analyses Syntactic analyses Semantic analyses •

    The input is sequence of characters • "Line is too long" line = io.gets() • The input is a syntax tree of Ruby program • Syntax highlighting, folding, basic linter features line = io.gets() line = io.gets() • Analyze based on the everything of program • Type checking (error detection), navigations, completion, refactoring, ... Assignment Method call Local variable ::IO String | nil (::IO#gets)

18. IDE Development • IDE products need own program analyzers, on

    top of different APIs, in different languages • For RubyMine (Java) • For Visual Studio Code (TypeScript) • For Emacs (elisp) • For VIM (vimscript) • The analyzers provide essentially the same set of features IDE 1 UI Analyzer 1 IDE 2 UI Analyzer 2 IDE 3 UI Analyzer 3

19. Extracting the analyzers? • Extracts language specific IDE features •

    Share the implementation among IDE frontends

20. Language Server Protocol (LSP) • Supported by VSCode and many

    text editors • We can implement the server in Ruby! A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. https://microsoft.github.io/language-server-protocol/ Microsoft, 2016

21. IDE Development (revised) • Implement language specific features based on

    LSP • IDEs use the implementation VSCode Emacs VIM Analyzer LSP LSP LSP

22. Language Servers for Ruby • Steep • Solargraph • Sorbet

    (C++) • vscode-ruby has it's own language server (TypeScript)

23. Steep Architecture IDE Frontend Language Server Type Checker Steep LSP

    • IDE frontend starts Steep process • The communication is on pipe (stdio in Steep)

24. You type in the editor line = io.getsa

25. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } }

26. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } } Steep updates the source code and starts type checking

27. Steep detects a type error

28. PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics":

    [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error

29. Editor shows an error line = io.getsa PublishDiagnostics Notification {

    "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error

30. LSP events From IDE frontend to server From server to

    IDE frontend • textDocument/didChange • textDocument/didOpen • textDocument/didSave • textDocument/completion • textDocument/definition • textDocument/rename • workspace/symbol • textDocument/formatting • textDocument/foldingRange • textDocument/semanticTokens • textDocument/publishDiagnostics • window/showMessage • $/progress

31. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end

32. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop

33. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification

34. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code

35. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code Type checks and reports the detected errors

36. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get

37. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } }

38. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking

39. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking

40. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking

41. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...

42. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...

43. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking...

44. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking... Still type checking...

45. Responsiveness • Returning response quickly to user's interaction • Tricks

    to make Steep analysis more responsive • Incremental type checking • Open files first • Drop unrelated code

46. Incremental Type Checking • Type checking Ruby code affected by

    the change • Finishes much faster (when the project is big) line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class User attr_reader :name line = io.gets 1 changed file requires type checking 99 unchanged files can be skipped

47. The fast path • When you change a Ruby file,

    it type checks only the file • Fast enough: Type checking a Ruby file takes = 500ms~1s • When you change RBS files, it runs full type checking • Slow: Type checking all code may take minutes • (Implementing incremental RBS validation improved, but still needs type checking all Ruby files)

48. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files

49. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files

50. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files

51. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute?

52. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute? def to_namespace namespace.append(self.name) end def alias ? kin end def absolute! self.class.new(namespace: namespace.absolute!, name: name) end def absolute? namespace.absolute?

53. Steep Architecture IDE Frontend Language Server & Type Checker Steep

    LSP • Many tricks for responsiveness in the type checker implementation • No clear boundary between language server and the type checker

54. IDE Development with Ruby • IDE features need advanced program

    analyses like type checking • LSP allows development of IDE in any language • Steep is a static type checker implemented in Ruby with LSP support • Responsiveness is the key requirement • Shared some tricks to make Steep responsive

55. • Language Server Protocol - https://microsoft.github.io/language-server- protocol/ • Steep -

    https://github.com/soutaro/steep • Solargraph - https://solargraph.org • Sorbet - https://sorbet.org • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode- ruby • LanguageServer::Protocol Gem - https://github.com/mtsmfm/ language_server-protocol-ruby