Slide 1

Slide 1 text

IDE Development with Ruby Soutaro Matsumoto
 @soutaro Euruko 2021

Slide 2

Slide 2 text

Soutaro Matsumoto • Working for Square from Tokyo • A Ruby core committer working for RBS • Develops a static type checker Steep • @soutaro (GitHub, Twitter)

Slide 3

Slide 3 text

$ steep check # Type checking files: ............................................................................F................F.F.F.F.... .................................................... lib/rbs/definition_builder/ancestor_builder.rb:324:39: [error] Cannot pass a value of type `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName` │ ::RBS::AST::Members::Include <: ::RBS::TypeName │ ::RBS::AST::Members::Base <: ::RBS::TypeName │ ::Object <: ::RBS::TypeName │ ::BasicObject <: ::RBS::TypeName │ │ Diagnostic ID: Ruby::ArgumentTypeMismatch │ └ NoMixinFoundError.check!(member, env: env, member: member) ~~~~~~ lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive │ Diagnostic ID: Ruby::ElseOnExhaustiveCase │ └ raise ~~~~~ ...

Slide 4

Slide 4 text

IDE Powered by Steep • Install Steep and VSCode extension • Diagnostics reporting, completion, navigations, hover

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

IDE Features • Text editor • Build system • Debugger • Test runner • UI builder

Slide 9

Slide 9 text

IDE Features • Text editor • Syntax highlight • Folding • Diagnostics reporting • Hover • Navigations • Completion • Refactoring

Slide 10

Slide 10 text

Syntax Highlighting/Folding (vscode-ruby)

Slide 11

Slide 11 text

Diagnostics reporting (Steep)

Slide 12

Slide 12 text

Hover (Steep)

Slide 13

Slide 13 text

Navigations (Steep)

Slide 14

Slide 14 text

Completion (Steep)

Slide 15

Slide 15 text

Refactoring (RubyMine)

Slide 16

Slide 16 text

IDE Features • IDE features widely depend on the knowledge of the language • Syntax highlighting and folding requires the grammar • Diagnostics reporting, navigation, refactoring, ... are built on top of program analyses including type checking • (Build system and debugger depend on the language runtime)

Slide 17

Slide 17 text

Program Analyses Levels Text analyses Syntactic analyses Semantic analyses • The input is sequence of characters • "Line is too long" line = io.gets() • The input is a syntax tree of Ruby program • Syntax highlighting, folding, basic linter features line = io.gets() line = io.gets() • Analyze based on the everything of program • Type checking (error detection), navigations, completion, refactoring, ... Assignment Method call Local variable ::IO String | nil (::IO#gets)

Slide 18

Slide 18 text

IDE Development • IDE products need own program analyzers, on top of different APIs, in different languages • For RubyMine (Java) • For Visual Studio Code (TypeScript) • For Emacs (elisp) • For VIM (vimscript) • The analyzers provide essentially the same set of features IDE 1 UI Analyzer 1 IDE 2 UI Analyzer 2 IDE 3 UI Analyzer 3

Slide 19

Slide 19 text

Extracting the analyzers? • Extracts language specific IDE features • Share the implementation among IDE frontends

Slide 20

Slide 20 text

Language Server Protocol (LSP) • Supported by VSCode and many text editors • We can implement the server in Ruby! A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. https://microsoft.github.io/language-server-protocol/ Microsoft, 2016

Slide 21

Slide 21 text

IDE Development (revised) • Implement language specific features based on LSP • IDEs use the implementation VSCode Emacs VIM Analyzer LSP LSP LSP

Slide 22

Slide 22 text

Language Servers for Ruby • Steep • Solargraph • Sorbet (C++) • vscode-ruby has it's own language server (TypeScript)

Slide 23

Slide 23 text

Steep Architecture IDE Frontend Language Server Type Checker Steep LSP • IDE frontend starts Steep process • The communication is on pipe (stdio in Steep)

Slide 24

Slide 24 text

You type in the editor line = io.getsa

Slide 25

Slide 25 text

You type in the editor line = io.getsa DidChangeTextDocument Notification { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } }

Slide 26

Slide 26 text

You type in the editor line = io.getsa DidChangeTextDocument Notification { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } } Steep updates the source code and starts type checking

Slide 27

Slide 27 text

Steep detects a type error

Slide 28

Slide 28 text

PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error

Slide 29

Slide 29 text

Editor shows an error line = io.getsa PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error

Slide 30

Slide 30 text

LSP events From IDE frontend to server From server to IDE frontend • textDocument/didChange • textDocument/didOpen • textDocument/didSave • textDocument/completion • textDocument/definition • textDocument/rename • workspace/symbol • textDocument/formatting • textDocument/foldingRange • textDocument/semanticTokens • textDocument/publishDiagnostics • window/showMessage • $/progress

Slide 31

Slide 31 text

Simplified Source Code while event = client.receive() case event[:method] when "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end

Slide 32

Slide 32 text

Simplified Source Code while event = client.receive() case event[:method] when "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop

Slide 33

Slide 33 text

Simplified Source Code while event = client.receive() case event[:method] when "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification

Slide 34

Slide 34 text

Simplified Source Code while event = client.receive() case event[:method] when "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code

Slide 35

Slide 35 text

Simplified Source Code while event = client.receive() case event[:method] when "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code Type checks and reports the detected errors

Slide 36

Slide 36 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get

Slide 37

Slide 37 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } }

Slide 38

Slide 38 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking

Slide 39

Slide 39 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking

Slide 40

Slide 40 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking

Slide 41

Slide 41 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...

Slide 42

Slide 42 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...

Slide 43

Slide 43 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking...

Slide 44

Slide 44 text

Problem • Running type checking on every single key hit blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking... Still type checking...

Slide 45

Slide 45 text

Responsiveness • Returning response quickly to user's interaction • Tricks to make Steep analysis more responsive • Incremental type checking • Open files first • Drop unrelated code

Slide 46

Slide 46 text

Incremental Type Checking • Type checking Ruby code affected by the change • Finishes much faster (when the project is big) line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class User attr_reader :name line = io.gets 1 changed file requires type checking 99 unchanged files can be skipped

Slide 47

Slide 47 text

The fast path • When you change a Ruby file, it type checks only the file • Fast enough: Type checking a Ruby file takes = 500ms~1s • When you change RBS files, it runs full type checking • Slow: Type checking all code may take minutes • (Implementing incremental RBS validation improved, but still needs type checking all Ruby files)

Slide 48

Slide 48 text

Open Files First • Open files have priority in the type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "Twitter" Ruby code opened RBS file you edit Not open files

Slide 49

Slide 49 text

Open Files First • Open files have priority in the type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "Twitter" Ruby code opened RBS file you edit Not open files

Slide 50

Slide 50 text

Open Files First • Open files have priority in the type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "Twitter" Ruby code opened RBS file you edit Not open files

Slide 51

Slide 51 text

Drop unrelated code • Completion needs more responsiveness (<= 300ms) • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute?

Slide 52

Slide 52 text

Drop unrelated code • Completion needs more responsiveness (<= 300ms) • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute? def to_namespace namespace.append(self.name) end def alias ? kin end def absolute! self.class.new(namespace: namespace.absolute!, name: name) end def absolute? namespace.absolute?

Slide 53

Slide 53 text

Steep Architecture IDE Frontend Language Server & Type Checker Steep LSP • Many tricks for responsiveness in the type checker implementation • No clear boundary between language server and the type checker

Slide 54

Slide 54 text

IDE Development with Ruby • IDE features need advanced program analyses like type checking • LSP allows development of IDE in any language • Steep is a static type checker implemented in Ruby with LSP support • Responsiveness is the key requirement • Shared some tricks to make Steep responsive

Slide 55

Slide 55 text

• Language Server Protocol - https://microsoft.github.io/language-server- protocol/ • Steep - https://github.com/soutaro/steep • Solargraph - https://solargraph.org • Sorbet - https://sorbet.org • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode- ruby • LanguageServer::Protocol Gem - https://github.com/mtsmfm/ language_server-protocol-ruby