Code indexing: How language servers understand our code

Slide 1

Slide 1 text

Vinicius Stock Code indexing How language servers understand our code

Slide 2

Slide 2 text

Vinicius Stock Senior dev @ Ruby DX team Shopify Twitter: @vinistock GitHub: @vinistock https://vinistock.com

Slide 3

Slide 3 text

What is a language server?

Slide 4

Slide 4 text

Source: https://microsoft.github.io/language-server-protocol/specification

Slide 5

Slide 5 text

Editor (Client) Language server STDIN STDOUT User Go to def Definition Location Jumps to def

Slide 6

Slide 6 text

Language servers can connect to any editor

Slide 7

Slide 7 text

We can collaborate in improving the experience

Slide 8

Slide 8 text

Improving the development experience with language servers RubyConf 2022

Slide 9

Slide 9 text

Shopify/vscode-ruby-lsp Shopify/ruby-lsp Ruby LSP

Slide 10

Slide 10 text

Semantic highlighting Document symbol Document link Hover Folding range Selection range Formatting On type formatting Diagnostic Code actions Document highlight Inlay hints Path completion Code Lens

Slide 11

Slide 11 text

Document symbol

Slide 12

Slide 12 text

Semantic highlighting Document symbol Document link Hover Folding range Selection range Formatting On type formatting Diagnostic Code actions Document highlight Inlay hints Path completion Code Lens

Slide 13

Slide 13 text

Diagnostic Code actions

Slide 14

Slide 14 text

Code actions

Slide 15

Slide 15 text

Semantic highlighting Document symbol Document link Hover Folding range Selection range Formatting On type formatting Diagnostic Code actions Document highlight Inlay hints Path completion Code Lens

Slide 16

Slide 16 text

Code Lens

Slide 17

Slide 17 text

To explore code indexing, we’ll think about go to definition

Slide 18

Slide 18 text

class Foo def process Bar.baz end end

Slide 19

Slide 19 text

{ "params": { "textDocument": { "uri": "file: / / / foo.rb" }, "position": { "line": 2, "character": 5 } } }

Slide 20

Slide 20 text

{ "uri": "file: / / / bar.rb" "range": { "start": { "line": 0, "character": 0 } "end": { "line": 5, "character": 2 } } }

Slide 21

Slide 21 text

The request and response are just file locations

Slide 22

Slide 22 text

What’s happening in between?

Slide 23

Slide 23 text

class Foo def process Bar.baz end end How do we determine what’s under the cursor? How do we find the definition?

Slide 24

Slide 24 text

Locating targets Part 1

Slide 25

Slide 25 text

We want to search an AST to find the node at the requested position

Slide 26

Slide 26 text

We must parse the file and then search the AST

Slide 27

Slide 27 text

0 class Foo 1 def process 2 Bar.baz 3 end 4 end CLASS CONST (Foo) BODY DEF IDENT (process) BODY CALL CONST (Bar) IDENT (baz)

Slide 28

Slide 28 text

More than one node covers the requested position

Slide 29

Slide 29 text

We’re looking for the most specific one

Slide 30

Slide 30 text

Converting position into string index Step 1

Slide 31

Slide 31 text

"position": { "line": 2,"character": 5 }

Slide 32

Slide 32 text

"position": { "line": 2,"character": 5 } source = "class Foo\n def process\n Bar.baz…"

Slide 33

Slide 33 text

"position": { "line": 2,"character": 5 } source = "class Foo\n def process\n Bar.baz…" line + character = > string index

Slide 34

Slide 34 text

"position": { "line": 2,"character": 5 } source = "class Foo\n def process\n Bar.baz…" line + character = > string index { "line": 2,"character": 5 } = > 28

Slide 35

Slide 35 text

Locating the AST node Step 2

Slide 36

Slide 36 text

1. Go through AST 2. Use target and candidate pointers 3. Compare proximity to requested index

Slide 37

Slide 37 text

0 class Foo 1 def process 2 Bar.baz 3 end 4 end CLASS CONST (Foo) BODY DEF IDENT (process) BODY CALL CONST (Bar) IDENT (baz)

Slide 38

Slide 38 text

def locate(node, index) end

Slide 39

Slide 39 text

def locate(node, index) queue = node.child_nodes target = node end

Slide 40

Slide 40 text

def locate(node, index) queue = node.child_nodes target = node until queue.empty? end target end

Slide 41

Slide 41 text

until queue.empty? candidate = queue.shift end

Slide 42

Slide 42 text

until queue.empty? candidate = queue.shift queue.concat(candidate.child_nodes) end

Slide 43

Slide 43 text

until queue.empty? candidate = queue.shift queue.concat(candidate.child_nodes) loc = candidate.location next unless loc.cover?(index) end

Slide 44

Slide 44 text

until queue.empty? candidate = queue.shift queue.concat(candidate.child_nodes) loc = candidate.location next unless loc.cover?(index) break if index < loc.start_char end

Slide 45

Slide 45 text

until queue.empty? # … break if index < loc.start_char tloc = target.location if loc.end_char - loc.start_char < = tloc.end_char - tloc.start_char target = candidate end end

Slide 46

Slide 46 text

We discovered the target 🎉

Slide 47

Slide 47 text

But where’s the definition?

Slide 48

Slide 48 text

Code indexing Part 2

Slide 49

Slide 49 text

We have to know what’s available in the codebase

Slide 50

Slide 50 text

Index Classes Modules Methods Constants Does this thing exist? Where is it?

Slide 51

Slide 51 text

We can’t populate the index on every definition request

Slide 52

Slide 52 text

We need to initialize the index on boot

Slide 53

Slide 53 text

And we have to keep it synchronized if files are modified

Slide 54

Slide 54 text

Let’s start by the initial indexing

Slide 55

Slide 55 text

We’ll focus implementing classes in 4 steps

Slide 56

Slide 56 text

Ruby code Parser AST Visitor Index AST Classes For all Ruby files

Slide 57

Slide 57 text

Collecting class declarations in an AST Step 1

Slide 58

Slide 58 text

class IndexVisitor < SyntaxTree : : Visitor end

Slide 59

Slide 59 text

class IndexVisitor < SyntaxTree : : Visitor attr_reader :constants def initialize @constants = [] end end

Slide 60

Slide 60 text

class IndexVisitor < SyntaxTree : : Visitor attr_reader :constants def initialize @constants = [] end def visit_class(node) @constants < < node super end end

Slide 61

Slide 61 text

visitor = IndexVisitor.new visitor.visit(ast) visitor.constants # = > [ # #, # # # ]

Slide 62

Slide 62 text

Creating the Index class Step 2

Slide 63

Slide 63 text

class Index include Singleton def initialize @knowledge = {} end def add(path, constants) constants.each do |const| end end end

Slide 64

Slide 64 text

class Index include Singleton def initialize @knowledge = {} end def add(path, constants) constants.each do |const| @knowledge[const.name] = { } end end end

Slide 65

Slide 65 text

class Index include Singleton def initialize @knowledge = {} end def add(path, constants) constants.each do |const| @knowledge[const.name] = { file: path, location: const.location } end end end

Slide 66

Slide 66 text

{ "Foo" = > { path: "/foo.rb", location: # } }

Slide 67

Slide 67 text

Populating the Index Step 3

Slide 68

Slide 68 text

index = Index.instance all_files_in_load_path.each |path| index.process("file : / / # { path}") end

Slide 69

Slide 69 text

class Index def process(uri) path = URI(uri).path end end

Slide 70

Slide 70 text

class Index def process(uri) path = URI(uri).path content = File.read(path) end end

Slide 71

Slide 71 text

class Index def process(uri) path = URI(uri).path content = File.read(path) ast = SyntaxTree.parse(content) end end

Slide 72

Slide 72 text

class Index def process(uri) path = URI(uri).path content = File.read(path) ast = SyntaxTree.parse(content) visitor = IndexVisitor.new visitor.visit(ast) end end

Slide 73

Slide 73 text

class Index def process(uri) path = URI(uri).path content = File.read(path) ast = SyntaxTree.parse(content) visitor = IndexVisitor.new visitor.visit(ast) add(path, visitor.constants) end end

Slide 74

Slide 74 text

The initial indexing is done

Slide 75

Slide 75 text

Synchronizing file modifications Step 4

Slide 76

Slide 76 text

If a file is modified, the existing classes might change

Slide 77

Slide 77 text

LSPs can request file watching

Slide 78

Slide 78 text

Remove related entries Index the new file Ruby file modified Deleted Added Remove and index Changed

Slide 79

Slide 79 text

def execute(req) case req[:method] when "workspace/didChangeWatchedFiles" changes = req.dig(:params, :changes) Index.instance.synchronize(changes) end end

Slide 80

Slide 80 text

class Index def synchronize(changes) changes.each do |change| case change[:type] end end end end

Slide 81

Slide 81 text

class Index def synchronize(changes) changes.each do |change| case change[:type] when "deleted" remove_entries_for(change[:uri]) end end end end

Slide 82

Slide 82 text

class Index def synchronize(changes) changes.each do |change| case change[:type] when "deleted" remove_entries_for(change[:uri]) when "created" process(change[:uri]) end end end end

Slide 83

Slide 83 text

class Index def synchronize(changes) changes.each do |change| case change[:type] when "deleted" remove_entries_for(change[:uri]) when "created" process(change[:uri]) when "changed" remove_entries_for(change[:uri]) process(change[:uri]) end end end end

Slide 84

Slide 84 text

class Index def remove_entries_for(uri) path = URI(uri).path @knowledge.each do |const, loc| if loc[:file] = = path @knowledge.delete(const) end end end end

Slide 85

Slide 85 text

We can now perform initial indexing and synchronization 🎉

Slide 86

Slide 86 text

We’re ready to implement go to definition

Slide 87

Slide 87 text

Implementing go to definition Part 3

Slide 88

Slide 88 text

1. Locate the target 2. Find target in index 3. Return declaration location

Slide 89

Slide 89 text

class Definition < BaseRequest def initialize(document, position) @document = document @position = position end end

Slide 90

Slide 90 text

class Definition < BaseRequest def run target = @document.locate_node(@position) file, location = case target end # … end end

Slide 91

Slide 91 text

class Definition < BaseRequest def run target = @document.locate_node(@position) file, location = case target when SyntaxTree : : Const Index.instance.fetch(target.value) end # … end end

Slide 92

Slide 92 text

class Definition < BaseRequest def run # … { uri: "file : / / # { file}" range: { start: { line: location.start_line, character: location.start_column }, end: { line: location.end_line, character: location.end_column } } } end end

Slide 93

Slide 93 text

Go to definition for classes is done 🎉

Slide 94

Slide 94 text

This is not yet released in the Ruby LSP

Slide 95

Slide 95 text

But I have a demo!

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

Other features also depend on an index

Slide 98

Slide 98 text

Signature help Workspace symbols Hover Autocomplete

Slide 99

Slide 99 text

Signature help

Slide 100

Slide 100 text

Signature help Workspace symbols Hover Autocomplete

Slide 101

Slide 101 text

Workspace symbols

Slide 102

Slide 102 text

Signature help Workspace symbols Hover Autocomplete

Slide 103

Slide 103 text

Hover

Slide 104

Slide 104 text

Signature help Workspace symbols Hover Autocomplete

Slide 105

Slide 105 text

Autocomplete

Slide 106

Slide 106 text

This approach is similar to what a typechecker does

Slide 107

Slide 107 text

And there are still many improvements we can make

Slide 108

Slide 108 text

• Use parallelism to build index • Cache the index • Include the missing parts (modules, constants, methods…) • Add the option to control what is indexed

Slide 109

Slide 109 text

> lib > my_gem foo.rb > .ruby-lsp > lib > my_gem foo.indexed Last modified: 3 days ago Last modified: 1 minute ago

Slide 110

Slide 110 text

# Gemfile gem "ruby_index"

Slide 111

Slide 111 text

Multiple gems could reuse the index

Slide 112

Slide 112 text

"ruby-lsp" "steep" "typeprof" "rdoc" "irb"

Slide 113

Slide 113 text

Shopify/vscode-ruby-lsp Shopify/ruby-lsp Ruby LSP

Slide 114

Slide 114 text

Developer experience is a part of making developers happy

Slide 115

Slide 115 text

Let’s collaborate on improving it

Slide 116

Slide 116 text

Thank you

Slide 117

Slide 117 text

• https://github.com/Shopify/vscode-ruby-lsp • https://github.com/Shopify/ruby-lsp • https://microsoft.github.io/language-server- protocol/specification • Code font: Cascadia Code • Screenshots made with VS Code References