Tweet
Tweet

Slide 1

Slide 1 text

API for docs Soutaro Matsumoto

Slide 2

Slide 2 text

Soutaro Matsumoto • Ruby committer working for RBS/ Steep • Working at Timee • Started writing a series of articles for ιϑτ΢ΣΞσβΠϯ New

Slide 3

Slide 3 text

Soutaro Matsumoto • Ruby committer working for RBS/ Steep • Working at Timee • Started writing a series of articles for ιϑτ΢ΣΞσβΠϯ New

Slide 4

Slide 4 text

Steep & RBS updates • RBS 3.4 (RubyKaigi 2024) → 3.9 • Steep 1.6 (RubyKaigi 2024) → 1.10

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Generics with default types rbs-3.6

Slide 7

Slide 7 text

Generics with default types rbs-3.6 steep:ignore annotation steep-1.7

Slide 8

Slide 8 text

Generics with default types rbs-3.6 steep:ignore annotation steep-1.7 Better project organization steep-1.9

Slide 9

Slide 9 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 steep:ignore annotation steep-1.7 Better project organization steep-1.9

Slide 10

Slide 10 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9

Slide 11

Slide 11 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork

Slide 12

Slide 12 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10

Slide 13

Slide 13 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10

Slide 14

Slide 14 text

Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10

Slide 15

Slide 15 text

Inline RBS declaration • Not yet... 🙇 • Working for RBS 4.0/Steep 2.0 with the feature • They will support inline RBS declaration directly, without rbs-inline gem

Slide 16

Slide 16 text

Documentation helper in Steep • Steep helps reading docs associated to the Ruby program components -- classes/modules/methods/interfaces/constants • Hover • Completion • Signature Help

Slide 17

Slide 17 text

Hover • Hover shows docs when you point the source code

Slide 18

Slide 18 text

Completion • Docs in the completion list

Slide 19

Slide 19 text

Signature Help • When you are writing the method arguments

Slide 20

Slide 20 text

Writing docs • You can put markdown text associated to RBS constructs

Slide 21

Slide 21 text

This is API • These features are implemented on the top of API that looks up docs with classes/modules/methods RBS fi les Internal data structure API Language Server

Slide 22

Slide 22 text

RDoc/YARD/... • RDoc and YARD generate human readable outputs

Slide 23

Slide 23 text

What is RDoc?

Slide 24

Slide 24 text

What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code

Slide 25

Slide 25 text

What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code The generated HTML fi les

Slide 26

Slide 26 text

What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code The generated HTML fi les The internal structure

Slide 27

Slide 27 text

## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Internal structure Output RBS fi les

Slide 28

Slide 28 text

## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Internal structure Output RBS fi les

Slide 29

Slide 29 text

RBS/Steep Implementation 1. The AST 2. The parser 3. The environment 4. The type checker 5. Displaying the docs RBS RBS RBS Steep Steep

Slide 30

Slide 30 text

The AST • The AST has comments associated to each RBS construct

Slide 31

Slide 31 text

The AST • The AST has comments associated to each RBS construct

Slide 32

Slide 32 text

The AST • The AST has comments associated to each RBS construct

Slide 33

Slide 33 text

The AST • The AST has comments associated to each RBS construct

Slide 34

Slide 34 text

The parser

Slide 35

Slide 35 text

The parser

Slide 36

Slide 36 text

The parser

Slide 37

Slide 37 text

The parser

Slide 38

Slide 38 text

... The parser

Slide 39

Slide 39 text

... The parser

Slide 40

Slide 40 text

... The parser current_token.range

Slide 41

Slide 41 text

... The parser current_token.range

Slide 42

Slide 42 text

... The parser current_token.range

Slide 43

Slide 43 text

The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module

Slide 44

Slide 44 text

The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module

Slide 45

Slide 45 text

The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module

Slide 46

Slide 46 text

The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module

Slide 47

Slide 47 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 48

Slide 48 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 49

Slide 49 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 50

Slide 50 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 51

Slide 51 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 52

Slide 52 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 53

Slide 53 text

Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding

Slide 54

Slide 54 text

Displaying the docs { "method": "document/hover", "id": 1, "params": { "textDocument": { uri: "file://..." }, "position": { "line": 10, "character": 20 } } } Language server { "id": 1, "result": { "contents": "## 📚 String#encoding\n\nReturns the encoding" } } 1. Find the Ruby node associated at the position 2. Fetch the docs for the node 3. Format the docs for LSP

Slide 55

Slide 55 text

Problems • Steep starts type checking when you edit only comments in RBS fi les

Slide 56

Slide 56 text

Problems • Steep starts type checking when you edit only comments in RBS fi les

Slide 57

Slide 57 text

s = String.new() s.encoding

Slide 58

Slide 58 text

s = String.new() s.encoding

Slide 59

Slide 59 text

s = String.new() s.encoding singleton(String) type

Slide 60

Slide 60 text

s = String.new() s.encoding singleton(String) type String class name

Slide 61

Slide 61 text

s = String.new() s.encoding singleton(String) type String class name docs

Slide 62

Slide 62 text

s = String.new() s.encoding singleton(String) type String class name docs

Slide 63

Slide 63 text

s = String.new() s.encoding singleton(String) type String class name docs overload

Slide 64

Slide 64 text

s = String.new() s.encoding singleton(String) type String class name docs overload docs

Slide 65

Slide 65 text

Index • Introducing an index would solve the problem • RBS docs are registered to the index • The language server pulls the docs from the index using identi fi ers • The index may be reused after server restart Index Language server Query the docs by identi fi ers

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

String#encoding

Slide 68

Slide 68 text

String#encoding Array.new

Slide 69

Slide 69 text

Identifying overloads String#encoding Array#[]

Slide 70

Slide 70 text

Identifying overloads String#encoding Array#[]

Slide 71

Slide 71 text

foo = Foo.new foo.foo("hello") foo.foo(123)

Slide 72

Slide 72 text

foo = Foo.new foo.foo("hello") foo.foo(123)

Slide 73

Slide 73 text

foo = Foo.new foo.foo("hello") foo.foo(123)

Slide 74

Slide 74 text

Overload identi fi er • With line number of the overload? • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 75

Slide 75 text

Overload identi fi er • With line number of the overload? • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 76

Slide 76 text

Overload identi fi er • With line number of the overload? • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 77

Slide 77 text

Overload identi fi er • The line number may change after comment update 😫 • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 78

Slide 78 text

Overload identi fi er • The line number may change after comment update 😫 • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 79

Slide 79 text

Overload identi fi er • The line number may change after comment update 😫 • Foo#[email protected]:4 • Foo#[email protected]:4

Slide 80

Slide 80 text

Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst

Slide 81

Slide 81 text

Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst

Slide 82

Slide 82 text

Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst

Slide 83

Slide 83 text

Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)

Slide 84

Slide 84 text

Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)

Slide 85

Slide 85 text

Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)

Slide 86

Slide 86 text

Annotate method names with types • Foo#foo@(String) -> void • Foo#foo@(Integer) -> void

Slide 87

Slide 87 text

JVM method descriptor • JVM method name contains parameter types and return type hello(Ljava/lang/String;I)Ljava/lang/String; String hello(String s, int i)

Slide 88

Slide 88 text

Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void

Slide 89

Slide 89 text

Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void Drop parameter names

Slide 90

Slide 90 text

Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void Drop parameter names Sort keyword args

Slide 91

Slide 91 text

s = String.new() s.encoding Type checker Doc system

Slide 92

Slide 92 text

s = String.new() s.encoding Type checker Doc system

Slide 93

Slide 93 text

s = String.new() s.encoding Type checker Doc system

Slide 94

Slide 94 text

s = String.new() s.encoding singleton(String) type String class name Type checker Doc system

Slide 95

Slide 95 text

s = String.new() s.encoding singleton(String) type String class name Type checker Doc system

Slide 96

Slide 96 text

s = String.new() s.encoding singleton(String) type String class name Type checker Doc system

Slide 97

Slide 97 text

s = String.new() s.encoding singleton(String) type String class name String#encoding@() -> Encoding overload Type checker Doc system

Slide 98

Slide 98 text

s = String.new() s.encoding singleton(String) type String class name String#encoding@() -> Encoding overload Type checker Doc system

Slide 99

Slide 99 text

🙇 Not fi nished yet

Slide 100

Slide 100 text

Can it be universal? • Making the index from other gems would be great idea • RDoc/YARD makes the index with their docs • Steep/ruby-lsp can use the index for their documentation features ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Universal index Output RBS fi les

Slide 101

Slide 101 text

Index content • Discussed the key of the index; assume we have a solution • How can we de fi ne the content of the index? • A data structure that can be shared with RBS/Steep, RDoc, YARD, which keeps the contents of all of them? • How about go-to-de fi nition?

Slide 102

Slide 102 text

Summary • The documentation needs APIs because it's used by IDE/editors to help programming • Steep already has implementation, but it is tightly coupled with type checker and causing problems • I plan to implement a doc system API (index) • The index may help other tools too (but I'm not sure if it's a good idea)