Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
π Marco Roth
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth
Full-Stack Developer & Open Source Contributor
Slide 3
Slide 3 text
Developer Tooling For The Modern
Hotwire & Rails Era
Marco Roth
Full-Stack Developer & Open Source Contributor
Slide 4
Slide 4 text
Understand LSPs
Building a simple LSP
What
β
s Next for Tooling
Slide 5
Slide 5 text
CableReady
Core Team
Maintainer
Slide 6
Slide 6 text
I β€ Open Source
Slide 7
Slide 7 text
rubyvideo.dev
Slide 8
Slide 8 text
Started to actively contribute in 2019
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Had enough
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
X
Slide 14
Slide 14 text
Superpower
β‘
Slide 15
Slide 15 text
The future of modern
Rails apps
Slide 16
Slide 16 text
I wanted to go back
doing Full-Stack Rails
Slide 17
Slide 17 text
I want to help build the
missing modern tooling
Slide 18
Slide 18 text
Went full-time on OSS from
Jan 2022 - Aug 2023
Slide 19
Slide 19 text
To work on Hotwire
and related projects
Slide 20
Slide 20 text
But, I noticed something
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
The JavaScript tooling
and DX is awesome
Slide 23
Slide 23 text
We didn
β
t really have anything
like it in the Ruby world
Slide 24
Slide 24 text
Or something quite as easy to
setup and get started
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
I started to miss these tools
while working in Ruby/Rails
Slide 27
Slide 27 text
No IDE
Slide 28
Slide 28 text
They felt clunky, heavyweight
and cumbersome to use
Slide 29
Slide 29 text
Made me
less productive
Slide 30
Slide 30 text
But this changed
in the last few years
Slide 31
Slide 31 text
Editor Evolution
Slide 32
Slide 32 text
Language Server Protocol
(LSP)
Slide 33
Slide 33 text
What is it and
why do we want to use it?
Slide 34
Slide 34 text
Language Intelligence
without the need for an IDE
Slide 35
Slide 35 text
Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
Slide 36
Slide 36 text
How does it work?
Slide 37
Slide 37 text
Your editor starts a
background process per
language
Slide 38
Slide 38 text
Communicates over
STDIN/STDOUT or TCP/IP
Slide 39
Slide 39 text
JSON-RPC
Slide 40
Slide 40 text
Similar to HTTP
Slide 41
Slide 41 text
Content-Length: ...\r\n
\r\n
{
"jsonrpc": "2.0",
"id": 1,
"method": "textDocument/completion",
"params": {
...
}
}
Slide 42
Slide 42 text
Stimulus LSP
Slide 43
Slide 43 text
What are we trying to solve?
Slide 44
Slide 44 text
Greet
Slide 45
Slide 45 text
Greet
Slide 46
Slide 46 text
Slide 47
Slide 47 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 48
Slide 48 text
Slide 49
Slide 49 text
Slide 50
Slide 50 text
Slide 51
Slide 51 text
These mistakes
happen so easily
Slide 52
Slide 52 text
I thought:
"We can do better!"
Slide 53
Slide 53 text
Jun 23, 2021
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
No content
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
Let
β
s take a look
how it works
Slide 59
Slide 59 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 60
Slide 60 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 61
Slide 61 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 62
Slide 62 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 63
Slide 63 text
The server returns
a JSON response
Slide 64
Slide 64 text
{
id: 1,
result: [
{
"label": "TypeScript",
"kind": 1,
"data": 1
},
{
"label": "JavaScript",
"kind": 1,
"data": 2
}
]
}
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
Greet
Slide 67
Slide 67 text
Embedded Languages
Slide 68
Slide 68 text
The Stimulus grammar is not
part of the HTML spec
Slide 69
Slide 69 text
So we need to extend the handling
for HTML-like documents
Slide 70
Slide 70 text
Language services
Slide 71
Slide 71 text
Which language do
you write it in?
Slide 72
Slide 72 text
No content
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
Existing Data Providers
Portability
Documentation/Integration
Existing Language Services
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
Data Providers
Slide 77
Slide 77 text
import { IHTMLDataProvider } from "vscode-html-languageservice"
export class StimulusHTMLDataProvider implements IHTMLDataProvider {
provideTags() {
...
}
provideAttributes(tag: string) {
...
}
provideValues(tag: string, attribute: string) {
...
}
}
Slide 78
Slide 78 text
Let
β
s implement it
Slide 79
Slide 79 text
provideTags() {
}
Slide 80
Slide 80 text
...
Slide 81
Slide 81 text
provideTags() {
}
Slide 82
Slide 82 text
provideTags() {
return []
}
Slide 83
Slide 83 text
provideAttributes(tag: string) {
}
Slide 84
Slide 84 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 85
Slide 85 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 86
Slide 86 text
provideAttributes(_tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 87
Slide 87 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 88
Slide 88 text
Parse JavaScript
Slide 89
Slide 89 text
Parse Stimulus Controllers
Slide 90
Slide 90 text
class ControllerDefinition {
readonly path: string
methods: Array = []
targets: Array = []
classes: Array = []
values: { [key: string]: Value } = {}
}
Slide 91
Slide 91 text
Acorn
Slide 92
Slide 92 text
import { Parser as AcornParser } from "acorn"
import { simple as walk } from "acorn-walk"
class Parser {
parseController(code: string, filename: string) {
...
}
}
Slide 93
Slide 93 text
f
app/javascript/controllers/hello_controller.js
Slide 94
Slide 94 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 95
Slide 95 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 96
Slide 96 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 97
Slide 97 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 98
Slide 98 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 99
Slide 99 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 100
Slide 100 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 101
Slide 101 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 102
Slide 102 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 103
Slide 103 text
We do something similar for
the Targets
Slide 104
Slide 104 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 105
Slide 105 text
Slide 106
Slide 106 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 107
Slide 107 text
provideAttributes(_tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 108
Slide 108 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 109
Slide 109 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 110
Slide 110 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 111
Slide 111 text
No content
Slide 112
Slide 112 text
Slide 113
Slide 113 text
provideValues(_tag: string, attribute: string) {
}
Slide 114
Slide 114 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 115
Slide 115 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 116
Slide 116 text
No content
Slide 117
Slide 117 text
https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
Slide 118
Slide 118 text
Jun 25, 2021
Slide 119
Slide 119 text
No content
Slide 120
Slide 120 text
What else can we do?
Slide 121
Slide 121 text
Can
β
t AI do all of that?
Slide 122
Slide 122 text
Maybe
Slide 123
Slide 123 text
Different Use Cases
Slide 124
Slide 124 text
AI for "boilerplate"
LSPs for reliability and accuracy
Slide 125
Slide 125 text
Limited Context Window
Missing relation between
relevant files
Slide 126
Slide 126 text
Diagnostics
Slide 127
Slide 127 text
No content
Slide 128
Slide 128 text
No content
Slide 129
Slide 129 text
No content
Slide 130
Slide 130 text
No content
Slide 131
Slide 131 text
No content
Slide 132
Slide 132 text
What else can we do?
Slide 133
Slide 133 text
Code Actions to
guide the user in the right
direction
Slide 134
Slide 134 text
No content
Slide 135
Slide 135 text
Fast forward to
October 2023
Slide 136
Slide 136 text
Available Now
Stimulus LSP
Slide 137
Slide 137 text
I asked the community for
feedback
Slide 138
Slide 138 text
And got a lot of awesome
feedback and ideas
Slide 139
Slide 139 text
Some of the naive approaches
we used were starting to fall
short
Slide 140
Slide 140 text
Especially the way we were parsing
the JavaScript with Acorn
Slide 141
Slide 141 text
We assumed and hard-coded
app/javascript/controllers/
Slide 142
Slide 142 text
import { application } from "controllers/application"
import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading"
eagerLoadControllersFrom("controllers", application)
Slide 143
Slide 143 text
No content
Slide 144
Slide 144 text
The Language Server works
Slide 145
Slide 145 text
but it produced a lot of false
positives and inaccurate results
Slide 146
Slide 146 text
Which is really
annoying as a user
Slide 147
Slide 147 text
We addressed and
fixed a lot of issues
Slide 148
Slide 148 text
Which technically ended up
in a rewrite of the parser
Slide 149
Slide 149 text
Fast forward to
May 2024
Slide 150
Slide 150 text
No content
Slide 151
Slide 151 text
No content
Slide 152
Slide 152 text
What
β
s next?
Slide 153
Slide 153 text
No content
Slide 154
Slide 154 text
No content
Slide 155
Slide 155 text
None of is this is
exclusive to the LSP
Slide 156
Slide 156 text
The diagnostics could
be used independently
Slide 157
Slide 157 text
Kind of like an independent
tool as RuboCop
Slide 158
Slide 158 text
Coming Soon
Stimulus Lint
Slide 159
Slide 159 text
Errors
Diagnostics
Best Practices
Recommendations
Slide 160
Slide 160 text
$ stimulus-lint
Stimulus Lint is inspecting 25 files
........................
25 files inspected, 0 offenses detected, 0 offenses autocorrectable
Slide 161
Slide 161 text
<%= tag.div(
data: {
controller: "filter",
filter_open_class: "border-white",
filter_close_class: "hover:bg-gray-100 border-gray-300"
}
) %>
Full support for HTML+ERB and Rails-specific helpers
Slide 162
Slide 162 text
Diagnostics
Code Actions
Code Completion
Refactoring Tools
Slide 163
Slide 163 text
Pre-Release
Turbo LSP
Slide 164
Slide 164 text
No content
Slide 165
Slide 165 text
Super Basic right now
Slide 166
Slide 166 text
Turbo also makes heavy use
of ERB and view helpers
Slide 167
Slide 167 text
So it
β
s only really useful if we
can provide intelligence for ERB
Slide 168
Slide 168 text
And that only works if
we actually understand ERB
Slide 169
Slide 169 text
Stimulus LSP understands
HTML in ERB documents
Slide 170
Slide 170 text
But it doesn
β
t understand
Embedded Ruby that outputs
HTML
Slide 171
Slide 171 text
No content
Slide 172
Slide 172 text
No content
Slide 173
Slide 173 text
Source: https://twitter.com/vinistock/status/1710280529889620038
Slide 174
Slide 174 text
ERB Support in Ruby LSP
Slide 175
Slide 175 text
RubyConf 2023
Hack Day
Slide 176
Slide 176 text
No content
Slide 177
Slide 177 text
We quickly realized that we need
something more powerful to make
things work beyond the basics
Slide 178
Slide 178 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 179
Slide 179 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 180
Slide 180 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 181
Slide 181 text
Source: https://github.com/Shopify/ruby-lsp/pull/2235
Slide 182
Slide 182 text
Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
Slide 183
Slide 183 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 184
Slide 184 text
We lost the HTML-context
in the process
Slide 185
Slide 185 text
<%= "Hello World" %>
>
Slide 186
Slide 186 text
HTML with embedded Ruby
(.html.erb, .rhtml, β¦)
Slide 187
Slide 187 text
ERB is versatile
Slide 188
Slide 188 text
a
Slide 189
Slide 189 text
Experiment
HTML-aware ERB Parser
Slide 190
Slide 190 text
Stimulus LSP
Stimulus Lint
Turbo LSP
Ruby LSP (maybe?)
and moreβ¦
Slide 191
Slide 191 text
type="text" <% end %> />
<% @posts.each do |post| %>
<%= post.title %>
<% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>Slide 192
Slide 192 text
Which made me think that it
could be useful to have an
HTML abstraction
Slide 193
Slide 193 text
<%= tag.p do %>
Hello World!
<% end %>
Hello World!
==Slide 194
Slide 194 text
==
<%= tag.div(
data: {
controller: "hello",
action: "click->hello#greet"
}
) %>
Slide 195
Slide 195 text
Needs more exploration
Slide 196
Slide 196 text
This might also open to door to
support Haml, Slim, Liquid, Phlex,
Blade and other template
languages
Slide 197
Slide 197 text
erb-languageservice
html-templating-languageservice
Slide 198
Slide 198 text
If the parser can understand any
ERB context and Rails-render
calls it is super powerful
Slide 199
Slide 199 text
This could power advanced
diagnostics
Slide 200
Slide 200 text
Slide 201
Slide 201 text
Slide 202
Slide 202 text
Slide 203
Slide 203 text
Slide 204
Slide 204 text
Modern Rails apps are composed of
multiple views/partials/components
Slide 205
Slide 205 text
<%= render partial: "input" %>
Slide 206
Slide 206 text
But if the parser understands
the Rails render logic
Slide 207
Slide 207 text
<%= render partial: "input" %>
Slide 208
Slide 208 text
<%= render partial: "input" %>
Slide 209
Slide 209 text
An advanced HTML-aware ERB
parser could prove super helpful
Slide 210
Slide 210 text
Stimulus LSP
Stimulus Lint
Turbo LSP
Ruby LSP
Slide 211
Slide 211 text
html_press gem
phlexing gem
deface gem
better_html gem
erb_lint gem
Probably even more
Slide 212
Slide 212 text
HTML+ERB Formatter
HTML+ERB Linter
A new ERB Engine?
Possibly even more?
Slide 213
Slide 213 text
Prism has a big effect on Ruby
internals and the tooling landscape
Slide 214
Slide 214 text
No content
Slide 215
Slide 215 text
An HTML-aware ERB parser could
have a similar effect for HTML
template tooling
Slide 216
Slide 216 text
In Progress
Hotwire Browser Extension
Slide 217
Slide 217 text
No content
Slide 218
Slide 218 text
No content
Slide 219
Slide 219 text
No content
Slide 220
Slide 220 text
No content
Slide 221
Slide 221 text
Highlight Stimulus Controllers & Turbo Frames
Slide 222
Slide 222 text
Sneak Peak
Slide 223
Slide 223 text
No content
Slide 224
Slide 224 text
No content
Slide 225
Slide 225 text
No content
Slide 226
Slide 226 text
Turbo Streams Debug Bar
Pause Streams Processing
Step through Streams
Revert Turbo Stream Actions
Show Diff of what an action changed
and moreβ¦
Slide 227
Slide 227 text
Turbo Frames Reload Button
R
Slide 228
Slide 228 text
Turbo Morphing Debug
Turbo Morphing Visual Diffs
Turbo Form Submission Debug
Turbo Streams Header Debug
ActionCable / Turbo Rails Cable Debug
Turbo Frame Lazy Loading Debug
Stimulus Controller Tree-View Panel
β¦
Ideas & Roadmap
Slide 229
Slide 229 text
It
β
s a work in progress
Slide 230
Slide 230 text
Let us know if you have any ideas,
we are just getting started
Slide 231
Slide 231 text
Conclusion
Slide 232
Slide 232 text
Developer Experience and
Developer Ergonomics matter.
Slide 233
Slide 233 text
This is the kind of tooling we
are currently missing.
Slide 234
Slide 234 text
There is so much potential
to level up even more
Slide 235
Slide 235 text
I am super excited for the
future of these tools.
Slide 236
Slide 236 text
I believe that tools like these are
part of the reason why a language/
framework keeps it
β
s relevancy
Slide 237
Slide 237 text
Or put differently
Slide 238
Slide 238 text
These tools are necessary and
expected from a modern
language and framework
Slide 239
Slide 239 text
We have been
lacking behind
Slide 240
Slide 240 text
X
Slide 241
Slide 241 text
I want to keep building with
Ruby and Rails
Slide 242
Slide 242 text
I want to help build the tools
needed for the future of Rails
applications.
Slide 243
Slide 243 text
WEEKLY
Slide 244
Slide 244 text
Hotwire Weekly
β’ A new Hotwire-focused newsletter
β’ Delivered Weekly
β’ Explore what
β
s happening in the world of Hotwire
β’ Progress and updates while we are building out hotwire.io
β’ Been running since Oct 2023
β’ Over 1550+ subscribers and growing
WEEKLY
Slide 245
Slide 245 text
t
@Hotwire_Weekly
M
@[email protected]
Hotwire Weekly
Sign up on hotwireweekly.com
Slide 246
Slide 246 text
A more productive Hotwire
ecosystem.
Slide 247
Slide 247 text
A more unified Hotwire
ecosystem.
Slide 248
Slide 248 text
Feedback Ideas Thoughts
Improvements
Come
fi
nd me!
Slide 249
Slide 249 text
Stickers
Slide 250
Slide 250 text
Thank you π
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth