Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
Marco Roth 👋
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth
Full-Stack Developer & Open Source Contributor
Slide 3
Slide 3 text
Developer Tooling For The Modern
Hotwire & Rails Era
Marco Roth
Full-Stack Developer & Open Source Contributor
Slide 4
Slide 4 text
Understand LSPs
Building a simple LSP
What
’
s Next for Tooling
Slide 5
Slide 5 text
CableReady
Core Team
Maintainer
Slide 6
Slide 6 text
I ❤ Open Source
Slide 7
Slide 7 text
Open Source Journey
Slide 8
Slide 8 text
rubyvideo.dev
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
https://rubyvideo.dev/organisations/ruby-turkiye
X
Slide 13
Slide 13 text
Started to actively contribute in 2019
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
Had enough
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
X
Slide 19
Slide 19 text
Superpower
⚡
Slide 20
Slide 20 text
The future of modern
Rails apps
Slide 21
Slide 21 text
I wanted to go back
doing Full-Stack Rails
Slide 22
Slide 22 text
I want to help build the
missing modern tooling
Slide 23
Slide 23 text
Went full-time on OSS from
Jan 2022 - Aug 2023
Slide 24
Slide 24 text
To work on Hotwire
and related projects
Slide 25
Slide 25 text
But, I noticed something
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
The JavaScript tooling
and DX is awesome
Slide 28
Slide 28 text
We didn
’
t really have anything
like it in the Ruby world
Slide 29
Slide 29 text
Or something quite as easy to
setup and get started
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
I started to miss these tools
while working in Ruby/Rails
Slide 32
Slide 32 text
No IDE
Slide 33
Slide 33 text
They felt clunky, heavyweight
and cumbersome to use
Slide 34
Slide 34 text
Made me
less productive
Slide 35
Slide 35 text
But this changed
in the last few years
Slide 36
Slide 36 text
Editor Evolution
Slide 37
Slide 37 text
Language Server Protocol
(LSP)
Slide 38
Slide 38 text
What is it and
why do we want to use it?
Slide 39
Slide 39 text
Language Intelligence
without the need for an IDE
Slide 40
Slide 40 text
Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
Slide 41
Slide 41 text
How does it work?
Slide 42
Slide 42 text
Your editor starts a
background process per
language
Slide 43
Slide 43 text
Communicates over
STDIN/STDOUT or TCP/IP
Slide 44
Slide 44 text
JSON-RPC
Slide 45
Slide 45 text
Similar to HTTP
Slide 46
Slide 46 text
Content-Length: ...\r\n
\r\n
{
"jsonrpc": "2.0",
"id": 1,
"method": "textDocument/completion",
"params": {
...
}
}
Slide 47
Slide 47 text
Request Message
Response Message
Notification Message
Slide 48
Slide 48 text
Text Document
Synchronization
Slide 49
Slide 49 text
textDocument/didOpen
textDocument/didChange
textDocument/didClose
textDocument/didSave
textDocument/publishDiagnostics
...
Slide 50
Slide 50 text
Stimulus LSP
Slide 51
Slide 51 text
What are we trying to solve?
Slide 52
Slide 52 text
Greet
Slide 53
Slide 53 text
Greet
Slide 54
Slide 54 text
Slide 55
Slide 55 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 56
Slide 56 text
Slide 57
Slide 57 text
Slide 58
Slide 58 text
Slide 59
Slide 59 text
These mistakes
happen so easily
Slide 60
Slide 60 text
I thought:
"We can do better!"
Slide 61
Slide 61 text
Jun 23, 2021
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
Let
’
s take a look
how it works
Slide 67
Slide 67 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 68
Slide 68 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 69
Slide 69 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 70
Slide 70 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 71
Slide 71 text
The server returns
a JSON response
Slide 72
Slide 72 text
{
id: 1,
result: [
{
"label": "TypeScript",
"kind": 1,
"data": 1
},
{
"label": "JavaScript",
"kind": 1,
"data": 2
}
]
}
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
Greet
Slide 75
Slide 75 text
Embedded Languages
Slide 76
Slide 76 text
The Stimulus grammar is not
part of the HTML spec
Slide 77
Slide 77 text
So we need to extend the handling
for HTML-like documents
Slide 78
Slide 78 text
Language services
Slide 79
Slide 79 text
Which language do
you write it in?
Slide 80
Slide 80 text
No content
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
Existing Data Providers
Portability
Documentation/Integration
Existing Language Services
Slide 83
Slide 83 text
No content
Slide 84
Slide 84 text
Data Providers
Slide 85
Slide 85 text
import { IHTMLDataProvider } from "vscode-html-languageservice"
export class StimulusHTMLDataProvider implements IHTMLDataProvider {
provideTags() {
...
}
provideAttributes(tag: string) {
...
}
provideValues(tag: string, attribute: string) {
...
}
}
Slide 86
Slide 86 text
Let
’
s implement it
Slide 87
Slide 87 text
provideTags() {
}
Slide 88
Slide 88 text
...
Slide 89
Slide 89 text
provideTags() {
}
Slide 90
Slide 90 text
provideTags() {
return []
}
Slide 91
Slide 91 text
provideAttributes(tag: string) {
}
Slide 92
Slide 92 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 93
Slide 93 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 94
Slide 94 text
provideAttributes(_tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 95
Slide 95 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 96
Slide 96 text
Parse JavaScript
Slide 97
Slide 97 text
Parse Stimulus Controllers
Slide 98
Slide 98 text
class ControllerDefinition {
readonly path: string
methods: Array = []
targets: Array = []
classes: Array = []
values: { [key: string]: Value } = {}
}
Slide 99
Slide 99 text
Acorn
Slide 100
Slide 100 text
import { Parser as AcornParser } from "acorn"
import { simple as walk } from "acorn-walk"
class Parser {
parseController(code: string, filename: string) {
...
}
}
Slide 101
Slide 101 text
f
app/javascript/controllers/hello_controller.js
Slide 102
Slide 102 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 103
Slide 103 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 104
Slide 104 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 105
Slide 105 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 106
Slide 106 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 107
Slide 107 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 108
Slide 108 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 109
Slide 109 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 110
Slide 110 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 111
Slide 111 text
We do something similar for
the Targets
Slide 112
Slide 112 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
...
}
greet() {
...
}
disconnect() {
...
}
}
Slide 113
Slide 113 text
Slide 114
Slide 114 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 115
Slide 115 text
provideAttributes(_tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 116
Slide 116 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 117
Slide 117 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 118
Slide 118 text
provideAttributes(_tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 119
Slide 119 text
No content
Slide 120
Slide 120 text
Slide 121
Slide 121 text
provideValues(_tag: string, attribute: string) {
}
Slide 122
Slide 122 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 123
Slide 123 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 124
Slide 124 text
No content
Slide 125
Slide 125 text
https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
Slide 126
Slide 126 text
Jun 25, 2021
Slide 127
Slide 127 text
No content
Slide 128
Slide 128 text
What else can we do?
Slide 129
Slide 129 text
Can
’
t AI do all of that?
Slide 130
Slide 130 text
Maybe
Slide 131
Slide 131 text
Different Use Cases
Slide 132
Slide 132 text
AI for "boilerplate"
LSPs for reliability and accuracy
Slide 133
Slide 133 text
Limited Context Window
Missing relation between
relevant files
Slide 134
Slide 134 text
Diagnostics
Slide 135
Slide 135 text
No content
Slide 136
Slide 136 text
No content
Slide 137
Slide 137 text
No content
Slide 138
Slide 138 text
No content
Slide 139
Slide 139 text
No content
Slide 140
Slide 140 text
What else can we do?
Slide 141
Slide 141 text
Code Actions to
guide the user in the right
direction
Slide 142
Slide 142 text
No content
Slide 143
Slide 143 text
Fast forward to
October 2023
Slide 144
Slide 144 text
Available Now
Stimulus LSP
Slide 145
Slide 145 text
I asked the community for
feedback
Slide 146
Slide 146 text
And got a lot of awesome
feedback and ideas
Slide 147
Slide 147 text
Some of the naive approaches
we used were starting to fall
short
Slide 148
Slide 148 text
Especially the way we were parsing
the JavaScript with Acorn
Slide 149
Slide 149 text
We assumed and hard-coded
app/javascript/controllers/
Slide 150
Slide 150 text
The controller identifiers
weren
’
t always accurate
Slide 151
Slide 151 text
import { application } from "controllers/application"
import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading"
eagerLoadControllersFrom("controllers", application)
Slide 152
Slide 152 text
No content
Slide 153
Slide 153 text
The Language Server works
Slide 154
Slide 154 text
but it produced a lot of false
positives and inaccurate results
Slide 155
Slide 155 text
Which is really
annoying as a user
Slide 156
Slide 156 text
We addressed and
fixed a lot of issues
Slide 157
Slide 157 text
Which technically ended up
in a rewrite of the parser
Slide 158
Slide 158 text
Fast forward to
May 2024
Slide 159
Slide 159 text
No content
Slide 160
Slide 160 text
No content
Slide 161
Slide 161 text
What
’
s next?
Slide 162
Slide 162 text
No content
Slide 163
Slide 163 text
No content
Slide 164
Slide 164 text
None of is this is
exclusive to the LSP
Slide 165
Slide 165 text
The diagnostics could
be used independently
Slide 166
Slide 166 text
Kind of like an independent
tool as RuboCop
Slide 167
Slide 167 text
Coming Soon
Stimulus Lint
Slide 168
Slide 168 text
Errors
Diagnostics
Best Practices
Recommendations
Slide 169
Slide 169 text
$ stimulus-lint
Stimulus Lint is inspecting 25 files
........................
25 files inspected, 0 offenses detected, 0 offenses autocorrectable
Slide 170
Slide 170 text
<%= tag.div(
data: {
controller: "filter",
filter_open_class: "border-white",
filter_close_class: "hover:bg-gray-100 border-gray-300"
}
) %>
Full support for HTML+ERB and Rails-specific helpers
Slide 171
Slide 171 text
Diagnostics
Code Actions
Code Completion
Refactoring Tools
Slide 172
Slide 172 text
Pre-Release
Turbo LSP
Slide 173
Slide 173 text
No content
Slide 174
Slide 174 text
Super Basic right now
Slide 175
Slide 175 text
Turbo also makes heavy use
of ERB and view helpers
Slide 176
Slide 176 text
So it
’
s only really useful if we
can provide intelligence for ERB
Slide 177
Slide 177 text
And that only works if
we actually understand ERB
Slide 178
Slide 178 text
Stimulus LSP understands
HTML in ERB documents
Slide 179
Slide 179 text
But it doesn
’
t understand
Embedded Ruby that outputs
HTML
Slide 180
Slide 180 text
No content
Slide 181
Slide 181 text
No content
Slide 182
Slide 182 text
Source: https://twitter.com/vinistock/status/1710280529889620038
Slide 183
Slide 183 text
ERB Support in Ruby LSP
Slide 184
Slide 184 text
RubyConf 2023
Hack Day
Slide 185
Slide 185 text
No content
Slide 186
Slide 186 text
We quickly realized that we need
something more powerful to make
things work beyond the basics
Slide 187
Slide 187 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 188
Slide 188 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 189
Slide 189 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 190
Slide 190 text
Source: https://github.com/Shopify/ruby-lsp/pull/2235
Slide 191
Slide 191 text
Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
Slide 192
Slide 192 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 193
Slide 193 text
We lost the HTML-context
in the process
Slide 194
Slide 194 text
<%= "Hello World" %>
>
Slide 195
Slide 195 text
HTML with embedded Ruby
(.html.erb, .rhtml, …)
Slide 196
Slide 196 text
ERB is versatile
Slide 197
Slide 197 text
a
Slide 198
Slide 198 text
Experiment
HTML-aware ERB Parser
Slide 199
Slide 199 text
Stimulus LSP
Stimulus Lint
Turbo LSP
Ruby LSP (maybe?)
and more…
Slide 200
Slide 200 text
type="text" <% end %> />
<% @posts.each do |post| %>
<%= post.title %>
<% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>Slide 201
Slide 201 text
Which made me think that it
could be useful to have an
HTML abstraction
Slide 202
Slide 202 text
<%= tag.p do %>
Hello World!
<% end %>
Hello World!
==Slide 203
Slide 203 text
==
<%= tag.div(
data: {
controller: "hello",
action: "click->hello#greet"
}
) %>
Slide 204
Slide 204 text
Needs more exploration
Slide 205
Slide 205 text
This might also open to door to
support Haml, Slim, Liquid, Phlex,
Blade and other template
languages
Slide 206
Slide 206 text
erb-languageservice
html-templating-languageservice
Slide 207
Slide 207 text
If the parser can understand any
ERB context and Rails-render
calls it is super powerful
Slide 208
Slide 208 text
This could power advanced
diagnostics
Slide 209
Slide 209 text
Slide 210
Slide 210 text
Slide 211
Slide 211 text
Slide 212
Slide 212 text
Slide 213
Slide 213 text
Modern Rails apps are composed of
multiple views/partials/components
Slide 214
Slide 214 text
<%= render partial: "input" %>
Slide 215
Slide 215 text
But if the parser understands
the Rails render logic
Slide 216
Slide 216 text
<%= render partial: "input" %>
Slide 217
Slide 217 text
<%= render partial: "input" %>
Slide 218
Slide 218 text
An advanced HTML-aware ERB
parser could prove super helpful
Slide 219
Slide 219 text
Stimulus LSP
Stimulus Lint
Turbo LSP
Ruby LSP
Slide 220
Slide 220 text
html_press gem
phlexing gem
deface gem
better_html gem
erb_lint gem
Probably even more
Slide 221
Slide 221 text
HTML+ERB Formatter
HTML+ERB Linter
A new ERB Engine?
Possibly even more?
Slide 222
Slide 222 text
Prism has a big effect on Ruby
internals and the tooling landscape
Slide 223
Slide 223 text
No content
Slide 224
Slide 224 text
An HTML-aware ERB parser could
have a similar effect for HTML
template tooling
Slide 225
Slide 225 text
In Progress
Hotwire Browser Extension
Slide 226
Slide 226 text
No content
Slide 227
Slide 227 text
No content
Slide 228
Slide 228 text
No content
Slide 229
Slide 229 text
No content
Slide 230
Slide 230 text
Highlight Stimulus Controllers & Turbo Frames
Slide 231
Slide 231 text
Highlight Stimulus Controllers & Turbo Frames
Slide 232
Slide 232 text
Turbo Event Logging
Slide 233
Slide 233 text
Sneak Peak
Slide 234
Slide 234 text
No content
Slide 235
Slide 235 text
No content
Slide 236
Slide 236 text
No content
Slide 237
Slide 237 text
Turbo Streams Debug Bar
Pause Streams Processing
Step through Streams
Revert Turbo Stream Actions
Show Diff of what an action changed
and more…
Slide 238
Slide 238 text
Turbo Frames Reload Button
R
Slide 239
Slide 239 text
Turbo Morphing Debug
Turbo Morphing Visual Diffs
Turbo Form Submission Debug
Turbo Streams Header Debug
ActionCable / Turbo Rails Cable Debug
Turbo Frame Lazy Loading Debug
Stimulus Controller Tree-View Panel
…
Ideas & Roadmap
Slide 240
Slide 240 text
It
’
s a work in progress
Slide 241
Slide 241 text
Let us know if you have any ideas,
we are just getting started
Slide 242
Slide 242 text
Conclusion
Slide 243
Slide 243 text
Developer Experience and
Developer Ergonomics matter.
Slide 244
Slide 244 text
This is the kind of tooling we
are currently missing.
Slide 245
Slide 245 text
There is so much potential
to level up even more
Slide 246
Slide 246 text
I am super excited for the
future of these tools.
Slide 247
Slide 247 text
I believe that tools like these are
part of the reason why a language/
framework keeps it
’
s relevancy
Slide 248
Slide 248 text
Or put differently
Slide 249
Slide 249 text
These tools are necessary and
expected from a modern
language and framework
Slide 250
Slide 250 text
We have been
lacking behind
Slide 251
Slide 251 text
X
Slide 252
Slide 252 text
I want to keep building with
Ruby and Rails
Slide 253
Slide 253 text
I want to help build the tools
needed for the future of Rails
applications.
Slide 254
Slide 254 text
WEEKLY
Slide 255
Slide 255 text
Hotwire Weekly
• A new Hotwire-focused newsletter
• Delivered Weekly
• Explore what
’
s happening in the world of Hotwire
• Progress and updates while we are building out hotwire.io
• Been running since Oct 2023
• Over 1600+ subscribers and growing
WEEKLY
Slide 256
Slide 256 text
No content
Slide 257
Slide 257 text
t
@Hotwire_Weekly
M
@[email protected]
Hotwire Weekly
Sign up on hotwireweekly.com
Slide 258
Slide 258 text
A more productive Hotwire
ecosystem.
Slide 259
Slide 259 text
A more unified Hotwire
ecosystem.
Slide 260
Slide 260 text
Thank you 🙏
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth