Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
π Marco Roth
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth
Full-Stack Developer & Open Source Contributor
Slide 3
Slide 3 text
Developer Tooling For The Modern
Hotwire & Rails Era
Marco Roth
Full-Stack Developer & Open Source Contributor
Slide 4
Slide 4 text
CableReady
Core Team
Maintainer
Slide 5
Slide 5 text
I β€ Open Source
Slide 6
Slide 6 text
Open Source Journey
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
Started to actively contribute in 2019
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Went full-time on OSS in January 2022
Slide 11
Slide 11 text
To work on Hotwire
and related projects
Slide 12
Slide 12 text
I want to improve Hotwire so
companies don
β
t need to use
React
Slide 13
Slide 13 text
While working on the
React applications
Slide 14
Slide 14 text
I noticed something
Slide 15
Slide 15 text
The tooling and DX is
awesome
Slide 16
Slide 16 text
We didn
β
t really have anything
like it in the Ruby world
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
I started to miss these tools
while working in Ruby/Rails
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
X
Slide 22
Slide 22 text
Superpower
β‘
Slide 23
Slide 23 text
The future of modern
Rails apps
Slide 24
Slide 24 text
I want to help build the
modern tooling needed
Slide 25
Slide 25 text
No IDE
Slide 26
Slide 26 text
They felt clunky, heavyweight
and cumbersome
Slide 27
Slide 27 text
Made me less productive
Slide 28
Slide 28 text
But this changed in the last
few years
Slide 29
Slide 29 text
Editor Evolution
Slide 30
Slide 30 text
Language Server Protocol
(LSP)
Slide 31
Slide 31 text
What is it and
why do we want to use it?
Slide 32
Slide 32 text
Language Intelligence
without the need for an IDE
Slide 33
Slide 33 text
Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
Slide 34
Slide 34 text
How does it work?
Slide 35
Slide 35 text
Your editor starts a one
background process per
language
Slide 36
Slide 36 text
Communicates over
STDIN/STDOUT or TCP/IP
Slide 37
Slide 37 text
JSON-RPC
Slide 38
Slide 38 text
Similar to HTTP
Slide 39
Slide 39 text
Content-Length: ...\r\n
\r\n
{
"jsonrpc": "2.0",
"id": 1,
"method": "textDocument/completion",
"params": {
...
}
}
Slide 40
Slide 40 text
Request Message
Response Message
Notification Message
Slide 41
Slide 41 text
Notification Messages
Slide 42
Slide 42 text
Text Document
Synchronization
Slide 43
Slide 43 text
textDocument/didOpen
textDocument/didChange
textDocument/didClose
textDocument/didSave
textDocument/publishDiagnostics
...
Slide 44
Slide 44 text
Source: https://microsoft.github.io/language-server-protocol/overviews/lsp/overview
Editor /
Client
Language
Server
Slide 45
Slide 45 text
Stimulus LSP
What are we trying to solve?
Slide 46
Slide 46 text
Greet
Slide 47
Slide 47 text
Greet
Slide 48
Slide 48 text
Slide 49
Slide 49 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 50
Slide 50 text
Slide 51
Slide 51 text
Slide 52
Slide 52 text
Slide 53
Slide 53 text
Slide 54
Slide 54 text
These mistakes
happen so easily
Slide 55
Slide 55 text
I thought:
"We can do better!"
Slide 56
Slide 56 text
Jun 23, 2021
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
Let
β
s take a look
how it works
Slide 62
Slide 62 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 63
Slide 63 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 64
Slide 64 text
{
id: 1,
method: "textDocument/completion",
params: {
"textDocument": {
"uri": "file:///.../lsp-example/.../test.txt"
},
"position": {
"line": 1,
"character": 7
},
"context": {
"triggerKind": 1
}
}
}
Slide 65
Slide 65 text
{
id: 1,
result: [
{
"label": "TypeScript",
"kind": 1,
"data": 1
},
{
"label": "JavaScript",
"kind": 1,
"data": 2
}
]
}
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
Greet
Slide 68
Slide 68 text
Embedded Languages
Slide 69
Slide 69 text
The Stimulus grammar is not
part of the HTML spec
Slide 70
Slide 70 text
So we need to extend the
handling for HTML/JavaScript
Slide 71
Slide 71 text
Language services
Slide 72
Slide 72 text
No content
Slide 73
Slide 73 text
Data providers
Slide 74
Slide 74 text
import { IHTMLDataProvider } from "vscode-html-languageservice"
export class StimulusHTMLDataProvider implements IHTMLDataProvider {
provideTags() {
...
}
provideAttributes(tag: string) {
...
}
provideValues(tag: string, attribute: string) {
...
}
}
Slide 75
Slide 75 text
Which language do
you write it in?
Slide 76
Slide 76 text
No content
Slide 77
Slide 77 text
Ruby LSP plugin?
Slide 78
Slide 78 text
No content
Slide 79
Slide 79 text
Existing Data Providers
Portability
Documentation/Integration
Existing Language Services
Slide 80
Slide 80 text
import { IHTMLDataProvider } from "vscode-html-languageservice"
export class StimulusHTMLDataProvider implements IHTMLDataProvider {
provideTags() {
...
}
provideAttributes(tag: string) {
...
}
provideValues(tag: string, attribute: string) {
...
}
}
Slide 81
Slide 81 text
Let
β
s implement it
Slide 82
Slide 82 text
provideTags() {
}
Slide 83
Slide 83 text
provideTags() {
return []
}
Slide 84
Slide 84 text
provideAttributes(tag: string) {
}
Slide 85
Slide 85 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 86
Slide 86 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 87
Slide 87 text
provideAttributes(tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 88
Slide 88 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 89
Slide 89 text
Parse JavaScript
Slide 90
Slide 90 text
Parse Stimulus Controllers
Slide 91
Slide 91 text
class ControllerDefinition {
readonly path: string
methods: Array = []
targets: Array = []
classes: Array = []
values: { [key: string]: Value } = {}
}
Slide 92
Slide 92 text
Acorn
Slide 93
Slide 93 text
import { Parser as AcornParser } from "acorn"
import { simple as walk } from "acorn-walk"
export class Parser {
parseController(code: string, filename: string) {
...
}
}
Slide 94
Slide 94 text
f
app/javascript/controllers/hello_controller.js
Slide 95
Slide 95 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
console.log("Connect")
}
greet() {
this.outputTarget.textContent =
`Hello ${this.nameTarget.value}`
}
disconnect() {
console.log("Disconnect")
}
}
Slide 96
Slide 96 text
export default class extends Controller {
static targets = ["name", "output"]
connect() {
console.log("Connect")
}
greet() {
this.outputTarget.textContent =
`Hello ${this.nameTarget.value}`
}
disconnect() {
console.log("Disconnect")
}
}
Slide 97
Slide 97 text
const pattern = "app/javascript/controllers/**/*_controller.js"
const controllerFiles = await glob(pattern)
controllerFiles.forEach(async path => {
const code = await fs.readFile(path, "utf8")
parser.parseController(code, path)
})
Slide 98
Slide 98 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 99
Slide 99 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 100
Slide 100 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 101
Slide 101 text
parseController(code: string, filename: string) {
const ast = this.parse(code)
const controller = new ControllerDefinition(filename)
walk(ast, {
MethodDefinition(node) {
if (node.kind === "method") {
controller.methods.push(node.key.name)
}
},
})
}
Slide 102
Slide 102 text
We do something similar for
the Targets
Slide 103
Slide 103 text
Slide 104
Slide 104 text
data-controller
data-action
data-[identifier]-target
data-[identifier]-[class]-class
data-[identifier]-[value]-value
data-[identifier]-[outlet]-outlet
Slide 105
Slide 105 text
provideAttributes(tag: string) {
return [
{ name: "data-controller" },
{ name: "data-action" },
]
}
Slide 106
Slide 106 text
provideAttributes(tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 107
Slide 107 text
provideAttributes(tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 108
Slide 108 text
provideAttributes(tag: string) {
const targets = controllers.map(controller =>
`data-${controller.identifier}-target`
)
return [
{ name: "data-controller" },
{ name: "data-action" },
...targets,
]
}
Slide 109
Slide 109 text
No content
Slide 110
Slide 110 text
Slide 111
Slide 111 text
provideValues(_tag: string, attribute: string) {
}
Slide 112
Slide 112 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 113
Slide 113 text
provideValues(_tag: string, attribute: string) {
if (attribute === "data-controller") {
return this.controllers.map(controller =>
controller.identifier
)
}
const match = attribute.match(/data-(.+)-target/)
const controller = this.controllers.find(controller =>
controller.identifier == match[1]
)
return controller.targets
}
Slide 114
Slide 114 text
No content
Slide 115
Slide 115 text
https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
Slide 116
Slide 116 text
Jun 25, 2021
Slide 117
Slide 117 text
No content
Slide 118
Slide 118 text
What else can we do?
Slide 119
Slide 119 text
Diagnostics
Slide 120
Slide 120 text
No content
Slide 121
Slide 121 text
No content
Slide 122
Slide 122 text
No content
Slide 123
Slide 123 text
No content
Slide 124
Slide 124 text
No content
Slide 125
Slide 125 text
What else can we do?
Slide 126
Slide 126 text
Code Actions to
guide the user in the right
direction
Slide 127
Slide 127 text
No content
Slide 128
Slide 128 text
After a few months
Slide 129
Slide 129 text
Available Now
Stimulus LSP
Slide 130
Slide 130 text
I asked the community for
feedback
Slide 131
Slide 131 text
And got a lot of awesome
feedback and ideas
Slide 132
Slide 132 text
Some of the naive approaches
we used were starting to fall
short
Slide 133
Slide 133 text
Especially the way we were parsing
the JavaScript with Acorn
Slide 134
Slide 134 text
We hard-coded
app/javascript/controllers/
Slide 135
Slide 135 text
The controller identifiers
weren
β
t always accurate
Slide 136
Slide 136 text
Inferring the identifier from the
filename is not always enough
Slide 137
Slide 137 text
import { application } from "./application"
import ClipboardController from "./clipboard_controller.js"
application.register("clipboard", ClipboardController)
import FileUploadController from "./file_upload_controller.js"
application.register("file-upload", FileUploadController)
f
app/javascript/controllers/index.js
Slide 138
Slide 138 text
import { application } from "./application"
import ClipboardController from "./clipboard_controller.js"
application.register("clipboard", ClipboardController)
import FileUploadController from "./file_upload_controller.js"
application.register("file-upload", FileUploadController)
f
app/javascript/controllers/index.js
Slide 139
Slide 139 text
import { application } from "controllers/application"
import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading"
eagerLoadControllersFrom("controllers", application)
Slide 140
Slide 140 text
No content
Slide 141
Slide 141 text
The Language Server works
Slide 142
Slide 142 text
But produced a lot of false
positives and inaccurate results
Slide 143
Slide 143 text
Which is really
annoying as a user
Slide 144
Slide 144 text
We addressed and
fixed a lot of issues
Slide 145
Slide 145 text
Which technically
ended up in a rewrite
Slide 146
Slide 146 text
No content
Slide 147
Slide 147 text
No content
Slide 148
Slide 148 text
What
β
s next?
Slide 149
Slide 149 text
Coming Soon
Stimulus Lint
Slide 150
Slide 150 text
No content
Slide 151
Slide 151 text
None of is this is
exclusive to the LSP
Slide 152
Slide 152 text
The diagnostics could
be used independently
Slide 153
Slide 153 text
RuboCop is also
an independent tool
Slide 154
Slide 154 text
Errors
Diagnostics
Best Practices
Recommendations
Slide 155
Slide 155 text
$ stimulus-lint
Stimulus Lint is inspecting 25 files
........................
25 files inspected, 0 offenses detected, 0 offenses autocorrectable
Slide 156
Slide 156 text
<%= tag.div(
data: {
controller: "filter",
filter_open_class: "border-white",
filter_close_class: "hover:bg-gray-100 border-gray-300"
}
) %>
Full support for HTML+ERB and Rails-specific helpers
Slide 157
Slide 157 text
Code Completion
Diagnostics
Code Actions
Refactoring Tools
Slide 158
Slide 158 text
Pre-Release
Turbo LSP
Slide 159
Slide 159 text
No content
Slide 160
Slide 160 text
Turbo also makes heavy use
of ERB and view helpers
Slide 161
Slide 161 text
So it
β
s only really useful if we
can provide intelligence
Slide 162
Slide 162 text
And that only works if
we actually understand ERB
Slide 163
Slide 163 text
Stimulus LSP understands
HTML in ERB documents
Slide 164
Slide 164 text
But it doesn
β
t understand
Embedded Ruby that outputs
HTML
Slide 165
Slide 165 text
ERB Support in Ruby LSP
Slide 166
Slide 166 text
No content
Slide 167
Slide 167 text
We quickly realized that we need
something more powerful to make
things work beyond the basics
Slide 168
Slide 168 text
<%= @user.firstname %>
<%= @user.lastname %>
Slide 169
Slide 169 text
βββββ
ββββββββββ @user.firstname βββββββββ
ββββββ @user.lastname ββββββ
ββββββ
Slide 170
Slide 170 text
@user.firstname
@user.lastname
Slide 171
Slide 171 text
Source: https://github.com/Shopify/ruby-lsp/pull/2235
Slide 172
Slide 172 text
Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
Slide 173
Slide 173 text
But we lost the whole
HTML-context in the process
Slide 174
Slide 174 text
So I started to dabble and explore what
would be possible with a new
HTML-aware ERB parser
Slide 175
Slide 175 text
The idea is to power the ERB support with
the new HTML-aware parser in these tools
Stimulus LSP
Turbo LSP
Stimulus Lint
Ruby LSP (maybe?)
Slide 176
Slide 176 text
type="text" <% end %> />
<% @posts.each do |post| %>
<%= post.title %>
<% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>Slide 177
Slide 177 text
Which made me think that it
could be useful to have an
HTML-AST
Slide 178
Slide 178 text
In which the ERB view helpers
also have an HTML tag node
for Rails View Helpers
Slide 179
Slide 179 text
<%= tag.p do %>
Hello world!
<% end %>
Hello world!
==Slide 180
Slide 180 text
This would also open to door to support
Haml, Slim, Liquid, Blade and other
template languages
Slide 181
Slide 181 text
If we can transform their
syntax into the same AST
Slide 182
Slide 182 text
Then we could build the whole
logic for the Stimulus LSP on
top of that AST
Slide 183
Slide 183 text
And we don
β
t really need to
care what templating language
it actually is
Slide 184
Slide 184 text
If this is done right it could also help with
other tooling such as linters, formatters,
and code navigation tools.
Slide 185
Slide 185 text
erb-languageservice
html-templating-languageservice
Slide 186
Slide 186 text
If the parser can understand any ERB
context and Rails-render calls it is super
powerful
Slide 187
Slide 187 text
This could power advanced
diagnostics
Slide 188
Slide 188 text
Slide 189
Slide 189 text
Slide 190
Slide 190 text
<%= render partial: "input" %>
Slide 191
Slide 191 text
But if the parser understands the
render logic
Slide 192
Slide 192 text
<%= render partial: "input" %>
Slide 193
Slide 193 text
<%= render partial: "input" %>
Slide 194
Slide 194 text
Modern Rails apps are composed of
multiple views/partials/components
Slide 195
Slide 195 text
I think an advanced HTML-aware ERB
parser could proof super helpful
Slide 196
Slide 196 text
Stimulus LSP
Stimulus Lint
Turbo LSP
Ruby LSP
Slide 197
Slide 197 text
html_press gem
phlexing gem
deface gem
better_html gem
erb_lint gem
Slide 198
Slide 198 text
HTML+ERB formatter
HTML+ERB Linter
Phoenix LiveView HEEx-like
Template Engine for ERB
Slide 199
Slide 199 text
No content
Slide 200
Slide 200 text
This would be super useful for modern
real-time applications powered by
Hotwire
Slide 201
Slide 201 text
I am not sure how feasible it is, but the
ERB parser would make it a lot more
approachable.
Slide 202
Slide 202 text
Prism has a big effect on Ruby
internals and the tooling landscape
Slide 203
Slide 203 text
An HTML-aware ERB parser could
have a similar effect for HTML
template tooling
Slide 204
Slide 204 text
In Progress
Hotwire Browser Extension
Slide 205
Slide 205 text
No content
Slide 206
Slide 206 text
No content
Slide 207
Slide 207 text
No content
Slide 208
Slide 208 text
It
β
s a work in progress
Slide 209
Slide 209 text
No content
Slide 210
Slide 210 text
Highlight Stimulus Controllers & Turbo Frames
Slide 211
Slide 211 text
Turbo Event Logging
Slide 212
Slide 212 text
Sneak Peak
Slide 213
Slide 213 text
No content
Slide 214
Slide 214 text
No content
Slide 215
Slide 215 text
Similarly for Stimulus Targets
Slide 216
Slide 216 text
No content
Slide 217
Slide 217 text
Turbo Streams Debug Bar
Pause Streams Processing
Step through Streams
Revert Turbo Stream Actions
Show Diff of what an action changed
and moreβ¦
Slide 218
Slide 218 text
Turbo Morphing Debug
Turbo Morphing Visual Diffs
Turbo Form Submission Debug
Turbo Streams Header Debug
ActionCable / Turbo Rails Cable Debug
Turbo Frame Lazy Loading Debug
Stimulus Controller Tree-View Panel
β¦
Ideas & Roadmap
Slide 219
Slide 219 text
Let us know if you have any ideas,
we are just getting started
Slide 220
Slide 220 text
Recap
Slide 221
Slide 221 text
Developer Experience and
Developer Ergonomics matter.
Slide 222
Slide 222 text
This is the kind of tooling we
are currently missing.
Slide 223
Slide 223 text
There is so much potential!
Slide 224
Slide 224 text
I am super excited for the
future of these tools.
Slide 225
Slide 225 text
I believe that tools like this are part of
the reason why a language/framework
keeps it
β
s relevancy
Slide 226
Slide 226 text
Or put differently
Slide 227
Slide 227 text
These tools are necessary and
expected from a modern
language and framework
Slide 228
Slide 228 text
X
Slide 229
Slide 229 text
I want to help build the tools
needed for the future of Rails
applications.
Slide 230
Slide 230 text
WEEKLY
Slide 231
Slide 231 text
Hotwire Weekly
β’ A new Hotwire-focused newsletter
β’ Delivered Weekly
β’ Explore what
β
s happening in the world of Hotwire
β’ Progress and updates while we are building out hotwire.io
β’ Been running since Oct 2023
β’ Over 1350+ subscribers and growing
WEEKLY
Slide 232
Slide 232 text
t
@Hotwire_Weekly
M
@[email protected]
Hotwire Weekly
Sign up on hotwireweekly.com
Slide 233
Slide 233 text
A more unified Hotwire
ecosystem
Slide 234
Slide 234 text
A more productive Hotwire
ecosystem
Slide 235
Slide 235 text
Feedback Ideas Thoughts
Improvements
Come
fi
nd me!
Slide 236
Slide 236 text
Stickers
Slide 237
Slide 237 text
Thank you π
t
@marcoroth_
M
@[email protected]
g
marcoroth.dev
g
@marcoroth