Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

👋 Marco Roth t @marcoroth_ M @[email protected] g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor

Slide 3

Slide 3 text

Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor

Slide 4

Slide 4 text

Understand LSPs Building a simple LSP What ’ s Next for Tooling

Slide 5

Slide 5 text

CableReady Core Team Maintainer

Slide 6

Slide 6 text

I ❤ Open Source

Slide 7

Slide 7 text

rubyvideo.dev

Slide 8

Slide 8 text

Started to actively contribute in 2019

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Had enough

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Superpower ⚡

Slide 15

Slide 15 text

The future of modern Rails apps

Slide 16

Slide 16 text

I want to help build the missing modern tooling

Slide 17

Slide 17 text

Went full-time on OSS from Jan 2022 - Aug 2023

Slide 18

Slide 18 text

To work on Hotwire and related projects

Slide 19

Slide 19 text

But, I noticed something

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

The JavaScript tooling and DX is awesome

Slide 22

Slide 22 text

We didn ’ t really have anything like it in the Ruby world

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

I started to miss these tools while working in Ruby/Rails

Slide 25

Slide 25 text

No IDE

Slide 26

Slide 26 text

They felt clunky, heavyweight and cumbersome to use

Slide 27

Slide 27 text

Made me less productive

Slide 28

Slide 28 text

But this changed in the last few years

Slide 29

Slide 29 text

Editor Evolution

Slide 30

Slide 30 text

Language Server Protocol (LSP)

Slide 31

Slide 31 text

What is it and why do we want to use it?

Slide 32

Slide 32 text

Language Intelligence without the need for an IDE

Slide 33

Slide 33 text

Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide

Slide 34

Slide 34 text

How does it work?

Slide 35

Slide 35 text

Your editor starts a background process per language

Slide 36

Slide 36 text

Communicates over STDIN/STDOUT or TCP/IP

Slide 37

Slide 37 text

JSON-RPC

Slide 38

Slide 38 text

Similar to HTTP

Slide 39

Slide 39 text

Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }

Slide 40

Slide 40 text

Stimulus LSP

Slide 41

Slide 41 text

What are we trying to solve?

Slide 42

Slide 42 text

Greet

Slide 43

Slide 43 text

Greet

Slide 44

Slide 44 text

Slide 45

Slide 45 text

data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

These mistakes happen so easily

Slide 50

Slide 50 text

I thought: "We can do better!"

Slide 51

Slide 51 text

Jun 23, 2021

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Let ’ s take a look how it works

Slide 57

Slide 57 text

{ id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }

Slide 58

Slide 58 text

{ id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }

Slide 59

Slide 59 text

{ id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }

Slide 60

Slide 60 text

{ id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }

Slide 61

Slide 61 text

The server returns a JSON response

Slide 62

Slide 62 text

{ id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

Greet

Slide 65

Slide 65 text

Embedded Languages

Slide 66

Slide 66 text

The Stimulus grammar is not part of the HTML spec

Slide 67

Slide 67 text

So we need to extend the handling for HTML-like documents

Slide 68

Slide 68 text

Language services

Slide 69

Slide 69 text

Which language do you write it in?

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

Existing Data Providers Portability Documentation/Integration Existing Language Services

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Data Providers

Slide 75

Slide 75 text

import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }

Slide 76

Slide 76 text

Let ’ s implement it

Slide 77

Slide 77 text

provideTags() { }

Slide 78

Slide 78 text

...

Slide 79

Slide 79 text

provideTags() { }

Slide 80

Slide 80 text

provideTags() { return [] }

Slide 81

Slide 81 text

provideAttributes(tag: string) { }

Slide 82

Slide 82 text

data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet

Slide 83

Slide 83 text

data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet

Slide 84

Slide 84 text

provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }

Slide 85

Slide 85 text

data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet

Slide 86

Slide 86 text

Parse JavaScript

Slide 87

Slide 87 text

Parse Stimulus Controllers

Slide 88

Slide 88 text

class ControllerDefinition { readonly path: string methods: Array = [] targets: Array = [] classes: Array = [] values: { [key: string]: Value } = {} }

Slide 89

Slide 89 text

Acorn

Slide 90

Slide 90 text

import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }

Slide 91

Slide 91 text

f app/javascript/controllers/hello_controller.js

Slide 92

Slide 92 text

export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }

Slide 93

Slide 93 text

export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }

Slide 94

Slide 94 text

const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })

Slide 95

Slide 95 text

Slide 96

Slide 96 text

Slide 97

Slide 97 text

parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }

Slide 98

Slide 98 text

Slide 99

Slide 99 text

Slide 100

Slide 100 text

Slide 101

Slide 101 text

We do something similar for the Targets

Slide 102

Slide 102 text

export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }

Slide 103

Slide 103 text

Slide 104

Slide 104 text

data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet

Slide 105

Slide 105 text

provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }

Slide 106

Slide 106 text

provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }

Slide 107

Slide 107 text

provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }

Slide 108

Slide 108 text

provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }

Slide 109

Slide 109 text

No content

Slide 110

Slide 110 text

Slide 111

Slide 111 text

provideValues(_tag: string, attribute: string) { }

Slide 112

Slide 112 text

provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }

Slide 113

Slide 113 text

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2

Slide 116

Slide 116 text

Jun 25, 2021

Slide 117

Slide 117 text

No content

Slide 118

Slide 118 text

What else can we do?

Slide 119

Slide 119 text

Can ’ t AI do all of that?

Slide 120

Slide 120 text

Maybe

Slide 121

Slide 121 text

Different Use Cases

Slide 122

Slide 122 text

AI for "boilerplate" LSPs for reliability and accuracy

Slide 123

Slide 123 text

Limited Context Window Missing relation between relevant files

Slide 124

Slide 124 text

Diagnostics

Slide 125

Slide 125 text

No content

Slide 126

Slide 126 text

No content

Slide 127

Slide 127 text

No content

Slide 128

Slide 128 text

No content

Slide 129

Slide 129 text

No content

Slide 130

Slide 130 text

What else can we do?

Slide 131

Slide 131 text

Code Actions to guide the user in the right direction

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

Fast forward to October 2023

Slide 134

Slide 134 text

Available Now Stimulus LSP

Slide 135

Slide 135 text

I asked the community for feedback

Slide 136

Slide 136 text

And got a lot of awesome feedback and ideas

Slide 137

Slide 137 text

Some of the naive approaches we used were starting to fall short

Slide 138

Slide 138 text

Especially the way we were parsing the JavaScript with Acorn

Slide 139

Slide 139 text

We assumed and hard-coded app/javascript/controllers/

Slide 140

Slide 140 text

import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

The Language Server works

Slide 143

Slide 143 text

But produced a lot of false positives and inaccurate results

Slide 144

Slide 144 text

Which is really annoying as a user

Slide 145

Slide 145 text

We addressed and fixed a lot of issues

Slide 146

Slide 146 text

Which technically ended up in a rewrite of the parser

Slide 147

Slide 147 text

Fast forward to May 2024

Slide 148

Slide 148 text

No content

Slide 149

Slide 149 text

No content

Slide 150

Slide 150 text

What ’ s next?

Slide 151

Slide 151 text

No content

Slide 152

Slide 152 text

No content

Slide 153

Slide 153 text

None of is this is exclusive to the LSP

Slide 154

Slide 154 text

The diagnostics could be used independently

Slide 155

Slide 155 text

Kind of like RuboCop is an independent tool

Slide 156

Slide 156 text

Coming Soon Stimulus Lint

Slide 157

Slide 157 text

Errors Diagnostics Best Practices Recommendations

Slide 158

Slide 158 text

$ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable

Slide 159

Slide 159 text

<%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers

Slide 160

Slide 160 text

Diagnostics Code Actions Code Completion Refactoring Tools

Slide 161

Slide 161 text

Pre-Release Turbo LSP

Slide 162

Slide 162 text

No content

Slide 163

Slide 163 text

Turbo also makes heavy use of ERB and view helpers

Slide 164

Slide 164 text

So it ’ s only really useful if we can provide intelligence

Slide 165

Slide 165 text

And that only works if we actually understand ERB

Slide 166

Slide 166 text

Stimulus LSP understands HTML in ERB documents

Slide 167

Slide 167 text

But it doesn ’ t understand Embedded Ruby that outputs HTML

Slide 168

Slide 168 text

No content

Slide 169

Slide 169 text

No content

Slide 170

Slide 170 text

Source: https://twitter.com/vinistock/status/1710280529889620038

Slide 171

Slide 171 text

ERB Support in Ruby LSP

Slide 172

Slide 172 text

RubyConf 2023 Hack Day

Slide 173

Slide 173 text

No content

Slide 174

Slide 174 text

We quickly realized that we need something more powerful to make things work beyond the basics

Slide 175

Slide 175 text

<%= @user.firstname %> <%= @user.lastname %>

Slide 176

Slide 176 text

<%= @user.firstname %> <%= @user.lastname %>

Slide 177

Slide 177 text

<%= @user.firstname %> <%= @user.lastname %>

Slide 178

Slide 178 text

Source: https://github.com/Shopify/ruby-lsp/pull/2235

Slide 179

Slide 179 text

Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024

Slide 180

Slide 180 text

<%= @user.firstname %> <%= @user.lastname %>

Slide 181

Slide 181 text

We lost the HTML-context in the process

Slide 182

Slide 182 text

<%= "Hello World" %>

Slide 183

Slide 183 text

HTML with embedded Ruby (.html.erb, .rhtml, …)

Slide 184

Slide 184 text

ERB is versatile

Slide 185

Slide 185 text

Slide 186

Slide 186 text

Experiment HTML-aware ERB Parser

Slide 187

Slide 187 text

Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP (maybe?) and more…

Slide 188

Slide 188 text

type="text" <% end %> /> <% @posts.each do |post| %>

<%= post.title %>

<% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>

Slide 189

Slide 189 text

Which made me think that it could be useful to have an HTML abstraction

Slide 190

Slide 190 text

<%= tag.p do %> Hello World! <% end %>

Hello World!

Slide 191

Slide 191 text

== <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>

Slide 192

Slide 192 text

Needs more exploration

Slide 193

Slide 193 text

This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages

Slide 194

Slide 194 text

erb-languageservice html-templating-languageservice

Slide 195

Slide 195 text

If the parser can understand any ERB context and Rails-render calls it is super powerful

Slide 196

Slide 196 text

This could power advanced diagnostics

Slide 197

Slide 197 text

Slide 198

Slide 198 text

Slide 199

Slide 199 text

Modern Rails apps are composed of multiple views/partials/components

Slide 200

Slide 200 text

<%= render partial: "input" %>

Slide 201

Slide 201 text

But if the parser understands the Rails render logic

Slide 202

Slide 202 text

<%= render partial: "input" %>

Slide 203

Slide 203 text

<%= render partial: "input" %>

Slide 204

Slide 204 text

I think an advanced HTML-aware ERB parser could prove super helpful

Slide 205

Slide 205 text

Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP

Slide 206

Slide 206 text

html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more

Slide 207

Slide 207 text

HTML+ERB Formatter HTML+ERB Linter and more?

Slide 208

Slide 208 text

Prism has a big effect on Ruby internals and the tooling landscape

Slide 209

Slide 209 text

An HTML-aware ERB parser could have a similar effect for HTML template tooling

Slide 210

Slide 210 text

In Progress Hotwire Browser Extension

Slide 211

Slide 211 text

No content

Slide 212

Slide 212 text

No content

Slide 213

Slide 213 text

No content

Slide 214

Slide 214 text

No content

Slide 215

Slide 215 text

Highlight Stimulus Controllers & Turbo Frames

Slide 216

Slide 216 text

Sneak Peak

Slide 217

Slide 217 text

No content

Slide 218

Slide 218 text

No content

Slide 219

Slide 219 text

No content

Slide 220

Slide 220 text

Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…

Slide 221

Slide 221 text

Turbo Frames Reload Button R

Slide 222

Slide 222 text

Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap

Slide 223

Slide 223 text

It ’ s a work in progress

Slide 224

Slide 224 text

Let us know if you have any ideas, we are just getting started

Slide 225

Slide 225 text

Conclusion

Slide 226

Slide 226 text

Developer Experience and Developer Ergonomics matter.

Slide 227

Slide 227 text

This is the kind of tooling we are currently missing.

Slide 228

Slide 228 text

There is so much potential to level up

Slide 229

Slide 229 text

I am super excited for the future of these tools.

Slide 230

Slide 230 text

I believe that tools like these are part of the reason why a language/ framework keeps it ’ s relevancy

Slide 231

Slide 231 text

Or put differently

Slide 232

Slide 232 text

These tools are necessary and expected from a modern language and framework

Slide 233

Slide 233 text

Slide 234

Slide 234 text

I want to keep building with Ruby and Rails

Slide 235

Slide 235 text

I want to help build the tools needed for the future of Rails applications.

Slide 236

Slide 236 text

WEEKLY

Slide 237

Slide 237 text

Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1450+ subscribers and growing WEEKLY

Slide 238

Slide 238 text

t @Hotwire_Weekly M @[email protected] Hotwire Weekly Sign up on hotwireweekly.com