Developer Tooling For The Modern Hotwire & Rails Era @ RedDotRubyConf 2024, Singapore

The evolution of developer experience tooling has been a game-changer in how we build and debug web applications.

This talk aims to showcase the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers.

The session will extend beyond LSPs, to explore the potential of browser extensions and development tools tailored specifically for Turbo, Stimulus, and Hotwire.

Transcript

2. 👋 Marco Roth t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
4. CableReady Core Team Maintainer
5. I ❤ Open Source
6. Open Source Journey
8. Started to actively contribute in 2019
10. Went full-time on OSS in January 2022
11. To work on Hotwire and related projects
12. I want to improve Hotwire so companies don ’ t need to use React
13. While working on the React applications
14. I noticed something
15. The tooling and DX is awesome
16. We didn ’ t really have anything like it in the Ruby world
18. I started to miss these tools while working in Ruby/Rails
21. X
22. Superpower ⚡
23. The future of modern Rails apps
24. I want to help build the modern tooling needed
25. No IDE
26. They felt clunky, heavyweight and cumbersome
27. Made me less productive
28. But this changed in the last few years
29. Editor Evolution
30. Language Server Protocol (LSP)
31. What is it and why do we want to use it?
32. Language Intelligence without the need for an IDE
33. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
34. How does it work?
35. Your editor starts a one background process per language
36. Communicates over STDIN/STDOUT or TCP/IP
37. JSON-RPC
38. Similar to HTTP
39. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
40. Request Message Response Message Notification Message
41. Notification Messages
42. Text Document Synchronization
43. textDocument/didOpen textDocument/didChange textDocument/didClose textDocument/didSave textDocument/publishDiagnostics ...
44. Source: https://microsoft.github.io/language-server-protocol/overviews/lsp/overview Editor / Client Language Server
45. Stimulus LSP What are we trying to solve?
46. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
47. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
48. <div |> </div>
49. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
50. <div data-controller="|"> </div>
51. <div data-controller="hello"> </div>
52. <div data-controller="user-profile"> </div>
53. <div data-controller="user_profile"> </div>
54. These mistakes happen so easily
55. I thought: "We can do better!"
56. Jun 23, 2021
61. Let ’ s take a look how it works
62. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
63. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
64. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
65. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
67. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
68. Embedded Languages
69. The Stimulus grammar is not part of the HTML spec
70. So we need to extend the handling for HTML/JavaScript
71. Language services
73. Data providers
74. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
75. Which language do you write it in?
77. Ruby LSP plugin?
79. Existing Data Providers Portability Documentation/Integration Existing Language Services
80. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
81. Let ’ s implement it
82. provideTags() { }
83. provideTags() { return [] }
84. provideAttributes(tag: string) { }
85. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
86. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
87. provideAttributes(tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
88. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
89. Parse JavaScript
90. Parse Stimulus Controllers
91. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
92. Acorn
93. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" export class Parser { parseController(code: string, filename: string) { ... } }
94. f app/javascript/controllers/hello_controller.js
95. export default class extends Controller { static targets = ["name", "output"] connect() { console.log("Connect") } greet() { this.outputTarget.textContent = `Hello ${this.nameTarget.value}` } disconnect() { console.log("Disconnect") } }
96. export default class extends Controller { static targets = ["name", "output"] connect() { console.log("Connect") } greet() { this.outputTarget.textContent = `Hello ${this.nameTarget.value}` } disconnect() { console.log("Disconnect") } }
97. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
98. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
99. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
100. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
101. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
102. We do something similar for the Targets
103. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
104. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
105. provideAttributes(tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
106. provideAttributes(tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
107. provideAttributes(tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
108. provideAttributes(tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
110. <div data-controller="|"> </div>
111. provideValues(_tag: string, attribute: string) { }
112. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
113. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
115. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
116. Jun 25, 2021
118. What else can we do?
119. Diagnostics
125. What else can we do?
126. Code Actions to guide the user in the right direction
128. After a few months
129. Available Now Stimulus LSP
130. I asked the community for feedback
131. And got a lot of awesome feedback and ideas
132. Some of the naive approaches we used were starting to fall short
133. Especially the way we were parsing the JavaScript with Acorn
134. We hard-coded app/javascript/controllers/
135. The controller identifiers weren ’ t always accurate
136. Inferring the identifier from the filename is not always enough
137. import { application } from "./application" import ClipboardController from "./clipboard_controller.js" application.register("clipboard", ClipboardController) import FileUploadController from "./file_upload_controller.js" application.register("file-upload", FileUploadController) f app/javascript/controllers/index.js
138. import { application } from "./application" import ClipboardController from "./clipboard_controller.js" application.register("clipboard", ClipboardController) import FileUploadController from "./file_upload_controller.js" application.register("file-upload", FileUploadController) f app/javascript/controllers/index.js
139. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
141. The Language Server works
142. But produced a lot of false positives and inaccurate results
143. Which is really annoying as a user
144. We addressed and fixed a lot of issues
145. Which technically ended up in a rewrite
148. What ’ s next?
149. Coming Soon Stimulus Lint
151. None of is this is exclusive to the LSP
152. The diagnostics could be used independently
153. RuboCop is also an independent tool
154. Errors Diagnostics Best Practices Recommendations
155. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
156. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
157. Code Completion Diagnostics Code Actions Refactoring Tools
158. Pre-Release Turbo LSP
160. Turbo also makes heavy use of ERB and view helpers
161. So it ’ s only really useful if we can provide intelligence
162. And that only works if we actually understand ERB
163. Stimulus LSP understands HTML in ERB documents
164. But it doesn ’ t understand Embedded Ruby that outputs HTML
165. ERB Support in Ruby LSP
167. We quickly realized that we need something more powerful to make things work beyond the basics
168. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
169. █████ ██████████ @user.firstname █████████ ██████ @user.lastname ██████ ██████
170. @user.firstname @user.lastname
171. Source: https://github.com/Shopify/ruby-lsp/pull/2235
172. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
173. But we lost the whole HTML-context in the process
174. So I started to dabble and explore what would be possible with a new HTML-aware ERB parser
175. The idea is to power the ERB support with the new HTML-aware parser in these tools Stimulus LSP Turbo LSP Stimulus Lint Ruby LSP (maybe?)
176. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
177. Which made me think that it could be useful to have an HTML-AST
178. In which the ERB view helpers also have an HTML tag node for Rails View Helpers
179. <%= tag.p do %> Hello world! <% end %> <p> Hello world! </p> ==
180. This would also open to door to support Haml, Slim, Liquid, Blade and other template languages
181. If we can transform their syntax into the same AST
182. Then we could build the whole logic for the Stimulus LSP on top of that AST
183. And we don ’ t really need to care what templating language it actually is
184. If this is done right it could also help with other tooling such as linters, formatters, and code navigation tools.
185. erb-languageservice html-templating-languageservice
186. If the parser can understand any ERB context and Rails-render calls it is super powerful
187. This could power advanced diagnostics
188. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
189. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
190. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
191. But if the parser understands the render logic
192. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
193. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
194. Modern Rails apps are composed of multiple views/partials/components
195. I think an advanced HTML-aware ERB parser could proof super helpful
196. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
197. html_press gem phlexing gem deface gem better_html gem erb_lint gem
198. HTML+ERB formatter HTML+ERB Linter Phoenix LiveView HEEx-like Template Engine for ERB
200. This would be super useful for modern real-time applications powered by Hotwire
201. I am not sure how feasible it is, but the ERB parser would make it a lot more approachable.
202. Prism has a big effect on Ruby internals and the tooling landscape
203. An HTML-aware ERB parser could have a similar effect for HTML template tooling
204. In Progress Hotwire Browser Extension
208. It ’ s a work in progress
210. Highlight Stimulus Controllers & Turbo Frames
211. Turbo Event Logging
212. Sneak Peak
215. Similarly for Stimulus Targets
217. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
218. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
219. Let us know if you have any ideas, we are just getting started
220. Recap
221. Developer Experience and Developer Ergonomics matter.
222. This is the kind of tooling we are currently missing.
223. There is so much potential!
224. I am super excited for the future of these tools.
225. I believe that tools like this are part of the reason why a language/framework keeps it ’ s relevancy
226. Or put differently
227. These tools are necessary and expected from a modern language and framework
228. X
229. I want to help build the tools needed for the future of Rails applications.
230. WEEKLY
231. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1350+ subscribers and growing WEEKLY
232. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
233. A more unified Hotwire ecosystem
234. A more productive Hotwire ecosystem
235. Feedback Ideas Thoughts Improvements Come fi nd me!
236. Stickers
237. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth