Developer Tooling For The Modern Rails & Hotwire Era @ Geneva.rb Meetup, October 2024

The evolution of developer experience (DX) tooling has been a game-changer in how we build, debug, and enhance web applications.

This talk aims to illuminate the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers, including how advanced browser extensions and tools specifically designed for the Hotwire ecosystem can level up your developer experience.

https://marcoroth.dev/talks/geneva-rb-meetup-october-2024
https://www.meetup.com/geneva-rb/events/303018643

Transcript

1. Oct 16, 2024 Developer Tooling For The Modern Hotwire & Rails Era Marco Roth
2. Yannis Jaquet 🙏
3. Yannis Jaquet 🙏
4. Marco Roth 👋 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
5. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
6. Understand LSPs Building a simple LSP What ’ s Next for Tooling
7. CableReady Core Team Maintainer
8. I ❤ Open Source
9. Open Source Journey
10. rubyvideo.dev
11. Started to actively contribute in 2019
13. Had enough
16. X
17. Superpower ⚡
18. The future of modern Rails apps
19. I wanted to go back doing Full-Stack Rails
20. I want to help build the missing modern tooling
21. Went full-time on OSS from Jan 2022 - Aug 2023
22. To work on Hotwire and related projects
23. But, I noticed something
25. The JavaScript tooling and DX is awesome
26. We didn ’ t really have anything like it in the Ruby world
27. Or something quite as easy to setup and get started
29. I started to miss these tools while working in Ruby/Rails
30. No IDE
31. They felt clunky, heavyweight and cumbersome to use
32. Made me less productive
33. But this changed in the last few years
34. Editor Evolution
35. Language Server Protocol (LSP)
36. What is it and why do we want to use it?
37. Language Intelligence without the need for an IDE
38. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
39. How does it work?
40. Your editor starts a background process per language
41. Communicates over STDIN/STDOUT or TCP/IP
42. JSON-RPC
43. Similar to HTTP
44. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
45. Request Message Response Message Notification Message
46. HTML Language Client Ruby Language Client Ruby Language Server HTML Language Server Editor Workspace Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
47. Notification Messages
48. Text Document Synchronization
49. textDocument/didOpen textDocument/didChange textDocument/didClose textDocument/didSave textDocument/publishDiagnostics ...
50. Source: https://microsoft.github.io/language-server-protocol/overviews/lsp/overview Editor / Client Language Server
51. Source: https://microsoft.github.io/language-server-protocol/overviews/lsp/overview Ruby Stimulus HTML Editor
52. Stimulus LSP
53. What are we trying to solve?
54. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
55. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
56. <div |> </div>
57. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
58. <div data-controller="|"> </div>
59. <div data-controller="user-profile"> </div>
60. <div data-controller="user_profile"> </div>
61. These mistakes happen so easily
62. I thought: "We can do better!"
63. Jun 23, 2021
68. Let ’ s take a look how it works
69. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
70. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
71. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
72. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
73. The server returns a JSON response
74. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
76. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
77. Embedded Languages
78. The Stimulus grammar is not part of the HTML spec
79. So we need to extend the handling for HTML-like documents
80. Language services
81. Which language do you write it in?
84. Existing Data Providers Portability Documentation/Integration Existing Language Services
86. Data Providers
87. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
88. Let ’ s implement it
89. provideTags() { }
90. <div data-controller="hello"> ... </div>
91. provideTags() { }
92. provideTags() { return [] }
93. provideAttributes(tag: string) { }
94. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
95. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
96. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
97. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
98. Parse JavaScript
99. Parse Stimulus Controllers
100. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
101. Acorn
102. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
103. f app/javascript/controllers/hello_controller.js
104. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
105. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
106. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
107. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
108. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
109. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
110. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
111. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
112. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
113. We do something similar for the Targets
114. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
115. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
116. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
117. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
118. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
119. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
120. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
122. <div data-controller="|"> </div>
123. provideValues(_tag: string, attribute: string) { }
124. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
125. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
127. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
128. Jun 25, 2021
130. What else can we do?
131. Can ’ t AI do all of that?
132. Maybe
133. Different Use Cases
134. AI for "boilerplate" LSPs for reliability and accuracy
135. Limited Context Window Missing relation between relevant files
136. Diagnostics
142. What else can we do?
143. Code Actions to guide the user in the right direction
145. Fast forward to October 2023
146. Available Now Stimulus LSP
147. I asked the community for feedback
148. And got a lot of awesome feedback and ideas
149. Some of the naive approaches we used were starting to fall short
150. Especially the way we were parsing the JavaScript with Acorn
151. We assumed and hard-coded app/javascript/controllers/
152. The controller identifiers weren ’ t always accurate
153. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
155. The Language Server works
156. but it produced a lot of false positives and inaccurate results
157. Which is really annoying as a user
158. We addressed and fixed a lot of issues
159. Which technically ended up in a rewrite of the parser
160. Fast forward to May 2024
163. What ’ s next?
166. None of is this is exclusive to the LSP
167. The diagnostics could be used independently
168. Kind of like an independent tool as RuboCop
169. Coming Soon Stimulus Lint
170. Errors Diagnostics Best Practices Recommendations
171. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
172. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
173. Diagnostics Code Actions Code Completion Refactoring Tools
174. Pre-Release Turbo LSP
176. Super Basic right now
177. Turbo also makes heavy use of ERB and view helpers
178. So it ’ s only really useful if we can provide intelligence for ERB
179. And that only works if we actually understand ERB
180. Stimulus LSP understands HTML in ERB documents
181. But it doesn ’ t understand Embedded Ruby that outputs HTML
184. Source: https://twitter.com/vinistock/status/1710280529889620038
185. ERB Support in Ruby LSP
186. RubyConf 2023 Hack Day
188. We quickly realized that we need something more powerful to make things work beyond the basics
189. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
190. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
191. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
192. Source: https://github.com/Shopify/ruby-lsp/pull/2235
193. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
194. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
195. We lost the HTML-context in the process
196. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
197. HTML with embedded Ruby (.html.erb, .rhtml, …)
198. ERB is versatile
199. a
200. Experiment HTML-aware ERB Parser
201. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP (maybe?) and more…
202. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
203. Which made me think that it could be useful to have an HTML abstraction
204. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
205. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
206. Needs more exploration
207. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
208. erb-languageservice html-templating-languageservice
209. If the parser can understand any ERB context and Rails-render calls it is super powerful
210. This could power advanced diagnostics
211. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
212. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
213. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
214. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
215. Modern Rails apps are composed of multiple views/partials/components
216. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
217. But if the parser understands the Rails render logic
218. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
219. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
220. An advanced HTML-aware ERB parser could prove super helpful
221. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
222. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
223. HTML+ERB Formatter HTML+ERB Linter A new ERB Engine? Possibly even more?
224. Prism has a big effect on Ruby internals and the tooling landscape
226. An HTML-aware ERB parser could have a similar effect for HTML template tooling
227. In Progress Hotwire Browser Extension
232. Highlight Stimulus Controllers & Turbo Frames
233. Highlight Stimulus Controllers & Turbo Frames
234. Turbo Event Logging
235. Sneak Peak
239. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
240. Turbo Frames Reload Button R
241. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
242. It ’ s a work in progress
243. Let us know if you have any ideas, we are just getting started
244. Conclusion
245. Developer Experience and Developer Ergonomics matter.
246. This is the kind of tooling we are currently missing.
247. There is so much potential to level up even more
248. I am super excited for the future of these tools.
249. I believe that tools like these are part of the reason why a language/ framework keeps it ’ s relevancy
250. Or put differently
251. These tools are necessary and expected from a modern language and framework
252. We have been lacking behind
253. X
254. I want to keep building with Ruby and Rails
255. I want to help build the tools needed for the future of Rails applications.
256. WEEKLY
258. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1550+ subscribers and growing WEEKLY
259. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
260. A more productive Hotwire ecosystem.
261. A more unified Hotwire ecosystem.
262. Stickers
264. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth