Developer Tooling For The Modern Rails & Hotwire Era @ Madison Ruby 2024

The evolution of developer experience tooling has been a game-changer in how we build and debug web applications.

This talk aims to showcase the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers.

The session will extend beyond LSPs, to explore the potential of browser extensions and development tools tailored specifically for Turbo, Stimulus, and Hotwire.

Transcript

2. 👋 Marco Roth t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Jim Remsik 🙏
4. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
5. CableReady Core Team Maintainer
6. I ❤ Open Source
8. Started to actively contribute in 2019
10. Had enough
13. X
14. Superpower ⚡
15. The future of modern Rails apps
16. I want to help build the modern tooling needed
17. Went full-time on OSS in January 2022
18. To work on Hotwire and related projects
19. But, I noticed something
20. The tooling and DX is awesome
21. We didn ’ t really have anything like it in the Ruby world
23. I started to miss these tools while working in Ruby/Rails
24. No IDE
25. They felt clunky, heavyweight and cumbersome to use
26. Made me less productive
27. But this changed in the last few years
28. Editor Evolution
29. Language Server Protocol (LSP)
30. What is it and why do we want to use it?
31. Language Intelligence without the need for an IDE
32. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
33. How does it work?
34. Your editor starts a one background process per language
35. Communicates over STDIN/STDOUT or TCP/IP
36. JSON-RPC
37. Similar to HTTP
38. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
39. Stimulus LSP What are we trying to solve?
40. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
41. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
42. <div |> </div>
43. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
44. <div data-controller="|"> </div>
45. <div data-controller="user-profile"> </div>
46. <div data-controller="user_profile"> </div>
47. These mistakes happen so easily
48. I thought: "We can do better!"
49. Jun 23, 2021
54. Let ’ s take a look how it works
55. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
56. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
57. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
58. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
59. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
61. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
62. Embedded Languages
63. The Stimulus grammar is not part of the HTML spec
64. So we need to extend the handling for HTML
65. Language services
67. Data providers
68. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
69. Which language do you write it in?
72. Existing Data Providers Portability Documentation/Integration Existing Language Services
73. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
74. Let ’ s implement it
75. provideTags() { }
76. provideTags() { return [] }
77. provideAttributes(tag: string) { }
78. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
79. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
80. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
81. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
82. Parse JavaScript
83. Parse Stimulus Controllers
84. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
85. Acorn
86. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
87. f app/javascript/controllers/hello_controller.js
88. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
89. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
90. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
91. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
92. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
93. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
94. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
95. We do something similar for the Targets
96. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
97. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
98. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
99. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
100. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
101. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
103. <div data-controller="|"> </div>
104. provideValues(_tag: string, attribute: string) { }
105. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
106. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
108. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
109. Jun 25, 2021
111. What else can we do?
112. Diagnostics
118. What else can we do?
119. Code Actions to guide the user in the right direction
121. Fast forward to October 2023
122. Available Now Stimulus LSP
123. I asked the community for feedback
124. And got a lot of awesome feedback and ideas
125. Some of the naive approaches we used were starting to fall short
126. Especially the way we were parsing the JavaScript with Acorn
127. We hard-coded app/javascript/controllers/
128. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
130. The Language Server works
131. But produced a lot of false positives and inaccurate results
132. Which is really annoying as a user
133. We addressed and fixed a lot of issues
134. Which technically ended up in a rewrite of the parser
135. Fast forward to May 2024
138. What ’ s next?
140. None of is this is exclusive to the LSP
141. The diagnostics could be used independently
142. Kind of like RuboCop is an independent tool
143. Coming Soon Stimulus Lint
144. Errors Diagnostics Best Practices Recommendations
145. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
146. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
147. Diagnostics Code Actions Code Completion Refactoring Tools
148. Pre-Release Turbo LSP
150. Turbo also makes heavy use of ERB and view helpers
151. So it ’ s only really useful if we can provide intelligence
152. And that only works if we actually understand ERB
153. Stimulus LSP understands HTML in ERB documents
154. But it doesn ’ t understand Embedded Ruby that outputs HTML
157. Source: https://twitter.com/vinistock/status/1710280529889620038
158. ERB Support in Ruby LSP
159. RubyConf 2023 Hack Day
161. We quickly realized that we need something more powerful to make things work beyond the basics
162. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
163. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
164. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
165. Source: https://github.com/Shopify/ruby-lsp/pull/2235
166. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
167. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
168. We lost the HTML-context in the process
169. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
170. Experiment HTML-aware ERB Parser
171. Stimulus LSP Turbo LSP Stimulus Lint Ruby LSP (maybe?)
172. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
173. Which made me think that it could be useful to have an HTML abstraction
174. In which the ERB view helpers also have an HTML tag node for Rails View Helpers
175. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
176. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
177. Hypertext Abstract Syntax Tree
178. { "type": "element", "tagName": "div", "properties": { "data-controller": "hello", "data-action": "click->hello#greet", }, "children": [] }
179. But we would have to adapt it to be compatible with ERB
180. Needs more exploration
181. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
182. As long as we can transform the template syntax into HAST
183. Then we could build the whole logic for the Stimulus LSP on top of that
184. erb-languageservice html-templating-languageservice
185. If the parser can understand any ERB context and Rails-render calls it is super powerful
186. This could power advanced diagnostics
187. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
188. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
189. Modern Rails apps are composed of multiple views/partials/components
190. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
191. But if the parser understands the render logic
192. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
193. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
194. I think an advanced HTML-aware ERB parser could proof super helpful
195. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
196. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
197. HTML+ERB Formatter HTML+ERB Linter and more?
198. Prism has a big effect on Ruby internals and the tooling landscape
199. An HTML-aware ERB parser could have a similar effect for HTML template tooling
200. In Progress Hotwire Browser Extension
204. It ’ s a work in progress
206. Highlight Stimulus Controllers & Turbo Frames
207. Turbo Event Logging
208. Sneak Peak
211. Similarly for Stimulus Targets
213. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
214. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
215. Let us know if you have any ideas, we are just getting started
216. Conclusion
217. Developer Experience and Developer Ergonomics matter.
218. This is the kind of tooling we are currently missing.
219. There is so much potential to level up
220. I am super excited for the future of these tools.
221. I believe that tools like these are part of the reason why a language/framework keeps it ’ s relevancy
222. Or put differently
223. These tools are necessary and expected from a modern language and framework
224. X
225. I want to keep building with Ruby and Rails
226. I want to help build the tools needed for the future of Rails applications.
227. WEEKLY
228. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1350+ subscribers and growing WEEKLY
229. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
230. A more productive Hotwire ecosystem
231. A more unified Hotwire ecosystem
232. Feedback Ideas Thoughts Improvements Come fi nd me!
233. Stickers
234. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth