Developer Tooling For The Modern Rails & Hotwire Era @ Rocky Mountain Ruby 2024, Boulder, CO

The evolution of developer experience (DX) tooling has been a game-changer in how we build, debug, and enhance web applications.

This talk aims to illuminate the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers, including how advanced browser extensions and tools specifically designed for the Hotwire ecosystem can level up your developer experience.

https://rockymtnruby.dev/schedule

Transcript

2. 👋 Marco Roth t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
4. Understand LSPs Building a simple LSP What ’ s Next for Tooling
5. CableReady Core Team Maintainer
6. I ❤ Open Source
7. rubyvideo.dev
8. Started to actively contribute in 2019
10. Had enough
13. X
14. Superpower ⚡
15. The future of modern Rails apps
16. I wanted to go back doing Full-Stack Rails
17. I want to help build the missing modern tooling
18. Went full-time on OSS from Jan 2022 - Aug 2023
19. To work on Hotwire and related projects
20. But, I noticed something
22. The JavaScript tooling and DX is awesome
23. We didn ’ t really have anything like it in the Ruby world
24. Or something quite as easy to setup and get started
26. I started to miss these tools while working in Ruby/Rails
27. No IDE
28. They felt clunky, heavyweight and cumbersome to use
29. Made me less productive
30. But this changed in the last few years
31. Editor Evolution
32. Language Server Protocol (LSP)
33. What is it and why do we want to use it?
34. Language Intelligence without the need for an IDE
35. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
36. How does it work?
37. Your editor starts a background process per language
38. Communicates over STDIN/STDOUT or TCP/IP
39. JSON-RPC
40. Similar to HTTP
41. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
42. Stimulus LSP
43. What are we trying to solve?
44. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
45. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
46. <div |> </div>
47. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
48. <div data-controller="|"> </div>
49. <div data-controller="user-profile"> </div>
50. <div data-controller="user_profile"> </div>
51. These mistakes happen so easily
52. I thought: "We can do better!"
53. Jun 23, 2021
58. Let ’ s take a look how it works
59. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
60. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
61. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
62. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
63. The server returns a JSON response
64. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
66. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
67. Embedded Languages
68. The Stimulus grammar is not part of the HTML spec
69. So we need to extend the handling for HTML-like documents
70. Language services
71. Which language do you write it in?
74. Existing Data Providers Portability Documentation/Integration Existing Language Services
76. Data Providers
77. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
78. Let ’ s implement it
79. provideTags() { }
80. <div data-controller="hello"> ... </div>
81. provideTags() { }
82. provideTags() { return [] }
83. provideAttributes(tag: string) { }
84. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
85. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
86. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
87. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
88. Parse JavaScript
89. Parse Stimulus Controllers
90. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
91. Acorn
92. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
93. f app/javascript/controllers/hello_controller.js
94. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
95. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
96. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
97. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
98. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
99. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
100. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
101. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
102. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
103. We do something similar for the Targets
104. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
105. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
106. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
107. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
108. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
109. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
110. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
112. <div data-controller="|"> </div>
113. provideValues(_tag: string, attribute: string) { }
114. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
115. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
117. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
118. Jun 25, 2021
120. What else can we do?
121. Can ’ t AI do all of that?
122. Maybe
123. Different Use Cases
124. AI for "boilerplate" LSPs for reliability and accuracy
125. Limited Context Window Missing relation between relevant files
126. Diagnostics
132. What else can we do?
133. Code Actions to guide the user in the right direction
135. Fast forward to October 2023
136. Available Now Stimulus LSP
137. I asked the community for feedback
138. And got a lot of awesome feedback and ideas
139. Some of the naive approaches we used were starting to fall short
140. Especially the way we were parsing the JavaScript with Acorn
141. We assumed and hard-coded app/javascript/controllers/
142. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
144. The Language Server works
145. but it produced a lot of false positives and inaccurate results
146. Which is really annoying as a user
147. We addressed and fixed a lot of issues
148. Which technically ended up in a rewrite of the parser
149. Fast forward to May 2024
152. What ’ s next?
155. None of is this is exclusive to the LSP
156. The diagnostics could be used independently
157. Kind of like an independent tool as RuboCop
158. Coming Soon Stimulus Lint
159. Errors Diagnostics Best Practices Recommendations
160. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
161. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
162. Diagnostics Code Actions Code Completion Refactoring Tools
163. Pre-Release Turbo LSP
165. Super Basic right now
166. Turbo also makes heavy use of ERB and view helpers
167. So it ’ s only really useful if we can provide intelligence for ERB
168. And that only works if we actually understand ERB
169. Stimulus LSP understands HTML in ERB documents
170. But it doesn ’ t understand Embedded Ruby that outputs HTML
173. Source: https://twitter.com/vinistock/status/1710280529889620038
174. ERB Support in Ruby LSP
175. RubyConf 2023 Hack Day
177. We quickly realized that we need something more powerful to make things work beyond the basics
178. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
179. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
180. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
181. Source: https://github.com/Shopify/ruby-lsp/pull/2235
182. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
183. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
184. We lost the HTML-context in the process
185. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
186. HTML with embedded Ruby (.html.erb, .rhtml, …)
187. ERB is versatile
188. a
189. Experiment HTML-aware ERB Parser
190. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP (maybe?) and more…
191. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
192. Which made me think that it could be useful to have an HTML abstraction
193. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
194. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
195. Needs more exploration
196. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
197. erb-languageservice html-templating-languageservice
198. If the parser can understand any ERB context and Rails-render calls it is super powerful
199. This could power advanced diagnostics
200. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
201. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
202. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
203. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
204. Modern Rails apps are composed of multiple views/partials/components
205. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
206. But if the parser understands the Rails render logic
207. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
208. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
209. An advanced HTML-aware ERB parser could prove super helpful
210. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
211. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
212. HTML+ERB Formatter HTML+ERB Linter A new ERB Engine? Possibly even more?
213. Prism has a big effect on Ruby internals and the tooling landscape
215. An HTML-aware ERB parser could have a similar effect for HTML template tooling
216. In Progress Hotwire Browser Extension
221. Highlight Stimulus Controllers & Turbo Frames
222. Sneak Peak
226. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
227. Turbo Frames Reload Button R
228. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
229. It ’ s a work in progress
230. Let us know if you have any ideas, we are just getting started
231. Conclusion
232. Developer Experience and Developer Ergonomics matter.
233. This is the kind of tooling we are currently missing.
234. There is so much potential to level up even more
235. I am super excited for the future of these tools.
236. I believe that tools like these are part of the reason why a language/ framework keeps it ’ s relevancy
237. Or put differently
238. These tools are necessary and expected from a modern language and framework
239. We have been lacking behind
240. X
241. I want to keep building with Ruby and Rails
242. I want to help build the tools needed for the future of Rails applications.
243. WEEKLY
244. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1550+ subscribers and growing WEEKLY
245. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
246. A more productive Hotwire ecosystem.
247. A more unified Hotwire ecosystem.
248. Feedback Ideas Thoughts Improvements Come fi nd me!
249. Stickers
250. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth