Developer Tooling For The Modern Rails & Hotwire Era @ EuRuKo 2024, Sarajevo, BiH

The evolution of developer experience (DX) tooling has been a game-changer in how we build, debug, and enhance web applications.

This talk aims to illuminate the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers, including how advanced browser extensions and tools specifically designed for the Hotwire ecosystem can level up your developer experience.

https://2024.euruko.org/speakers/marco_roth

Transcript

2. 👋 Marco Roth t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
4. Understand LSPs Building a simple LSP What ’ s Next for Tooling
5. CableReady Core Team Maintainer
6. I ❤ Open Source
7. rubyvideo.dev
8. Started to actively contribute in 2019
10. Had enough
13. X
14. Superpower ⚡
15. The future of modern Rails apps
16. I want to help build the missing modern tooling
17. Went full-time on OSS from Jan 2022 - Aug 2023
18. To work on Hotwire and related projects
19. But, I noticed something
21. The JavaScript tooling and DX is awesome
22. We didn ’ t really have anything like it in the Ruby world
24. I started to miss these tools while working in Ruby/Rails
25. No IDE
26. They felt clunky, heavyweight and cumbersome to use
27. Made me less productive
28. But this changed in the last few years
29. Editor Evolution
30. Language Server Protocol (LSP)
31. What is it and why do we want to use it?
32. Language Intelligence without the need for an IDE
33. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
34. How does it work?
35. Your editor starts a background process per language
36. Communicates over STDIN/STDOUT or TCP/IP
37. JSON-RPC
38. Similar to HTTP
39. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
40. Stimulus LSP
41. What are we trying to solve?
42. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
43. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
44. <div |> </div>
45. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
46. <div data-controller="|"> </div>
47. <div data-controller="user-profile"> </div>
48. <div data-controller="user_profile"> </div>
49. These mistakes happen so easily
50. I thought: "We can do better!"
51. Jun 23, 2021
56. Let ’ s take a look how it works
57. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
58. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
59. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
60. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
61. The server returns a JSON response
62. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
64. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
65. Embedded Languages
66. The Stimulus grammar is not part of the HTML spec
67. So we need to extend the handling for HTML-like documents
68. Language services
69. Which language do you write it in?
72. Existing Data Providers Portability Documentation/Integration Existing Language Services
74. Data Providers
75. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
76. Let ’ s implement it
77. provideTags() { }
78. <div data-controller="hello"> ... </div>
79. provideTags() { }
80. provideTags() { return [] }
81. provideAttributes(tag: string) { }
82. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
83. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
84. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
85. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
86. Parse JavaScript
87. Parse Stimulus Controllers
88. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
89. Acorn
90. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
91. f app/javascript/controllers/hello_controller.js
92. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
93. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
94. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
95. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
96. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
97. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
98. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
99. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
100. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
101. We do something similar for the Targets
102. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
103. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
104. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
105. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
106. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
107. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
108. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
110. <div data-controller="|"> </div>
111. provideValues(_tag: string, attribute: string) { }
112. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
113. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
115. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
116. Jun 25, 2021
118. What else can we do?
119. Can ’ t AI do all of that?
120. Maybe
121. Different Use Cases
122. AI for "boilerplate" LSPs for reliability and accuracy
123. Limited Context Window Missing relation between relevant files
124. Diagnostics
130. What else can we do?
131. Code Actions to guide the user in the right direction
133. Fast forward to October 2023
134. Available Now Stimulus LSP
135. I asked the community for feedback
136. And got a lot of awesome feedback and ideas
137. Some of the naive approaches we used were starting to fall short
138. Especially the way we were parsing the JavaScript with Acorn
139. We assumed and hard-coded app/javascript/controllers/
140. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
142. The Language Server works
143. But produced a lot of false positives and inaccurate results
144. Which is really annoying as a user
145. We addressed and fixed a lot of issues
146. Which technically ended up in a rewrite of the parser
147. Fast forward to May 2024
150. What ’ s next?
153. None of is this is exclusive to the LSP
154. The diagnostics could be used independently
155. Kind of like RuboCop is an independent tool
156. Coming Soon Stimulus Lint
157. Errors Diagnostics Best Practices Recommendations
158. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
159. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
160. Diagnostics Code Actions Code Completion Refactoring Tools
161. Pre-Release Turbo LSP
163. Turbo also makes heavy use of ERB and view helpers
164. So it ’ s only really useful if we can provide intelligence
165. And that only works if we actually understand ERB
166. Stimulus LSP understands HTML in ERB documents
167. But it doesn ’ t understand Embedded Ruby that outputs HTML
170. Source: https://twitter.com/vinistock/status/1710280529889620038
171. ERB Support in Ruby LSP
172. RubyConf 2023 Hack Day
174. We quickly realized that we need something more powerful to make things work beyond the basics
175. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
176. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
177. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
178. Source: https://github.com/Shopify/ruby-lsp/pull/2235
179. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
180. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
181. We lost the HTML-context in the process
182. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
183. HTML with embedded Ruby (.html.erb, .rhtml, …)
184. ERB is versatile
185. a
186. Experiment HTML-aware ERB Parser
187. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP (maybe?) and more…
188. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
189. Which made me think that it could be useful to have an HTML abstraction
190. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
191. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
192. Needs more exploration
193. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
194. erb-languageservice html-templating-languageservice
195. If the parser can understand any ERB context and Rails-render calls it is super powerful
196. This could power advanced diagnostics
197. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
198. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
199. Modern Rails apps are composed of multiple views/partials/components
200. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
201. But if the parser understands the Rails render logic
202. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
203. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
204. I think an advanced HTML-aware ERB parser could prove super helpful
205. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
206. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
207. HTML+ERB Formatter HTML+ERB Linter and more?
208. Prism has a big effect on Ruby internals and the tooling landscape
209. An HTML-aware ERB parser could have a similar effect for HTML template tooling
210. In Progress Hotwire Browser Extension
215. Highlight Stimulus Controllers & Turbo Frames
216. Sneak Peak
220. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
221. Turbo Frames Reload Button R
222. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
223. It ’ s a work in progress
224. Let us know if you have any ideas, we are just getting started
225. Conclusion
226. Developer Experience and Developer Ergonomics matter.
227. This is the kind of tooling we are currently missing.
228. There is so much potential to level up
229. I am super excited for the future of these tools.
230. I believe that tools like these are part of the reason why a language/ framework keeps it ’ s relevancy
231. Or put differently
232. These tools are necessary and expected from a modern language and framework
233. X
234. I want to keep building with Ruby and Rails
235. I want to help build the tools needed for the future of Rails applications.
236. WEEKLY
237. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1450+ subscribers and growing WEEKLY
238. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
239. A more productive Hotwire ecosystem.
240. A more unified Hotwire ecosystem.
241. Feedback Ideas Thoughts Improvements Come fi nd me!
242. Stickers
243. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth