[RailsWorld 2023] Untangling cables & demystifying twisted transistors

Video: https://www.youtube.com/watch?v=GrHpop5HtxM

---

More and more Rails applications adopt real-time features, and it’s not surprising—Action Cable and Hotwire brought development experience to the next level regarding dealing with WebSockets. You need zero knowledge of the underlying tech to start crafting a new masterpiece of web art! However, you will need this knowledge later to deal with ever-sophisticated feature requirements and security and scalability concerns.

The variety of questions that arise when developers work with Rails’ real-time tooling is broad, from “Which delivery guarantees does Action Cable provide?” to “Can I scale my Hotwire application to handle dozens of thousands of concurrent users?”. To answer them, we need to learn our tools first.

In my talk, I will help you better to understand Rails’ real-time component—Action Cable. I want to open this black box for you and sort through the internals so you can work with Action Cable efficiently and confidently.

Transcript

1. Vladimir Dementyev Evil Martians Untangling Cables & Demystifying Twisted Transistors 2023 1. It's Rails 2. Freak on a Cable 3. Y'all Want a Stream 4. Make Me ping 5. Wire Tongue 6. Threaded Transistor 7. Throw me & Forget 8. Turbo Streams Everywhere 9. (Not) Alone I Reconnect Bonus: See You on the Other Cable Lyrics by Vladimir "Palkan" Dementyev Produced by Evil Martians, inc.
2. 1. It's Rails
3. Rails is a software development philosophy 3 palkan_tula palkan
4. Conceptual Compression 4 palkan_tula palkan
5. Unboxing time! 5 palkan_tula palkan
6. Unboxing time! 5 palkan_tula palkan
7. 2. Freak on a Cable
9. github.com/palkan
10. github.com/palkan evilmartians.com
11. github.com/palkan evilmartians.com anycable.io
12. The Book on abstraction layers from and for Rails 8 palkan_tula palkan
13. 3. Y'all Want a Stream
14. palkan_tula palkan Channels Layer 10 class ChatChannel < ApplicationCable::Channel def subscribed @room = Room.find_by(id: params[:id]) return reject unless @room stream_for @room end def speak(data) broadcast_to(@room, event: "newMessage", user_id: user.id, body: data.fetch("body") ) end end
15. 11 palkan_tula palkan class ChatChannel < ApplicationCable::Channel def subscribed @room = Room.find_by(id: params[:id]) return reject unless @room stream_for @room end def speak(data) broadcast_to(@room, event: "newMessage", user_id: user.id, body: data.fetch("body") ) end end import { createConsumer } from "@rails/actioncable" let cable = createConsumer() let sub = cable.subscriptions.create( {channel: "ChatChannel", id: 2023}, { received({user_id, body}) { console.log(`Message from ${user_id}:${body}`); } } ) sub.perform("speak", {body: "Hoi!"})
17. palkan_tula palkan 13 Protocol Web Server Action Cable Server I/O loop Rails Executor Connection Channel Pub/Sub Subscriber Map Client Socket Consumer Web Socket Sub Pubsub Adapter Monitor Channel Channel Worker Pool Internal Pool Sub Sub Map of Action Cable HTTP rack.hijack #connect #subscribed { command: "subscribe" } OPEN { command: "message" } #receive CLOSE #disconnect #unsubscribed #stream_from #subscribe #unsubscribe broadcast #invoke_callback #send
18. 4. Make Me Ping
19. palkan_tula palkan @rails/actioncable 15 import { createConsumer } from "@rails/actioncable" let cable = createConsumer() cable.subscriptions.create( {channel: "ChatChannel", id: 2023}, { received({user_id, body}) { console.log(`Message from ${user_id}:${body}`); } } )
20. palkan_tula palkan @rails/actioncable 16 import { createConsumer } from "@rails/actioncable" let cable = createConsumer() cable.subscriptions.create( {channel: "ChatChannel", id: 2023}, { received({user_id, body}) { console.log(`Message from ${user_id}:${body}`); } } ) ~ WebSocket connection Abstraction to subscribe for and receive updates
21. palkan_tula palkan One consumer per session (e.g., a browser tab) 17 @rails/actioncable
22. palkan_tula palkan github.com/le0pard/cable-shared-worker 18
23. palkan_tula palkan One consumer per session (e.g., a browser tab) Many subscriptions per consumer (multiplexing) 19 @rails/actioncable
24. palkan_tula palkan One consumer per session (e.g., a browser tab) Many subscriptions per consumer (multiplexing) — how many? 20 @rails/actioncable
25. We may* have as many subscriptions as canals in Amsterdam 21 palkan_tula palkan * Doesn't mean we should though Consumer Monitor Sub Sub Sub Web Socket Sub Sub
26. palkan_tula palkan @rails/actioncable 22 import { createConsumer } from "@rails/actioncable" let cable = createConsumer() let a = cable.subscriptions.create("UserChannel") // SUBSCRIBE let b = cable.subscriptions.create("UserChannel") // <NONE> a.unsubscribe() // <NONE> b.unsubscribe() // UNSUBSCRIBE
27. palkan_tula palkan One consumer per session (e.g., a browser tab) Many subscriptions per consumer Monitor to keep consumer connected 23 @rails/actioncable
28. palkan_tula palkan Keep alive 24 Consumer Monitor Web Socket stale timer
29. palkan_tula palkan Keep alive 24 Consumer Monitor Web Socket ping stale timer
30. palkan_tula palkan Keep alive 24 Consumer Monitor Web Socket stale timer reconnect
31. palkan_tula palkan Action Cable Pings Help to identify broken TCP connections (from both sides, but not always) 25
32. palkan_tula palkan Action Cable Pings Help to identify broken TCP connections (from both sides, but not always) Hardly configurable (and may be too frequent to cause battery drain for mobile devices) 25
33. 26 palkan_tula palkan # config/application.rb # Re-define internal constant module ActionCable::Server::Connections BEAT_INTERVAL = 10 end import { ConnetionMonitor } from "@rails/actioncable" // Adjust stale timeout (2xBEAT_INTERVAL) ConnectionMonitor.staleThreshold = 20
34. palkan_tula palkan More clients github.com/anycable/anycable-client 27
35. palkan_tula palkan More clients github.com/anycable/anycable-client <Write your own> 27
36. 5. Wire Tongue
37. Protocol is a language in which the server and client speak to each other 29 palkan_tula palkan Protocol Web Server Consumer Web Socket HTTP
38. palkan_tula palkan docs.anycable.io 30
39. 31 palkan_tula palkan { "command":"subscribe", "identifier":"{\"channel\" \"UserChannel\"}" } { "command":"message", "identifier":"{\"channel\":\"UserChannel\"}", "data":"{\"action\":\"speak\",\"text\":\"hoi\"}" } { "command":"unsubscribe", "identifier":"{\"channel\":\"UserChannel\"}" } {"type":"welcome"} { "type":"confirm_subscription", "identifier":"{\"channel\":\"UserChannel\"}" } { "type":"reject_subscription", "identifier":"{\"channel\":\"UserChannel\"}" } { "identifier":"{\"channel\":\"UserChannel\"}", "message":"{\"text\":\"hoi\"}" } {"type":"ping"} { "type":"disconnect", "reason":"unauthorized", "reconnect":false } No type!
40. palkan_tula palkan Protocol is incomplete No IDs (session, message) No perform ACKs No error handling 32
41. palkan_tula palkan 33 let cable = createConsumer() for(let i=0; i<10; i++) { let sub = cable.subscriptions.create("UserChannel") sub.unsubscribe() } // Will I be subscribed or not? cable.subscriptions.create("UserChannel")
42. palkan_tula palkan github.com/palkan/wsdirector - loop: multiplier: 10 actions: - send: data: &sub command: "subscribe" identifier: "{\"channel\":\"BenchmarkChannel\"}" - send: data: command: "unsubscribe" identifier: "{\"channel\":\"BenchmarkChannel\"}" - send: <<: *sub - send: data: command: "message" identifier: "{\"channel\":\"BenchmarkChannel\"}" data: "{\"action\":\"echo\",\"test\":42}" - receive: data: identifier: "{\"channel\":\"BenchmarkChannel\"}" message: {action: "echo", test: 42} 34
45. 36 RuntimeError - Unable to find subscription with identifier: {"channel":"BenchmarkChannel"} RuntimeError - Already subscribed to {"channel":"BenchmarkChannel"}
46. palkan_tula palkan Solution? 37
47. 38 (First rows only note) Some random Rails application
48. palkan_tula palkan 39 let cable = createConsumer() for(let i=0; i<10; i++) { let sub = cable.subscriptions.create("UserChannel") sub.unsubscribe() } // Will I be subscribed or not? It depends. Why so? cable.subscriptions.create("UserChannel")
49. 6. Threaded Transistor
50. palkan_tula palkan 41 Web Server Web Socket HTTP Action Cable Executor
51. palkan_tula palkan 41 Web Server Action Cable Server Web Socket HTTP rack.hijack Action Cable Executor
52. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Client Socket Web Socket HTTP rack.hijack Action Cable Executor
53. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Client Socket Web Socket Worker Pool HTTP rack.hijack OPEN Action Cable Executor
54. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Rails Executor Client Socket Web Socket Worker Pool HTTP rack.hijack OPEN Action Cable Executor
55. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Rails Executor Connection Client Socket Web Socket Worker Pool HTTP rack.hijack #connect OPEN Action Cable Executor
56. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Rails Executor Connection Channel Client Socket Web Socket Worker Pool HTTP rack.hijack #connect {command: "subscribe"} OPEN #subscribed Action Cable Executor
57. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Rails Executor Connection Channel Client Socket Web Socket Worker Pool HTTP rack.hijack #connect {command: "subscribe"} OPEN {command: "message"} #receive #subscribed Action Cable Executor
58. palkan_tula palkan 41 Web Server Action Cable Server I/O loop Rails Executor Connection Channel Client Socket Web Socket Worker Pool HTTP rack.hijack #connect #unsubscribed {command: "subscribe"} OPEN {command: "message"} #receive CLOSE #disconnect #subscribed Action Cable Executor
59. palkan_tula palkan Action Cable Server Thread pool executor (4 threads by default) Must be taken into account for shared resources (e.g., database pool) Uses Rails executor to cleanup after work 42
60. 43 palkan_tula palkan class Connection identified_by :user def connect self.user = find_verified_user Current.account = user.account end end
61. 44 palkan_tula palkan class Connection identified_by :user def connect self.user = find_verified_user # BAD — will be reset at the end # of the #handle_open call Current.account = user.account end end
62. class Connection identified_by :user before_command do # GOOD — set for every command Current.account = user.account end def connect self.user = find_verified_user end end 44 palkan_tula palkan class Connection identified_by :user def connect self.user = find_verified_user # BAD — will be reset at the end # of the #handle_open call Current.account = user.account end end
63. palkan_tula palkan Units of Work Lifecycle callbacks (connect, disconnect) 45
64. palkan_tula palkan Units of Work Lifecycle callbacks (connect, disconnect) Incoming commands 45
65. palkan_tula palkan Units of Work Lifecycle callbacks (connect, disconnect) Incoming commands Outgoing messages 45
66. 7. Throw me & Forget
67. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool Pub/Sub
68. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool broadcast message Pub/Sub
69. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool broadcast message #broadcast Pub/Sub
70. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool broadcast message #invoke_callback #broadcast Pub/Sub
71. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool broadcast message #invoke_callback #broadcast #stream_from callback is executed Pub/Sub
72. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool broadcast message #invoke_callback #send #broadcast #transmit #stream_from callback is executed Pub/Sub
73. palkan_tula palkan 47 Rails Executor Connection Pub/Sub Subscriber Map Client Socket Pubsub Adapter Channel Worker Pool Internal Pool Pub/Sub at-most once
74. 48 The pitfalls of realtime-ification, RailsConf, 2022
75. palkan_tula palkan Yet Another Pitfall 49 100.times { ActionCable.server.broadcast "test", {text: "Count: #{_1}"} } # Will a client receive 1,2,3,4,...,99 in order?
78. palkan_tula palkan And One More 52 def subscribed stream_from "test" # Q: Will this client receive the message? ActionCable.server.broadcast "test", {text: "Welkom!"} end
79. Are you tired of threads? 53 palkan_tula palkan I am
80. 8. Turbo Streams Everywhere
81. palkan_tula palkan Turbo Streams z Automatic subscriptions via HTML (<turbo-stream-source>) Built-in Turbo::StreamsChannel Zero application code to connect cables 55
82. palkan_tula palkan Turbo Streams 56 www.my-hot.app
83. palkan_tula palkan Turbo Streams 54 www.my-hot.app <turbo-stream-source> SUBSCRIBE Connection Channel
84. palkan_tula palkan DOM update Turbo Streams 54 www.my-hot.app <turbo-stream-source> HTML Action Cable broadcast from anywhere Connection Channel
85. palkan_tula palkan UNSUBSCRIBE Turbo Streams 54 www.my-hot.app Connection
86. palkan_tula palkan Turbo Streams Prone to subscribe/unsubscribe race conditions Managed in a decentralized way (via HTML partials) 57
88. 59 anycable.substack.com
89. 9. (Not) Alone I ReConnect
90. palkan_tula palkan Server Restart N clients re-connect N clients re-subscribe each to M channels each Threaded executor's work queue is: (N+1)*M 61
91. palkan_tula palkan Server Restart Threaded executor's work queue is: (N+1)*M Server can not keep up with confirmations, clients re-issue subscribe commands—! 62
92. palkan_tula palkan Server Restart Threaded executor's work queue is: (N+1)*M Server can not keep up with confirmations, clients re-issue subscribe commands—! 62 Thundering Herd
93. palkan_tula palkan Server Restart Threaded executor's work queue is: (N+1)*M Server can not keep up with confirmations, clients re-issue subscribe commands—! 62 Connection Avalanche
94. Connection avalanches lead to significant load spikes 63 palkan_tula palkan
95. How to protect Rails from connection avalanches? 64 palkan_tula palkan
96. palkan_tula palkan Avalanche Protection Reconnect with backoff & jitter 65
98. palkan_tula palkan Avalanche Protection Reconnect with backoff & jitter Merge subscriptions 67
99. palkan_tula palkan Avalanche Protection Reconnect with backoff & jitter Merge subscriptions Avoid slow calls in #subscribed 67
100. palkan_tula palkan Avalanche Protection Reconnect with backoff & jitter Merge subscriptions Avoid slow calls in #subscribed Serialize subscribe commands 67
101. palkan_tula palkan 68 import { createCable } from "@anycable/web" let cable = createCable({ concurrentSubscribes: false })
102. palkan_tula palkan Isolate WebSocket and regular HTTP clusters 69 Avalanche Protection
103. palkan_tula palkan Isolate WebSocket and regular HTTP clusters AnyCable: disconnect-less deploys 69 Avalanche Protection
104. palkan_tula palkan Isolate WebSocket and regular HTTP clusters AnyCable: disconnect-less deploys AnyCable: resumable sessions 69 Avalanche Protection
105. See you on the other Cable
106. palkan_tula palkan 71 Map of AnyCable Protocol+ AnyCable-Go I/O poll Rails Executor Connection Channel Cache Session Cable Web Socket Channel Monitor Goroutine Pools Sub HTTP #connect #subscribed #disconnect #unsubscribed broadcast Encoder Transport Hub Sub Channel Channel Logger gRPC Pool RPC Controller Read Chan Write Chan Broadcaster Broker Hub Pub/Sub Shard Shard subscribe OPEN DATA CLOSE
107. Thank You Slides: evilmartians.com/events Twitter: @palkan_tula, @evilmartians, @any_cable