Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[RailsWorld 2023] Untangling cables & demystifying twisted transistors

[RailsWorld 2023] Untangling cables & demystifying twisted transistors

Video: https://www.youtube.com/watch?v=GrHpop5HtxM

---

More and more Rails applications adopt real-time features, and it’s not surprising—Action Cable and Hotwire brought development experience to the next level regarding dealing with WebSockets. You need zero knowledge of the underlying tech to start crafting a new masterpiece of web art! However, you will need this knowledge later to deal with ever-sophisticated feature requirements and security and scalability concerns.

The variety of questions that arise when developers work with Rails’ real-time tooling is broad, from “Which delivery guarantees does Action Cable provide?” to “Can I scale my Hotwire application to handle dozens of thousands of concurrent users?”. To answer them, we need to learn our tools first.

In my talk, I will help you better to understand Rails’ real-time component—Action Cable. I want to open this black box for you and sort through the internals so you can work with Action Cable efficiently and confidently.

Vladimir Dementyev

October 06, 2023
Tweet

More Decks by Vladimir Dementyev

Other Decks in Programming

Transcript

  1. Vladimir Dementyev
    Evil Martians
    Untangling Cables &
    Demystifying Twisted Transistors
    2023
    1. It's Rails
    2. Freak on a Cable
    3. Y'all Want a Stream
    4. Make Me ping
    5. Wire Tongue
    6. Threaded Transistor
    7. Throw me & Forget
    8. Turbo Streams Everywhere
    9. (Not) Alone I Reconnect
    Bonus: See You on the Other Cable
    Lyrics by Vladimir "Palkan" Dementyev
    Produced by Evil Martians, inc.

    View full-size slide

  2. 1. It's Rails

    View full-size slide

  3. Rails is a
    software
    development
    philosophy
    3 palkan_tula
    palkan

    View full-size slide

  4. Conceptual
    Compression
    4 palkan_tula
    palkan

    View full-size slide

  5. Unboxing time!
    5 palkan_tula
    palkan

    View full-size slide

  6. Unboxing time!
    5 palkan_tula
    palkan

    View full-size slide

  7. 2. Freak on a Cable

    View full-size slide

  8. github.com/palkan

    View full-size slide

  9. github.com/palkan
    evilmartians.com

    View full-size slide

  10. github.com/palkan
    evilmartians.com
    anycable.io

    View full-size slide

  11. The Book on
    abstraction
    layers from and
    for Rails
    8 palkan_tula
    palkan

    View full-size slide

  12. 3. Y'all Want a Stream

    View full-size slide

  13. palkan_tula
    palkan
    Channels Layer
    10
    class ChatChannel < ApplicationCable::Channel
    def subscribed
    @room = Room.find_by(id: params[:id])
    return reject unless @room
    stream_for @room
    end
    def speak(data)
    broadcast_to(@room,
    event: "newMessage", user_id: user.id, body: data.fetch("body")
    )
    end
    end

    View full-size slide

  14. 11 palkan_tula
    palkan
    class ChatChannel < ApplicationCable::Channel
    def subscribed
    @room = Room.find_by(id: params[:id])
    return reject unless @room
    stream_for @room
    end
    def speak(data)
    broadcast_to(@room,
    event: "newMessage", user_id: user.id,
    body: data.fetch("body")
    )
    end
    end
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    let sub = cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )
    sub.perform("speak", {body: "Hoi!"})

    View full-size slide

  15. palkan_tula
    palkan
    13
    Protocol
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Pub/Sub
    Subscriber
    Map
    Client
    Socket
    Consumer
    Web
    Socket
    Sub
    Pubsub
    Adapter
    Monitor
    Channel Channel
    Worker Pool
    Internal Pool
    Sub Sub
    Map of
    Action Cable
    HTTP
    rack.hijack
    #connect
    #subscribed
    { command: "subscribe" }
    OPEN
    { command: "message" }
    #receive
    CLOSE
    #disconnect
    #unsubscribed
    #stream_from
    #subscribe
    #unsubscribe
    broadcast
    #invoke_callback
    #send

    View full-size slide

  16. 4. Make Me Ping

    View full-size slide

  17. palkan_tula
    palkan
    @rails/actioncable
    15
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )

    View full-size slide

  18. palkan_tula
    palkan
    @rails/actioncable
    16
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    cable.subscriptions.create(
    {channel: "ChatChannel", id: 2023},
    {
    received({user_id, body}) {
    console.log(`Message from ${user_id}:${body}`);
    }
    }
    )
    ~ WebSocket connection
    Abstraction to subscribe for
    and receive updates

    View full-size slide

  19. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    17
    @rails/actioncable

    View full-size slide

  20. palkan_tula
    palkan
    github.com/le0pard/cable-shared-worker
    18

    View full-size slide

  21. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    (multiplexing)
    19
    @rails/actioncable

    View full-size slide

  22. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    (multiplexing) — how many?
    20
    @rails/actioncable

    View full-size slide

  23. We may* have
    as many
    subscriptions
    as canals in
    Amsterdam
    21 palkan_tula
    palkan
    * Doesn't mean we should though
    Consumer
    Monitor
    Sub Sub Sub
    Web
    Socket
    Sub Sub

    View full-size slide

  24. palkan_tula
    palkan
    @rails/actioncable
    22
    import { createConsumer } from "@rails/actioncable"
    let cable = createConsumer()
    let a = cable.subscriptions.create("UserChannel") // SUBSCRIBE
    let b = cable.subscriptions.create("UserChannel") //
    a.unsubscribe() //
    b.unsubscribe() // UNSUBSCRIBE

    View full-size slide

  25. palkan_tula
    palkan
    One consumer per session (e.g., a
    browser tab)
    Many subscriptions per consumer
    Monitor to keep consumer connected
    23
    @rails/actioncable

    View full-size slide

  26. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    stale timer

    View full-size slide

  27. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    ping
    stale timer

    View full-size slide

  28. palkan_tula
    palkan
    Keep alive
    24
    Consumer
    Monitor
    Web
    Socket
    stale timer reconnect

    View full-size slide

  29. palkan_tula
    palkan
    Action Cable Pings
    Help to identify broken TCP connections
    (from both sides, but not always)
    25

    View full-size slide

  30. palkan_tula
    palkan
    Action Cable Pings
    Help to identify broken TCP connections
    (from both sides, but not always)
    Hardly configurable (and may be too
    frequent to cause battery drain for mobile
    devices)
    25

    View full-size slide

  31. 26 palkan_tula
    palkan
    # config/application.rb
    # Re-define internal constant
    module ActionCable::Server::Connections
    BEAT_INTERVAL = 10
    end
    import { ConnetionMonitor } from "@rails/actioncable"
    // Adjust stale timeout (2xBEAT_INTERVAL)
    ConnectionMonitor.staleThreshold = 20

    View full-size slide

  32. palkan_tula
    palkan
    More clients
    github.com/anycable/anycable-client
    27

    View full-size slide

  33. palkan_tula
    palkan
    More clients
    github.com/anycable/anycable-client

    27

    View full-size slide

  34. 5. Wire Tongue

    View full-size slide

  35. Protocol is a
    language in
    which the
    server and
    client speak to
    each other
    29 palkan_tula
    palkan
    Protocol
    Web Server
    Consumer
    Web
    Socket
    HTTP

    View full-size slide

  36. palkan_tula
    palkan
    docs.anycable.io
    30

    View full-size slide

  37. 31 palkan_tula
    palkan
    {
    "command":"subscribe",
    "identifier":"{\"channel\" \"UserChannel\"}"
    }
    {
    "command":"message",
    "identifier":"{\"channel\":\"UserChannel\"}",
    "data":"{\"action\":\"speak\",\"text\":\"hoi\"}"
    }
    {
    "command":"unsubscribe",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {"type":"welcome"}
    {
    "type":"confirm_subscription",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {
    "type":"reject_subscription",
    "identifier":"{\"channel\":\"UserChannel\"}"
    }
    {
    "identifier":"{\"channel\":\"UserChannel\"}",
    "message":"{\"text\":\"hoi\"}"
    }
    {"type":"ping"}
    {
    "type":"disconnect",
    "reason":"unauthorized",
    "reconnect":false
    }
    No type!

    View full-size slide

  38. palkan_tula
    palkan
    Protocol is incomplete
    No IDs (session, message)
    No perform ACKs
    No error handling
    32

    View full-size slide

  39. palkan_tula
    palkan
    33
    let cable = createConsumer()
    for(let i=0; i<10; i++) {
    let sub = cable.subscriptions.create("UserChannel")
    sub.unsubscribe()
    }
    // Will I be subscribed or not?
    cable.subscriptions.create("UserChannel")

    View full-size slide

  40. palkan_tula
    palkan
    github.com/palkan/wsdirector
    - loop:
    multiplier: 10
    actions:
    - send:
    data: &sub
    command: "subscribe"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    - send:
    data:
    command: "unsubscribe"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    - send:
    <<: *sub
    - send:
    data:
    command: "message"
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    data: "{\"action\":\"echo\",\"test\":42}"
    - receive:
    data:
    identifier: "{\"channel\":\"BenchmarkChannel\"}"
    message: {action: "echo", test: 42}
    34

    View full-size slide

  41. 36
    RuntimeError - Unable to find subscription with identifier: {"channel":"BenchmarkChannel"}
    RuntimeError - Already subscribed to {"channel":"BenchmarkChannel"}

    View full-size slide

  42. palkan_tula
    palkan
    Solution?
    37

    View full-size slide

  43. 38
    (First rows only note) Some random Rails application

    View full-size slide

  44. palkan_tula
    palkan
    39
    let cable = createConsumer()
    for(let i=0; i<10; i++) {
    let sub = cable.subscriptions.create("UserChannel")
    sub.unsubscribe()
    }
    // Will I be subscribed or not? It depends. Why so?
    cable.subscriptions.create("UserChannel")

    View full-size slide

  45. 6. Threaded Transistor

    View full-size slide

  46. palkan_tula
    palkan
    41
    Web
    Server
    Web
    Socket
    HTTP
    Action Cable
    Executor

    View full-size slide

  47. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    Web
    Socket
    HTTP
    rack.hijack
    Action Cable
    Executor

    View full-size slide

  48. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Client
    Socket
    Web
    Socket
    HTTP
    rack.hijack
    Action Cable
    Executor

    View full-size slide

  49. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    OPEN
    Action Cable
    Executor

    View full-size slide

  50. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    OPEN
    Action Cable
    Executor

    View full-size slide

  51. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    OPEN
    Action Cable
    Executor

    View full-size slide

  52. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    {command: "subscribe"}
    OPEN
    #subscribed
    Action Cable
    Executor

    View full-size slide

  53. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    {command: "subscribe"}
    OPEN
    {command: "message"}
    #receive
    #subscribed
    Action Cable
    Executor

    View full-size slide

  54. palkan_tula
    palkan
    41
    Web
    Server
    Action Cable
    Server
    I/O
    loop
    Rails
    Executor
    Connection
    Channel
    Client
    Socket
    Web
    Socket
    Worker Pool
    HTTP
    rack.hijack
    #connect
    #unsubscribed
    {command: "subscribe"}
    OPEN
    {command: "message"}
    #receive
    CLOSE
    #disconnect
    #subscribed
    Action Cable
    Executor

    View full-size slide

  55. palkan_tula
    palkan
    Action Cable Server
    Thread pool executor (4 threads by default)
    Must be taken into account for shared
    resources (e.g., database pool)
    Uses Rails executor to cleanup after work
    42

    View full-size slide

  56. 43 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    Current.account = user.account
    end
    end

    View full-size slide

  57. 44 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    # BAD — will be reset at the end
    # of the #handle_open call
    Current.account = user.account
    end
    end

    View full-size slide

  58. class Connection
    identified_by :user
    before_command do
    # GOOD — set for every command
    Current.account = user.account
    end
    def connect
    self.user = find_verified_user
    end
    end
    44 palkan_tula
    palkan
    class Connection
    identified_by :user
    def connect
    self.user = find_verified_user
    # BAD — will be reset at the end
    # of the #handle_open call
    Current.account = user.account
    end
    end

    View full-size slide

  59. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    45

    View full-size slide

  60. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    Incoming commands
    45

    View full-size slide

  61. palkan_tula
    palkan
    Units of Work
    Lifecycle callbacks (connect, disconnect)
    Incoming commands
    Outgoing messages
    45

    View full-size slide

  62. 7. Throw me & Forget

    View full-size slide

  63. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    Pub/Sub

    View full-size slide

  64. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    Pub/Sub

    View full-size slide

  65. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #broadcast
    Pub/Sub

    View full-size slide

  66. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #broadcast
    Pub/Sub

    View full-size slide

  67. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #broadcast
    #stream_from callback
    is executed
    Pub/Sub

    View full-size slide

  68. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    broadcast message
    #invoke_callback
    #send
    #broadcast
    #transmit
    #stream_from callback
    is executed
    Pub/Sub

    View full-size slide

  69. palkan_tula
    palkan
    47
    Rails
    Executor
    Connection
    Pub/Sub
    Subscriber
    Map Client
    Socket
    Pubsub
    Adapter
    Channel
    Worker Pool
    Internal Pool
    Pub/Sub
    at-most once

    View full-size slide

  70. 48
    The pitfalls of realtime-ification, RailsConf, 2022

    View full-size slide

  71. palkan_tula
    palkan
    Yet Another Pitfall
    49
    100.times {
    ActionCable.server.broadcast "test", {text: "Count: #{_1}"}
    }
    # Will a client receive 1,2,3,4,...,99 in order?

    View full-size slide

  72. palkan_tula
    palkan
    And One More
    52
    def subscribed
    stream_from "test"
    # Q: Will this client receive the message?
    ActionCable.server.broadcast "test", {text: "Welkom!"}
    end

    View full-size slide

  73. Are you tired of
    threads?
    53 palkan_tula
    palkan
    I am

    View full-size slide

  74. 8. Turbo Streams Everywhere

    View full-size slide

  75. palkan_tula
    palkan
    Turbo Streams z
    Automatic subscriptions via HTML
    ()
    Built-in Turbo::StreamsChannel
    Zero application code to connect cables
    55

    View full-size slide

  76. palkan_tula
    palkan
    Turbo Streams
    56
    www.my-hot.app

    View full-size slide

  77. palkan_tula
    palkan
    Turbo Streams
    54
    www.my-hot.app

    SUBSCRIBE
    Connection
    Channel

    View full-size slide

  78. palkan_tula
    palkan
    DOM
    update
    Turbo Streams
    54
    www.my-hot.app

    HTML
    Action Cable broadcast
    from anywhere
    Connection
    Channel

    View full-size slide

  79. palkan_tula
    palkan
    UNSUBSCRIBE
    Turbo Streams
    54
    www.my-hot.app
    Connection

    View full-size slide

  80. palkan_tula
    palkan
    Turbo Streams
    Prone to subscribe/unsubscribe race
    conditions
    Managed in a decentralized way (via
    HTML partials)
    57

    View full-size slide

  81. 59
    anycable.substack.com

    View full-size slide

  82. 9. (Not) Alone I ReConnect

    View full-size slide

  83. palkan_tula
    palkan
    Server Restart
    N clients re-connect
    N clients re-subscribe each to M channels each
    Threaded executor's work queue is:
    (N+1)*M
    61

    View full-size slide

  84. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62

    View full-size slide

  85. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62
    Thundering
    Herd

    View full-size slide

  86. palkan_tula
    palkan
    Server Restart
    Threaded executor's work queue is:
    (N+1)*M
    Server can not keep up with confirmations,
    clients re-issue subscribe commands—!
    62
    Connection
    Avalanche

    View full-size slide

  87. Connection
    avalanches lead
    to significant
    load spikes
    63 palkan_tula
    palkan

    View full-size slide

  88. How to protect
    Rails from
    connection
    avalanches?
    64 palkan_tula
    palkan

    View full-size slide

  89. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    65

    View full-size slide

  90. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    Merge subscriptions
    67

    View full-size slide

  91. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    Merge subscriptions
    Avoid slow calls in #subscribed
    67

    View full-size slide

  92. palkan_tula
    palkan
    Avalanche Protection
    Reconnect with backoff & jitter
    Merge subscriptions
    Avoid slow calls in #subscribed
    Serialize subscribe commands
    67

    View full-size slide

  93. palkan_tula
    palkan
    68
    import { createCable } from "@anycable/web"
    let cable = createCable({
    concurrentSubscribes: false
    })

    View full-size slide

  94. palkan_tula
    palkan
    Isolate WebSocket and regular HTTP
    clusters
    69
    Avalanche Protection

    View full-size slide

  95. palkan_tula
    palkan
    Isolate WebSocket and regular HTTP
    clusters
    AnyCable: disconnect-less deploys
    69
    Avalanche Protection

    View full-size slide

  96. palkan_tula
    palkan
    Isolate WebSocket and regular HTTP
    clusters
    AnyCable: disconnect-less deploys
    AnyCable: resumable sessions
    69
    Avalanche Protection

    View full-size slide

  97. See you on the other Cable

    View full-size slide

  98. palkan_tula
    palkan
    71
    Map of
    AnyCable
    Protocol+
    AnyCable-Go
    I/O
    poll
    Rails
    Executor
    Connection
    Channel
    Cache
    Session
    Cable
    Web
    Socket
    Channel
    Monitor
    Goroutine Pools
    Sub
    HTTP
    #connect
    #subscribed
    #disconnect
    #unsubscribed
    broadcast
    Encoder
    Transport
    Hub
    Sub
    Channel
    Channel
    Logger
    gRPC Pool
    RPC
    Controller
    Read Chan
    Write Chan
    Broadcaster
    Broker
    Hub
    Pub/Sub
    Shard
    Shard
    subscribe
    OPEN
    DATA
    CLOSE

    View full-size slide

  99. Thank You
    Slides: evilmartians.com/events
    Twitter: @palkan_tula, @evilmartians,
    @any_cable

    View full-size slide