$30 off During Our Annual Pro Sale. View Details »

Kaigi On Rails

Nate Berkopec
October 21, 2022
1.4k

Kaigi On Rails

Nate Berkopec

October 21, 2022
Tweet

Transcript

  1. @nateberkopec for Kaigi on Rails 2022
    JA text on slides by @cafedomancer
    Queues in
    Rails Apps
    Sidekiq, Puma, and the GVL
    Rails ΞϓϦ
    έʔγϣϯʹ
    ͓͚ΔΩϡʔ

    View Slide

  2. Hi Nate,


    Our application is really slow,
    but our dashboards look OK.
    We don't really understand
    what's going on.


    Can you help us?


    Nate ༷


    ͍ͭ΋͓ੈ࿩ʹͳ͓ͬͯΓ·͢ɻ


    ฐࣾͰӡ༻͍ͯ͠ΔΞϓϦέʔ
    γϣϯʹύϑΥʔϚϯε্ͷ໰୊
    ͕͋Γɺ͓٬༷͔Βෆຬͷ੠͕د
    ͤΒΕ͍ͯ·͢ɻμογϡϘʔυ
    ΛݟΔݶΓͰ͸ɺಛʹ໰୊͸ݟ౰
    ͨΓ·ͤΜͰͨ͠ɻฐࣾͷํͰ
    ͸ɺͲͷΑ͏ͳ໰୊͕ى͖͍ͯΔ
    ͷ͔Λ೺ѲͰ͖͓ͯΓ·ͤΜɻ͓
    ๩͍͠ͱ͜ΖେมڪॖͳͷͰ͢

    View Slide

  3. Queues


    Ωϡʔ

    View Slide

  4. View Slide

  5. Nate Berkopec

    View Slide

  6. Agenda
    Three kinds of queues
    • Sidekiq
    • Puma/Unicorn
    • GVL
    3 ͭͷछྨͷΩϡʔ
    ΞδΣϯμ

    View Slide

  7. Queue


    Ωϡʔ
    “Servers”


    αʔόʔ
    Work


    ࡞ۀ

    View Slide

  8. Queues increase availability


    Ωϡʔ͸Մ༻ੑΛ޲্ͤ͞Δ

    View Slide

  9. Christoph Roser/allaboutlean.com

    View Slide

  10. View Slide

  11. Queues can be organized in a “queueing system”.


    Ωϡʔ͸ʮΩϡʔΠϯάγεςϜʯʹ


    Αͬͯߏ੒͞ΕΔ

    View Slide

  12. Total Time = Queue Time + Service Time


    ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ

    View Slide

  13. Total Time = Queue Time + Service Time


    ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ

    View Slide

  14. Sidekiq

    View Slide

  15. Queues (Redis)
    Process types, with threads


    ϓϩηεʮλΠϓʯͱεϨου


    Ωϡʔ


    View Slide

  16. View Slide

  17. Total Time = Queue Time + Service Time


    ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ

    View Slide

  18. Important Point / ॏཁͳϙΠϯτ


    Add queue time instrumentation for Sidekiq


    Ωϡʔ࣌ؒͷܭଌπʔϧΛ Sidekiq ʹಋೖ͠Α͏


    View Slide

  19. Queues (Redis)
    Process types, with threads


    ϓϩηεʮλΠϓʯͱεϨου


    Ωϡʔ


    View Slide

  20. Domain Driven


    Queues
    •invoices


    •mailers


    •payments
    υϝΠϯۦಈΩϡʔ
    •੥ٻॻ


    •ϝʔϧ


    •ࢧ෷͍

    View Slide

  21. View Slide

  22. View Slide

  23. Every Job Class has an “SLA”


    ͢΂ͯͷδϣϒΫϥεʹ͸ SLA ͕͋Δ

    View Slide

  24. within_30_seconds


    within_5_minutes


    within_1_hour


    within_1_week

    View Slide

  25. “Shards” w/
    Consistent Hashing


    ίϯγεςϯτ
    ϋογϡ๏ʹΑΔ
    γϟʔυ

    View Slide

  26. Predicted Latency = Current rate of
    job processing x # of Jobs in Queue
    ༧૝͞ΕΔϨΠςϯγʔ =


    ݱࡏͷδϣϒॲཧ཰ x Ωϡʔͷδϣϒ਺


    View Slide

  27. https://tinyurl.com/kellysidekiq

    View Slide

  28. TODO
    1. Arrange
    queues
    based on
    SLAs


    2. Create alerts
    based on
    queue SLAs
    1. SLA ʹج͍ͮ
    ͯΩϡʔΛ

    ഑ஔ͠Α͏


    2. SLA ʹج͍ͮ
    ͯΞϥʔτΛ

    ࡞੒͠Α͏

    View Slide

  29. Web

    View Slide

  30. Total Time = Queue Time + Service Time


    ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ

    View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. Header: X-Request-Start

    View Slide

  35. View Slide

  36. More queue
    time?
    1 worker 4 workers
    Ωϡʔ࣌ؒͳ͕௕
    ͍ͷ͸ͲΕ͔ʁ

    View Slide

  37. 4 pods


    1 worker
    1 pod


    4 workers
    More queue
    time?
    Ωϡʔ࣌ؒͳ͕௕
    ͍ͷ͸ͲΕ͔ʁ

    View Slide

  38. Queue time = 1/s

    View Slide

  39. 1 master process per container


    ίϯςφ͋ͨΓ 1 Ϛελʔϓϩηε

    View Slide

  40. View Slide

  41. Always use at least 4 child processes/workers


    ࠷௿Ͱ΋ 4 ͭͷϓϩηε/ϫʔΧʔΛ࢖༻͠Α͏

    View Slide

  42. TODO
    1. Measure web
    request
    queueing


    2. Set up
    autoscaling
    based on this


    3. Use at least 4
    child
    processes
    1. Web ϦΫΤετ
    ͷΩϡʔΠϯά
    Λܭଌ͠Α͏


    2. ্هʹج͍ͮͯ
    ΦʔτεέʔϦ
    ϯάΛ

    ઃఆ͠Α͏


    3. গͳ͘ͱ΋ 4 ͭ
    ͷϓϩηεΛ

    ࢖༻͠Α͏

    View Slide

  43. Global VM Lock (GVL)

    View Slide

  44. Great Valuable Lock


    Ractor VM Lock

    View Slide

  45. View Slide

  46. 1.Exit


    2.I/O


    3.Interrupt (100ms)

    View Slide

  47. View Slide

  48. View Slide

  49. 100 Threads?

    View Slide

  50. # of Threads
    Service


    Time

    View Slide

  51. View Slide

  52. tinyurl.com/gvlruby

    View Slide

  53. wait_for_less_busy_worker
    tinyurl.com/pumasleep

    View Slide

  54. Worker 1: 0/5 Threads Busy


    Worker 2: 1/5 Threads Busy


    Worker 3: 4/5 Threads Busy

    View Slide

  55. TODO
    1. Set thread
    count to 5
    (web) or 10
    (Sidekiq)


    2. Monitor
    service time
    under load


    3. Upgrade to
    Puma 6
    1. εϨου਺Λ 5 (web)
    ·ͨ͸ 10 (Sidekiq) ʹ

    ઃఆ͠Α͏


    2. ෛՙ࣌ͷαʔϏε࣌
    ؒΛ

    ϞχλϦϯά͠Α͏


    3. Puma 6 ʹόʔδϣϯ

    Ξοϓ͠Α͏

    View Slide

  56. TODO
    1. Arrange queues
    based on SLAs


    2. Create alerts based
    on queue SLAs


    3. Measure web request
    queueing


    4. Set up autoscaling
    based on this


    5. Use at least 4 child
    processes


    6. Set thread count to 5
    (web) or 10 (Sidekiq)


    7. Monitor service time
    under load


    8. Upgrade to Puma 6
    1. SLA ʹج͍ͮͯΩϡʔΛ

    ഑ஔ͠Α͏


    2. SLA ʹج͍ͮͯΞϥʔτΛ

    ࡞੒͠Α͏


    3. Web ϦΫΤετͷΩϡʔΠϯ
    άΛܭଌ͠Α͏


    4. ্هʹج͍ͮͯΦʔτεέʔ
    ϦϯάΛઃఆ͠Α͏


    5. গͳ͘ͱ΋ 4 ͭͷϓϩηεΛ
    ࢖༻͠Α͏


    6. εϨου਺Λ 5 (web) ·ͨ͸
    10 (Sidekiq) ʹઃఆ͠Α͏


    7. ෛՙ࣌ͷαʔϏε࣌ؒΛϞχ
    λϦϯά͠Α͏


    8. Puma 6 ʹόʔδϣϯΞοϓ͠
    Α͏

    View Slide

  57. @nateberkopec for Kaigi on Rails 2022
    JA text on slides by @cafedomancer
    Queues in
    Rails Apps
    Sidekiq, Puma, and the GVL
    Rails ΞϓϦ
    έʔγϣϯʹ
    ͓͚ΔΩϡʔ

    View Slide