Kaigi On Rails

Transcript

1. @nateberkopec for Kaigi on Rails 2022 JA text on slides by @cafedomancer Queues in Rails Apps Sidekiq, Puma, and the GVL Rails ΞϓϦ έʔγϣϯʹ ͓͚ΔΩϡʔ
2. Hi Nate, Our application is really slow, but our dashboards look OK. We don't really understand what's going on. Can you help us? Nate ༷ ͍ͭ΋͓ੈ࿩ʹͳ͓ͬͯΓ·͢ɻ ฐࣾͰӡ༻͍ͯ͠ΔΞϓϦέʔ γϣϯʹύϑΥʔϚϯε্ͷ໰୊ ͕͋Γɺ͓٬༷͔Βෆຬͷ੠͕د ͤΒΕ͍ͯ·͢ɻμογϡϘʔυ ΛݟΔݶΓͰ͸ɺಛʹ໰୊͸ݟ౰ ͨΓ·ͤΜͰͨ͠ɻฐࣾͷํͰ ͸ɺͲͷΑ͏ͳ໰୊͕ى͖͍ͯΔ ͷ͔Λ೺ѲͰ͖͓ͯΓ·ͤΜɻ͓ ๩͍͠ͱ͜ΖେมڪॖͳͷͰ͢
3. Queues Ωϡʔ
5. Nate Berkopec
6. Agenda Three kinds of queues • Sidekiq • Puma/Unicorn • GVL 3 ͭͷछྨͷΩϡʔ ΞδΣϯμ
7. Queue Ωϡʔ “Servers” αʔόʔ Work ࡞ۀ
8. Queues increase availability Ωϡʔ͸Մ༻ੑΛ޲্ͤ͞Δ
9. Christoph Roser/allaboutlean.com
11. Queues can be organized in a “queueing system”. Ωϡʔ͸ʮΩϡʔΠϯάγεςϜʯʹ Αͬͯߏ੒͞ΕΔ
12. Total Time = Queue Time + Service Time ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ
13. Total Time = Queue Time + Service Time ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ
14. Sidekiq
15. Queues (Redis) Process types, with threads ϓϩηεʮλΠϓʯͱεϨου Ωϡʔ
17. Total Time = Queue Time + Service Time ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ
18. Important Point / ॏཁͳϙΠϯτ Add queue time instrumentation for Sidekiq Ωϡʔ࣌ؒͷܭଌπʔϧΛ Sidekiq ʹಋೖ͠Α͏
19. Queues (Redis) Process types, with threads ϓϩηεʮλΠϓʯͱεϨου Ωϡʔ
20. Domain Driven Queues •invoices •mailers •payments υϝΠϯۦಈΩϡʔ •੥ٻॻ •ϝʔϧ •ࢧ෷͍
23. Every Job Class has an “SLA” ͢΂ͯͷδϣϒΫϥεʹ͸ SLA ͕͋Δ
24. within_30_seconds within_5_minutes within_1_hour within_1_week
25. “Shards” w/ Consistent Hashing ίϯγεςϯτ ϋογϡ๏ʹΑΔ γϟʔυ
26. Predicted Latency = Current rate of job processing x # of Jobs in Queue ༧૝͞ΕΔϨΠςϯγʔ = ݱࡏͷδϣϒॲཧ཰ x Ωϡʔͷδϣϒ਺
27. https://tinyurl.com/kellysidekiq
28. TODO 1. Arrange queues based on SLAs 2. Create alerts based on queue SLAs 1. SLA ʹج͍ͮ ͯΩϡʔΛ 
 ഑ஔ͠Α͏ 2. SLA ʹج͍ͮ ͯΞϥʔτΛ 
 ࡞੒͠Α͏
29. Web
30. Total Time = Queue Time + Service Time ߹ܭ࣌ؒ = Ωϡʔ࣌ؒ + αʔϏε࣌ؒ
34. Header: X-Request-Start
36. More queue time? 1 worker 4 workers Ωϡʔ࣌ؒͳ͕௕ ͍ͷ͸ͲΕ͔ʁ
37. 4 pods 1 worker 1 pod 4 workers More queue time? Ωϡʔ࣌ؒͳ͕௕ ͍ͷ͸ͲΕ͔ʁ
38. Queue time = 1/s
39. 1 master process per container ίϯςφ͋ͨΓ 1 Ϛελʔϓϩηε
41. Always use at least 4 child processes/workers ࠷௿Ͱ΋ 4 ͭͷϓϩηε/ϫʔΧʔΛ࢖༻͠Α͏
42. TODO 1. Measure web request queueing 2. Set up autoscaling based on this 3. Use at least 4 child processes 1. Web ϦΫΤετ ͷΩϡʔΠϯά Λܭଌ͠Α͏ 2. ্هʹج͍ͮͯ ΦʔτεέʔϦ ϯάΛ 
 ઃఆ͠Α͏ 3. গͳ͘ͱ΋ 4 ͭ ͷϓϩηεΛ 
 ࢖༻͠Α͏
43. Global VM Lock (GVL)
44. Great Valuable Lock Ractor VM Lock
46. 1.Exit 2.I/O 3.Interrupt (100ms)
49. 100 Threads?
50. # of Threads Service Time
52. tinyurl.com/gvlruby
53. wait_for_less_busy_worker tinyurl.com/pumasleep
54. Worker 1: 0/5 Threads Busy Worker 2: 1/5 Threads Busy Worker 3: 4/5 Threads Busy
55. TODO 1. Set thread count to 5 (web) or 10 (Sidekiq) 2. Monitor service time under load 3. Upgrade to Puma 6 1. εϨου਺Λ 5 (web) ·ͨ͸ 10 (Sidekiq) ʹ 
 ઃఆ͠Α͏ 2. ෛՙ࣌ͷαʔϏε࣌ ؒΛ 
 ϞχλϦϯά͠Α͏ 3. Puma 6 ʹόʔδϣϯ 
 Ξοϓ͠Α͏
56. TODO 1. Arrange queues based on SLAs 2. Create alerts based on queue SLAs 3. Measure web request queueing 4. Set up autoscaling based on this 5. Use at least 4 child processes 6. Set thread count to 5 (web) or 10 (Sidekiq) 7. Monitor service time under load 8. Upgrade to Puma 6 1. SLA ʹج͍ͮͯΩϡʔΛ 
 ഑ஔ͠Α͏ 2. SLA ʹج͍ͮͯΞϥʔτΛ 
 ࡞੒͠Α͏ 3. Web ϦΫΤετͷΩϡʔΠϯ άΛܭଌ͠Α͏ 4. ্هʹج͍ͮͯΦʔτεέʔ ϦϯάΛઃఆ͠Α͏ 5. গͳ͘ͱ΋ 4 ͭͷϓϩηεΛ ࢖༻͠Α͏ 6. εϨου਺Λ 5 (web) ·ͨ͸ 10 (Sidekiq) ʹઃఆ͠Α͏ 7. ෛՙ࣌ͷαʔϏε࣌ؒΛϞχ λϦϯά͠Α͏ 8. Puma 6 ʹόʔδϣϯΞοϓ͠ Α͏
57. @nateberkopec for Kaigi on Rails 2022 JA text on slides by @cafedomancer Queues in Rails Apps Sidekiq, Puma, and the GVL Rails ΞϓϦ έʔγϣϯʹ ͓͚ΔΩϡʔ