Goで実装された高速な 仮想待合室サーバの実装と詳解

ペパボのテックカンファレンスで話しました。

Transcript

1. GoͰ࣮૷͞Εͨߴ଎ͳ 
 Ծ૝଴߹ࣨαʔόͷ࣮૷ͱৄղ ʙے೑৯ಊɺૣ͘෱Ԭʹ΋Ͱ͖ͯ΄͍͠ΜͰ͚͢Ͳฤʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   
   
   
   
   
   ($1ɺ"JS fl PXɺ1VC4VCɺ%BUB fl PX
3. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
8. ౰೔݊΋͋Γ·͢ʂʂ1
9. αʔϏεͱࣄۀ෦੍  ˞ଞʹ΋ΤϯδχΞϦϯά෦໳ɺάϧʔϓɾνʔϜ͕͋Δ͕આ໌ͷ؆ུԽͷͨΊਤ͔Β͸লུ
   ٕज़෦
   ίʔϙϨʔτΤϯδχΞϦϯάάϧʔϓ
   ٕज़෦σʔλج൫νʔϜ
   $50ࣨ
   ݚڀ։ൃνʔϜ
   ࣛࣇౡΤϯδχΞϦϯάνʔϜ
   ηΩϡϦςΟରࡦࣨ
   σβΠϯࣨ
10. 43&ͱͯ͠ͷ੹຿Λෛ͍ͬͯΔϝϯόʔ 
11. ٕज़ελοΫ Platform Datastore Monitor On-premises,Google Cloud,AWS Orchastration OpenStack,Kubernetes RDS,MySQL,PostgreSQL Mackerel,Prometheus,Datadog https://tech.pepabo.com/tech-stack/
13. ࠓ೔࿩͢͜ͱ γεςϜߏ੒֓ཁ ൵ܶ Ծ૝଴߹࣮ࣨ૷ৄղ ๻ͷ༮গظͷՄѪ͍ࣸਅ100બ
14. γεςϜߏ੒֓ཁ OpenStack Managed LB Backend APIs nginx proxy role proxy roleͰෳ਺ͷछྨͷapp roleʹϓϩΩγ nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role
15. ൵ܶ
16. ൵ܶ͸ى͖΍͍͢ ͓٬༷ͷݶఆ঎඼ൢചͳͲʹΑΔେྔΞΫηε ΧʔτϘοτʹΑΔෆਖ਼ͳେྔΞΫηε DDOS߈ܸ
17. ൵ܶ͸
 ๷͛Δͷ͔ʁ
18. ΤΠοʂʂʂ
19. ʮϨʔτϦϛοτΛ
    ͔͚ΔͱͲ͏ͳΔʁʯ
20. ஌ΒΜͷ͔ʁ
21. ΤΠοʂʂʂ ঎඼͕
    ചΕͳ͘ͳΔͷͩ
22. ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ 1. େྔΞΫηε͕ൃੜ͢Δ 2. ϨʔτϦϛοτΛ͔͚Δ 3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ 4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍ 5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍ 6. ձࣾͷച্্͕͕Βͳ͍ 7. Զͷڅྉ্͕͕Βͳ͍
23. ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ 1. େྔΞΫηε͕ൃੜ͢Δ 2. ϨʔτϦϛοτΛ͔͚Δ 3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ 4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍ 5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍ 6. ձࣾͷച্্͕͕Βͳ͍ 7.Զͷڅྉ্͕͕Βͳ͍!!!௒ࠔΔ!!!
24. –Kazuhiko Yamashita 2023. “Ծ૝଴߹ࣨͱ͸څྉͰ͋Δ.”
25. Ծ૝଴߹ࣨ ઌணॱʹྲྀྔ੍ޚ͠ͳ͕Β
 αΠτ΁Ҋ಺͢Δ͜ͱͰ ߴෛՙ࣌Ͱ΋҆ఆͨ͠ΞΫηε͕Մೳ
26. Ծ૝଴߹ࣨ OpenStack Managed LB Backend APIs ଴߹ࣨAPIΛ։ൃͨ͠ nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role nginx app unicorn 
 php-fpm app role proxy role nginx waiting room
27. ৄղ଴߹ࣨγεςϜ
28. Ծ૝଴߹ࣨ֓ཁ proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room ڐՄϦετ Site A 200 Site B ... Site C ... ΫϥΠΞϯτʹ଴ͪ൪߸Λ࠾൪ͯ͠ɺڐՄϦετͷ൪߸ΑΓ
    খ͍͞൪߸Λ͍࣋ͬͯΕ͹Ұఆ࣌ؒΞΫηε͕Մೳ Client A ΫϥΠΞϯτ൪߸:100 ΞΫηεڐՄ Client B ΫϥΠΞϯτ൪߸:300 ଴߹ࣨ΁
29. ৄղ଴߹ࣨγεςϜ ଴߹ࣨىಈܖػ ଴ͪ൪߸ͷ࠾൪ ڐՄΫϥΠΞϯτͷ൑ఆ
30. ଴߹ࣨىಈܖػ location / { limit_req zone=example nodelay; limit_req_status 512; error_page 512 =200 @waitingroom; root /var/www/nginx; } ϨʔτϦϛοτʹ֘౰ͨ͠Β
 ಛఆͷΤϯυϙΠϯτʹ
 ϦμΠϨΫτ OHJOYͷSBUFMJNJU
    NPEVMFΛར༻͠ɺ
    ա৒ΞΫηεΛݕ஌
31. ଴߹ࣨىಈܖػ location @waitingroom { mruby_rewrite_handler enable_waitingroom.rb; proxy_pass origin_backend/$request_uri; internal; } location ~ ^/queues { proxy_pass waitingroom; internal; } ଴߹ࣨAPI͸mrubyͰ
 ίʔϧͯ͠ɺ଴߹ࣨΛ
 ىಈ͢Δ
32. ଴߹ࣨىಈܖػ 1.ϊϯϒϩοΩϯάʹϦΫΤετ 2.੍ݶதͰॱ൪଴ͪঢ়ଶͳΒ503 3.੍ݶ͞Εͯͳ͚Ε͹όοΫΤϯυ΁ϓϩΩγ
33. ngx_http_subrequest location / { # subrequest } location /subreqest { puts "hello" } # http://localhost/ => hello ͋ΔϩέʔγϣϯʹདྷͨϦΫΤετΛ
    ଞͷϩέʔγϣϯʹϦΫΤετͨ݁͠ՌͰ
    Ԡ౴͢ΔΑ͏ͳ࢖͍ํ
34. Nginx::Async::HTTP.sub_request ϦΫΤετΛ౤͛ΔͱϒϩοΫΛ։์͠ɺ
    Ϩεϙϯε࣌ʹίʔϧόοΫΛड͚औΓ࠶։ SFRVFTU SFRVFTU NSVCZ SFTQPOTF SFTQPOTF NSVCZ TVCSFRFTU 3FRVFTU 3FTQPOTF
35. ଴߹ࣨىಈܖػ location @waitingroom { mruby_rewrite_handler enable_waitingroom.rb; proxy_pass origin_backend/$request_uri; internal; } location ~ ^/queues { proxy_pass waitingroom; internal; } /queuesΛ࣮ߦ͍ͯ͠Δ
 API͸nginxΛϒϩοΫ͠ͳ͍
36. ଴߹ࣨඇ੍ݶ࣌ͷϦΫΤετભҠ Waiting Room nginx Client Origin 1.ϦΫΤετ 2.ىಈ͞Ε͍ͯΔ͔ʁ 3.ະىಈ͔
 ڐՄࡁΈΫϥΠΞϯτ 4.ϦΫΤετ 5.Ϩεϙϯε 6.Ϩεϙϯε
37. ଴߹੍ࣨݶ࣌ͷϦΫΤετભҠ Waiting Room nginx Client Origin 1.ϦΫΤετ 2.ىಈ͞Ε͍ͯΔ͔ʁ 3.ىಈத͔ͭɺ
 ະڐՄΫϥΠΞϯτ 4.503Ϩεϙϯε ଴߹ࣨىಈঢ়ଶͷͱ͖͸ϨʔτϦϛοτʹؔ܎ͳ͘ 
 Ұఆ࣌ؒɺ଴߹ࣨʹҊ಺͢Δ
38. ଴߹ࣨAPIͷ֓ཁ (PͰ࣮૷͞Εͨ)551αʔό https://github.com/pyama86/ngx_waitingroom go on-memory cache redis query query response response ଟ૚ͷߴ଎ͳσʔλετΞΛར༻ͯ͠ɺߴ଎ʹ࣮ߦ
39. ଴߹ࣨىಈ 4&5/9 &91*3& ;"%% ܭࢉྔ:O(1) Ωʔྫ: a.com
 SET if Not eXistsͷུͰɺΩʔ͕ଘࡏ͠ͳ͍৔߹ͷΈ஋Λઃఆ͢Δɻ Ωʔ͕ଘࡏ͢Δ৔߹ʹ1ɺΩʔ͕ͳ͍৔߹ʹ0Λฦ٫͢ΔͷͰɺϩοΫͷΑ͏ʹར༻Ͱ͖Δɻ BDPNͰ଴߹ࣨΛىಈ͢Δ৔߹ ܭࢉྔ: O(log(N)) Ωʔྫ: queue-domains
 ϝϯόʔ+είΞͰϋογϡͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻܭࢉྔ΋௿͘ɺॏෳഉআͰ͖ΔͷͰɺ ଴߹͕ࣨىಈ͍ͯ͠ΔυϝΠϯͷϦετʹར༻͍ͯ͠Δɻ ܭࢉྔ:O(1) Ωʔྫ: a.com
 TTLΛઃఆ͢ΔAPIɻυϝΠϯ໊ΛΩʔʹઃఆ͢Δ͜ͱͰɺ଴߹ࣨͷىಈ࣌ؒΛ TTLΛ༻੍͍ͯޚ͢Δ͜ͱ͕Ͱ͖Δɻྫ͑͹TTL=300ʹ͢Δͱ5෼ؒ͸଴߹͕ࣨىಈɻ
40. σʔλߏ଄ key value queue-domains [a.com, b.com, c.com] a.com 200 b.com 300 c.com 400 ͳΔ΂͘υϝΠϯΛى఺ʹ0  ͰΞΫηεͰ͖Δߏ଄ ZADD SETNX
41. ଴ͪ൪߸ͷ࠾൪
42. ଴ͪ൪߸ͷ࠾൪ Client 1.ϦΫΤετ 2.1001൪Λ෷͍ग़͠ waiting room ࠾൪:1000 ڐՄࡁΈ൪߸:200 
    ΫϥΠΞϯτ͸൪·Ͱ࠾൪͞Ε͍ͯΔ
    
    ൪·Ͱ͸ڐՄࡁΈͳͷͰɺΫϥΠΞϯτ͕଴ͪߦྻʹଘࡏ
    
    ൪Λ࣍ͷΫϥΠΞϯτʹ෷͍ग़͍ͨ͠
43. ଴ͪ൪߸ͷ࠾൪ Client ݱ୅ͷ8&#͸ଟॏͰϦΫΤετ͕ߦΘΕΔͨΊɺ
    ϦΫΤετ͝ͱͰ͸ͳ͘ɺΫϥΠΞϯτ͝ͱʹ 
 ޮ཰Α͘࠾൪͢Δʹ͸޻෉͕ඞཁ waiting room index.html main.css main.js main.jpg ޻෉͠ͳ͍ͱ1ΫϥΠΞϯτʹ 
 4ͭ࠾൪ͯ͠͠·͏
44. ଴ͪ൪߸ͷ࠾൪ ·ͣ͸࠾൪࣌ࠁΛ෷͍ग़͠ϖʔδʹϦμΠϨΫτͯ͠ɺ 
 KBWBTDSJQUͰ਺ඵ͓͖ʹϙʔϦϯά͠ 
 ࠾൪࣌ࠁʹͳͬͨΒ࠾൪͢Δ͜ͱͰଟॏ࠾൪Λ੍ݶ 1.index.html,main.cs.. 2.redirect 503.hml ΫϥΠΞϯτ൪߸: NULL ΫϥΠΞϯτID: NULL ࠾൪࣌ࠁ: 10ඵޙ waiting room client 3.̐ඵ͝ͱʹ503.html΁ϦΫΤετ ΫϥΠΞϯτ൪߸: 1 ΫϥΠΞϯτID: UUID ࠾൪࣌ࠁ: DONE 4.࠾൪
45. ଴ͪ൪߸ͷ࠾൪ ΫϥΠΞϯτͷঢ়ଶ͸҉߸Խͨ͠$PPLJFʹอଘ ΫϥΠΞϯτ൪߸ ΫϥΠΞϯτID ΫϥΠΞϯτ൪߸ൃߦ࣌ࠁ αʔόͰ࠾൪͍ͯ͠Δ௨͠൪߸ ΫϥΠΞϯτࣝผࢠ(UUID) ΫϥΠΞϯτ൪߸Λൃߦ͢Δ࣌ࠁɻUnixTimeͰ֨ೲ͞Ε͓ͯΓɺ γεςϜશମͰ࣌ࠁ͕େ͖ͣ͘ΕΔ͜ͱ͸ ૝ఆ͍ͯ͠ͳ͍
46. ଴ͪ൪߸ͷ࠾൪ */$3 &91*3& ܭࢉྔ O(1) Ωʔྫ:a.com
 Χ΢ϯλɻࢦఆͨ͠ΩʔΛΞτϛοΫʹΠϯΫϦϝϯτͰ͖Δɻ FYBNQMFDPNͰ଴߹ࣨΛىಈ͢Δ৔߹ ܭࢉྔ O(1) Ωʔྫ: a.com
 લड़ͷ௨Γɻ
47. ͜͜·Ͱ࿩ͨ͜͠ͱ • ଴߹ࣨ͸nginxͷϨʔτϦϛοτ౸ୡΛܖػʹىಈ͠ɺࢦఆͨ࣌ؒ͠ىಈ͠ଓ͚Δ • ΫϥΠΞϯτ͸ΫοΩʔʹΫϥΠΞϯτ൪߸΍ΫϥΠΞϯτIDΛ࣋ͭ • ΫϥΠΞϯτ൪߸ͷ෷͍ग़͠͸ɺRedisͷINCRͰΧ΢ϯτΞοϓͨ͠൪߸͕෷͍ग़͞ΕΔ ڐՄ൪߸Ϧετ a.com 200 b.com 300 c.com 400 ࠾൪Ϧετ a.com 500 b.com 600 c.com 700 αʔόαΠυσʔλετΞ ΫϥΠΞϯτCookie ΫϥΠΞϯτ൪߸ 100 ΫϥΠΞϯτID UUID ࠾൪࣌ࠁ 1986/04/10
48. ڐՄΫϥΠΞϯτͷ൑ఆ
49. ڐՄ൪߸ͷߋ৽ proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ҰͭͷڐՄϦετΛෳ਺ͷQSPYZ
    SPMF͕ࢀর͢Δ৔߹ʹɺ
    ͲͷΑ͏ʹߋ৽͢Δ͔ʁ 1෼͝ͱʹ200ΫϥΠΞϯτͣͭ ڐՄ͢Δͱ͍͏Α͏ͳ 
 ੍ޚΛ͍ͨ͠ 200→400→600....
50. զʑʹ͸goroutine͕͋Δ͡Όͳ͍͔ waiting room http server access controller ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ࢦఆִؒͰϦετΛࢀর͠ɺڐՄϦετΛߋ৽͢Δ goroutineΛىಈ͢Δ ߋ৽
51. ڐՄ൪߸Ϧετͷഉଞ੍ޚ waiting room http server access controller ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 SETNXΛ׆༻͢Δ͜ͱͰඞͣυϝΠϯ͋ͨΓͷ
 ߋ৽ϓϩηε͸ඞͣҰͭʹͳΔ waiting room http server access controller SETNX ok SETNX ng
52. ڐՄ൪߸ͷߋ৽ ;3"/(& &91*3& 4&5/9 ܭࢉྔ: O(log(N)+M) Ωʔྫ queue-domains
 ଴߹͕ࣨىಈ͞Ε͍ͯΔ͢΂ͯͷυϝΠϯΛϦετͰऔಘ͢Δɻ HPSPVUJOFͰࢦఆඵ਺ײִؒͰϧʔϓ͢Δ ܭࢉྔ:O(1) Ωʔྫ: a.com_lock ϩοΫΛऔΕͨproxy roleͷΈ͕ɺڐՄ൪߸Λߋ৽Մೳɻ ෳ਺ͷproxy role͕ಉ࣌ʹߋ৽͢Δ͜ͱΛഉଞ੍ޚ͢Δɻ ܭࢉྔ O(1) Ωʔྫ:a.com_lock ࣍ճͷڐՄ൪߸ߋ৽࣌ؒ·Ͱͷ஋ΛTTLʹઃఆ͢Δ͜ͱͰϩοΫ͕औΒΕͨ··ʹͳΔͨΊ ߋ৽ִؒΛTTLͰ੍ޚͰ͖Δɻ60ඵִؒͰڐՄ൪߸Λߋ৽͍ͨ͠ͳΒ60ඵΛઃఆɻ 4&5&9 ܭࢉྔ O(1)Ωʔྫ:a.com ݱࡏͷڐՄ൪߸ʹ௥ՃڐՄ͍ͨ͠ΫϥΠΞϯτ਺Λ௥Ճͯ͠ɺTTLͱͱ΋ʹॻ͖ࠐΈɻ
53. ΞΫηεڐՄ ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ΫϥΠΞϯτͷΞΫηεΛڐՄͨ͠৔߹͸ɺ ΫϥΠΞϯτIDΛΩʔʹRedisʹอଘ͠ɺ TTLͷظؒΞΫηεڐՄ Site A ΫϥΠΞϯτ൪߸:100 ΫϥΠΞϯτID: UUID1 waiting room ࢀর Ωʔ TTL UUID1 600 ॻ͖ࠐΈ Client
54. ੑೳධՁ
55. Cookie࢖͑ΔϕϯνϚʔΧʔॻ͍ͨ https://github.com/pyama86/ngx-smart-ratelimit/tree/master/bench CPU Apple M1 MAX ϝϞϦ 64G 10ฒྻ 10,000ϦΫΤετ keepaliveͳ͠ ଌఆ؀ڥ ݁Ռ 5000 request / sec ฏۉ0.2msecͰԠ౴Մೳ
56. ੩తίϯςϯπ഑৴ͱͷൺֱ ݁Ռ 8000 request / sec ฏۉ0.125msec 0.125msec/request ÷ 0.2msec/request = 1.6
57. ·ͱΊ
58. ΋͏͋ͱ͸ҿञͷΈɺ ҿञͦ͜͢΂ͯʂʂʂ
59. શํҐ࠾༻͍ͯ͠·͢ɻ 
 ͜ͷ͋ͱҿञͨ͠ΒɺԿͰ΋࿩͠·͢ɻ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU