Goで実装された高速な 仮想待合室サーバの実装と詳解

ペパボのテックカンファレンスで話しました。

Transcript

1. GoͰ࣮૷͞Εͨߴ଎ͳ
   

   Ծ૝଴߹ࣨαʔόͷ࣮૷ͱৄղ
   ʙے೑৯ಊɺૣ͘෱Ԭʹ΋Ͱ͖ͯ΄͍͠ΜͰ͚͢Ͳฤʙ
   

   View Slide

2. ࢁԼ࿨඙!QZBNB
   (.0ϖύϘٕज़ج൫νʔϜ
   γχΞɾϓϦϯγύϧ
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   ($1ɺ"JS
   fl
   PXɺ1VC4VCɺ%BUB
   fl
   PX
   

   View Slide

3. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. ౰೔݊΋͋Γ·͢ʂʂ1
   

   View Slide

9. αʔϏεͱࣄۀ෦੍
   
   ˞ଞʹ΋ΤϯδχΞϦϯά෦໳ɺάϧʔϓɾνʔϜ͕͋Δ͕આ໌ͷ؆ུԽͷͨΊਤ͔Β͸লུ
   ٕज़෦ίʔϙϨʔτΤϯδχΞϦϯάάϧʔϓ ٕज़෦σʔλج൫νʔϜ
   $50ࣨݚڀ։ൃνʔϜ ࣛࣇౡΤϯδχΞϦϯάνʔϜ
   ηΩϡϦςΟରࡦࣨ σβΠϯࣨ
   

   View Slide

10. 43&ͱͯ͠ͷ੹຿Λෛ͍ͬͯΔϝϯόʔ
    
    

    View Slide

11. ٕज़ελοΫ
    Platform
    Datastore
    Monitor
    On-premises,Google Cloud,AWS
    Orchastration
    OpenStack,Kubernetes
    RDS,MySQL,PostgreSQL
    Mackerel,Prometheus,Datadog
    https://tech.pepabo.com/tech-stack/
    

    View Slide

12. View Slide

13. ࠓ೔࿩͢͜ͱ
    γεςϜߏ੒֓ཁ
    
    ൵ܶ
    
    Ծ૝଴߹࣮ࣨ૷ৄղ
    
    ๻ͷ༮গظͷՄѪ͍ࣸਅ100બ
    
    

    View Slide

14. γεςϜߏ੒֓ཁ
    OpenStack
    
    
    Managed LB
    Backend
    
    
    APIs
    nginx
    proxy role
    proxy roleͰෳ਺ͷछྨͷapp roleʹϓϩΩγ
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    

    View Slide

15. ൵ܶ
    

    View Slide

16. ൵ܶ͸ى͖΍͍͢
    ͓٬༷ͷݶఆ঎඼ൢചͳͲʹΑΔେྔΞΫηε
    
    ΧʔτϘοτʹΑΔෆਖ਼ͳେྔΞΫηε
    
    DDOS߈ܸ
    
    

    View Slide

17. ൵ܶ͸

    ๷͛Δͷ͔ʁ
    

    View Slide

18. ΤΠοʂʂʂ
    

    View Slide

19. ʮϨʔτϦϛοτΛ
    ͔͚ΔͱͲ͏ͳΔʁʯ
    

    View Slide

20. ஌ΒΜͷ͔ʁ
    

    View Slide

21. ΤΠοʂʂʂ
    ঎඼͕
    ചΕͳ͘ͳΔͷͩ
    

    View Slide

22. ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ
    1. େྔΞΫηε͕ൃੜ͢Δ
    
    2. ϨʔτϦϛοτΛ͔͚Δ
    
    3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ
    
    4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍
    
    5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍
    
    6. ձࣾͷച্্͕͕Βͳ͍
    
    7. Զͷڅྉ্͕͕Βͳ͍
    

    View Slide

23. ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ
    1. େྔΞΫηε͕ൃੜ͢Δ
    
    2. ϨʔτϦϛοτΛ͔͚Δ
    
    3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ
    
    4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍
    
    5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍
    
    6. ձࣾͷച্্͕͕Βͳ͍
    
    7.Զͷڅྉ্͕͕Βͳ͍!!!௒ࠔΔ!!!
    

    View Slide

24. –Kazuhiko Yamashita 2023.
    “Ծ૝଴߹ࣨͱ͸څྉͰ͋Δ.”
    

    View Slide

25. Ծ૝଴߹ࣨ
    ઌணॱʹྲྀྔ੍ޚ͠ͳ͕Β

    αΠτ΁Ҋ಺͢Δ͜ͱͰ
    ߴෛՙ࣌Ͱ΋҆ఆͨ͠ΞΫηε͕Մೳ
    

    View Slide

26. Ծ૝଴߹ࣨ
    OpenStack
    
    
    Managed LB
    Backend
    
    
    APIs
    ଴߹ࣨAPIΛ։ൃͨ͠
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    nginx
    app
    
    
    unicorn
    

    php-fpm
    app role
    proxy role
    nginx
    waiting room
    

    View Slide

27. ৄղ଴߹ࣨγεςϜ
    

    View Slide

28. Ծ૝଴߹ࣨ֓ཁ
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    ڐՄϦετ
    Site A 200
    Site B ...
    Site C ...
    ΫϥΠΞϯτʹ଴ͪ൪߸Λ࠾൪ͯ͠ɺڐՄϦετͷ൪߸ΑΓ
    খ͍͞൪߸Λ͍࣋ͬͯΕ͹Ұఆ࣌ؒΞΫηε͕Մೳ
    Client A
    ΫϥΠΞϯτ൪߸:100
    ΞΫηεڐՄ
    Client B
    ΫϥΠΞϯτ൪߸:300
    ଴߹ࣨ΁
    

    View Slide

29. ৄղ଴߹ࣨγεςϜ
    ଴߹ࣨىಈܖػ
    
    ଴ͪ൪߸ͷ࠾൪
    
    ڐՄΫϥΠΞϯτͷ൑ఆ
    

    View Slide

30. ଴߹ࣨىಈܖػ
    location / {
    
    
    limit_req zone=example nodelay;
    
    
    limit_req_status 512;
    
    
    error_page 512 =200 @waitingroom;
    
    
    root /var/www/nginx;
    
    
    }
    
    
    ϨʔτϦϛοτʹ֘౰ͨ͠Β

    ಛఆͷΤϯυϙΠϯτʹ

    ϦμΠϨΫτ
    OHJOYͷSBUFMJNJUNPEVMFΛར༻͠ɺ
    ա৒ΞΫηεΛݕ஌
    

    View Slide

31. ଴߹ࣨىಈܖػ
    location @waitingroom {
    
    
    mruby_rewrite_handler enable_waitingroom.rb;
    
    
    proxy_pass origin_backend/$request_uri;
    
    
    internal;
    
    
    }
    
    
    location ~ ^/queues {
    
    
    proxy_pass waitingroom;
    
    
    internal;
    
    
    }
    ଴߹ࣨAPI͸mrubyͰ

    ίʔϧͯ͠ɺ଴߹ࣨΛ

    ىಈ͢Δ
    

    View Slide

32. ଴߹ࣨىಈܖػ
    1.ϊϯϒϩοΩϯάʹϦΫΤετ
    2.੍ݶதͰॱ൪଴ͪঢ়ଶͳΒ503
    3.੍ݶ͞Εͯͳ͚Ε͹όοΫΤϯυ΁ϓϩΩγ
    

    View Slide

33. ngx_http_subrequest
    location / {
    
    
    # subrequest
    
    
    }
    
    
    location /subreqest {
    
    
    puts "hello"
    
    
    }
    
    
    # http://localhost/
    
    
    => hello
    ͋ΔϩέʔγϣϯʹདྷͨϦΫΤετΛ
    ଞͷϩέʔγϣϯʹϦΫΤετͨ݁͠ՌͰ
    Ԡ౴͢ΔΑ͏ͳ࢖͍ํ
    

    View Slide

34. Nginx::Async::HTTP.sub_request
    ϦΫΤετΛ౤͛ΔͱϒϩοΫΛ։์͠ɺ
    Ϩεϙϯε࣌ʹίʔϧόοΫΛड͚औΓ࠶։
    SFRVFTU
    SFRVFTU NSVCZ
    SFTQPOTF
    SFTQPOTF
    NSVCZ
    TVCSFRFTU
    3FRVFTU 3FTQPOTF
    

    View Slide

35. ଴߹ࣨىಈܖػ
    location @waitingroom {
    
    
    mruby_rewrite_handler enable_waitingroom.rb;
    
    
    proxy_pass origin_backend/$request_uri;
    
    
    internal;
    
    
    }
    
    
    location ~ ^/queues {
    
    
    proxy_pass waitingroom;
    
    
    internal;
    
    
    }
    /queuesΛ࣮ߦ͍ͯ͠Δ

    API͸nginxΛϒϩοΫ͠ͳ͍
    

    View Slide

36. ଴߹ࣨඇ੍ݶ࣌ͷϦΫΤετભҠ
    Waiting Room
    nginx
    Client Origin
    1.ϦΫΤετ
    2.ىಈ͞Ε͍ͯΔ͔ʁ 3.ະىಈ͔

    ڐՄࡁΈΫϥΠΞϯτ
    4.ϦΫΤετ
    5.Ϩεϙϯε
    6.Ϩεϙϯε
    

    View Slide

37. ଴߹੍ࣨݶ࣌ͷϦΫΤετભҠ
    Waiting Room
    nginx
    Client Origin
    1.ϦΫΤετ
    2.ىಈ͞Ε͍ͯΔ͔ʁ
    3.ىಈத͔ͭɺ

    ະڐՄΫϥΠΞϯτ
    4.503Ϩεϙϯε
    ଴߹ࣨىಈঢ়ଶͷͱ͖͸ϨʔτϦϛοτʹؔ܎ͳ͘
    

    Ұఆ࣌ؒɺ଴߹ࣨʹҊ಺͢Δ
    

    View Slide

38. ଴߹ࣨAPIͷ֓ཁ
    (PͰ࣮૷͞Εͨ)551αʔό
    https://github.com/pyama86/ngx_waitingroom
    go on-memory cache redis
    query query
    response
    response
    ଟ૚ͷߴ଎ͳσʔλετΞΛར༻ͯ͠ɺߴ଎ʹ࣮ߦ
    

    View Slide

39. ଴߹ࣨىಈ
    4&5/9
    &91*3&
    ;"%%
    ܭࢉྔ:O(1) Ωʔྫ: a.com

    SET if Not eXistsͷུͰɺΩʔ͕ଘࡏ͠ͳ͍৔߹ͷΈ஋Λઃఆ͢Δɻ
    
    Ωʔ͕ଘࡏ͢Δ৔߹ʹ1ɺΩʔ͕ͳ͍৔߹ʹ0Λฦ٫͢ΔͷͰɺϩοΫͷΑ͏ʹར༻Ͱ͖Δɻ
    BDPNͰ଴߹ࣨΛىಈ͢Δ৔߹
    ܭࢉྔ: O(log(N)) Ωʔྫ: queue-domains

    ϝϯόʔ+είΞͰϋογϡͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻܭࢉྔ΋௿͘ɺॏෳഉআͰ͖ΔͷͰɺ
    ଴߹͕ࣨىಈ͍ͯ͠ΔυϝΠϯͷϦετʹར༻͍ͯ͠Δɻ
    ܭࢉྔ:O(1) Ωʔྫ: a.com

    TTLΛઃఆ͢ΔAPIɻυϝΠϯ໊ΛΩʔʹઃఆ͢Δ͜ͱͰɺ଴߹ࣨͷىಈ࣌ؒΛ
    
    TTLΛ༻੍͍ͯޚ͢Δ͜ͱ͕Ͱ͖Δɻྫ͑͹TTL=300ʹ͢Δͱ5෼ؒ͸଴߹͕ࣨىಈɻ
    

    View Slide

40. σʔλߏ଄
    key value
    queue-domains [a.com, b.com, c.com]
    a.com 200
    b.com 300
    c.com 400
    ͳΔ΂͘υϝΠϯΛى఺ʹ0
    ͰΞΫηεͰ͖Δߏ଄
    ZADD
    SETNX
    

    View Slide

41. ଴ͪ൪߸ͷ࠾൪
    

    View Slide

42. ଴ͪ൪߸ͷ࠾൪
    Client
    1.ϦΫΤετ
    2.1001൪Λ෷͍ग़͠
    waiting room
    ࠾൪:1000
    ڐՄࡁΈ൪߸:200
    ΫϥΠΞϯτ͸൪·Ͱ࠾൪͞Ε͍ͯΔ
    ൪·Ͱ͸ڐՄࡁΈͳͷͰɺΫϥΠΞϯτ͕଴ͪߦྻʹଘࡏ
    ൪Λ࣍ͷΫϥΠΞϯτʹ෷͍ग़͍ͨ͠
    

    View Slide

43. ଴ͪ൪߸ͷ࠾൪
    Client
    ݱ୅ͷ8͸ଟॏͰϦΫΤετ͕ߦΘΕΔͨΊɺ
    ϦΫΤετ͝ͱͰ͸ͳ͘ɺΫϥΠΞϯτ͝ͱʹ
    

    ޮ཰Α͘࠾൪͢Δʹ͸޻෉͕ඞཁ
    waiting room
    index.html
    main.css
    main.js
    main.jpg
    ޻෉͠ͳ͍ͱ1ΫϥΠΞϯτʹ
    

    4ͭ࠾൪ͯ͠͠·͏
    
    
    

    View Slide

44. ଴ͪ൪߸ͷ࠾൪
    ·ͣ͸࠾൪࣌ࠁΛ෷͍ग़͠ϖʔδʹϦμΠϨΫτͯ͠ɺ
    

    KBWBTDSJQUͰ਺ඵ͓͖ʹϙʔϦϯά͠
    

    ࠾൪࣌ࠁʹͳͬͨΒ࠾൪͢Δ͜ͱͰଟॏ࠾൪Λ੍ݶ
    1.index.html,main.cs..
    2.redirect 503.hml
    ΫϥΠΞϯτ൪߸: NULL
    
    
    ΫϥΠΞϯτID: NULL
    
    
    ࠾൪࣌ࠁ: 10ඵޙ
    
    
    waiting room
    client
    3.̐ඵ͝ͱʹ503.html΁ϦΫΤετ
    ΫϥΠΞϯτ൪߸: 1
    
    
    ΫϥΠΞϯτID: UUID
    
    
    ࠾൪࣌ࠁ: DONE
    
    
    4.࠾൪
    

    View Slide

45. ଴ͪ൪߸ͷ࠾൪
    ΫϥΠΞϯτͷঢ়ଶ͸҉߸Խͨ͠$PPLJFʹอଘ
    ΫϥΠΞϯτ൪߸
    ΫϥΠΞϯτID
    ΫϥΠΞϯτ൪߸ൃߦ࣌ࠁ
    αʔόͰ࠾൪͍ͯ͠Δ௨͠൪߸
    ΫϥΠΞϯτࣝผࢠ(UUID)
    ΫϥΠΞϯτ൪߸Λൃߦ͢Δ࣌ࠁɻUnixTimeͰ֨ೲ͞Ε͓ͯΓɺ
    
    γεςϜશମͰ࣌ࠁ͕େ͖ͣ͘ΕΔ͜ͱ͸
    ૝ఆ͍ͯ͠ͳ͍
    

    View Slide

46. ଴ͪ൪߸ͷ࠾൪
    */$3
    &91*3&
    ܭࢉྔ O(1) Ωʔྫ:a.com

    Χ΢ϯλɻࢦఆͨ͠ΩʔΛΞτϛοΫʹΠϯΫϦϝϯτͰ͖Δɻ
    FYBNQMFDPNͰ଴߹ࣨΛىಈ͢Δ৔߹
    ܭࢉྔ O(1) Ωʔྫ: a.com

    લड़ͷ௨Γɻ
    

    View Slide

47. ͜͜·Ͱ࿩ͨ͜͠ͱ
    • ଴߹ࣨ͸nginxͷϨʔτϦϛοτ౸ୡΛܖػʹىಈ͠ɺࢦఆͨ࣌ؒ͠ىಈ͠ଓ͚Δ
    
    • ΫϥΠΞϯτ͸ΫοΩʔʹΫϥΠΞϯτ൪߸΍ΫϥΠΞϯτIDΛ࣋ͭ
    
    • ΫϥΠΞϯτ൪߸ͷ෷͍ग़͠͸ɺRedisͷINCRͰΧ΢ϯτΞοϓͨ͠൪߸͕෷͍ग़͞ΕΔ
    ڐՄ൪߸Ϧετ
    a.com 200
    b.com 300
    c.com 400
    ࠾൪Ϧετ
    a.com 500
    b.com 600
    c.com 700
    αʔόαΠυσʔλετΞ ΫϥΠΞϯτCookie
    ΫϥΠΞϯτ൪߸ 100
    ΫϥΠΞϯτID UUID
    ࠾൪࣌ࠁ 1986/04/10
    

    View Slide

48. ڐՄΫϥΠΞϯτͷ൑ఆ
    

    View Slide

49. ڐՄ൪߸ͷߋ৽
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    proxy role
    nginx
    waiting room
    ڐՄ൪߸Ϧετ
    Site A 200
    Site B 1000
    Site C 440
    ҰͭͷڐՄϦετΛෳ਺ͷQSPYZSPMF͕ࢀর͢Δ৔߹ʹɺ
    ͲͷΑ͏ʹߋ৽͢Δ͔ʁ
    1෼͝ͱʹ200ΫϥΠΞϯτͣͭ
    
    
    ڐՄ͢Δͱ͍͏Α͏ͳ
    

    ੍ޚΛ͍ͨ͠
    
    
    200→400→600....
    

    View Slide

50. զʑʹ͸goroutine͕͋Δ͡Όͳ͍͔
    waiting room
    http server
    access controller ڐՄ൪߸Ϧετ
    Site A 200
    Site B 1000
    Site C 440
    ࢦఆִؒͰϦετΛࢀর͠ɺڐՄϦετΛߋ৽͢Δ
    goroutineΛىಈ͢Δ
    ߋ৽
    

    View Slide

51. ڐՄ൪߸Ϧετͷഉଞ੍ޚ
    waiting room
    http server
    access
    controller
    ڐՄ൪߸Ϧετ
    Site A 200
    Site B 1000
    Site C 440
    SETNXΛ׆༻͢Δ͜ͱͰඞͣυϝΠϯ͋ͨΓͷ

    ߋ৽ϓϩηε͸ඞͣҰͭʹͳΔ
    waiting room
    http server
    access
    controller
    SETNX ok
    SETNX ng
    

    View Slide

52. ڐՄ൪߸ͷߋ৽
    ;3"/(&
    &91*3&
    4&5/9
    ܭࢉྔ: O(log(N)+M) Ωʔྫ queue-domains

    ଴߹͕ࣨىಈ͞Ε͍ͯΔ͢΂ͯͷυϝΠϯΛϦετͰऔಘ͢Δɻ
    HPSPVUJOFͰࢦఆඵ਺ײִؒͰϧʔϓ͢Δ
    ܭࢉྔ:O(1) Ωʔྫ: a.com_lock
    
    ϩοΫΛऔΕͨproxy roleͷΈ͕ɺڐՄ൪߸Λߋ৽Մೳɻ
    
    ෳ਺ͷproxy role͕ಉ࣌ʹߋ৽͢Δ͜ͱΛഉଞ੍ޚ͢Δɻ
    ܭࢉྔ O(1) Ωʔྫ:a.com_lock
    
    ࣍ճͷڐՄ൪߸ߋ৽࣌ؒ·Ͱͷ஋ΛTTLʹઃఆ͢Δ͜ͱͰϩοΫ͕औΒΕͨ··ʹͳΔͨΊ
    
    ߋ৽ִؒΛTTLͰ੍ޚͰ͖Δɻ60ඵִؒͰڐՄ൪߸Λߋ৽͍ͨ͠ͳΒ60ඵΛઃఆɻ
    4&5&9
    ܭࢉྔ O(1)Ωʔྫ:a.com
    
    ݱࡏͷڐՄ൪߸ʹ௥ՃڐՄ͍ͨ͠ΫϥΠΞϯτ਺Λ௥Ճͯ͠ɺTTLͱͱ΋ʹॻ͖ࠐΈɻ
    

    View Slide

53. ΞΫηεڐՄ
    ڐՄ൪߸Ϧετ
    Site A 200
    Site B 1000
    Site C 440
    ΫϥΠΞϯτͷΞΫηεΛڐՄͨ͠৔߹͸ɺ
    ΫϥΠΞϯτIDΛΩʔʹRedisʹอଘ͠ɺ
    TTLͷظؒΞΫηεڐՄ
    Site A
    
    ΫϥΠΞϯτ൪߸:100
    
    ΫϥΠΞϯτID: UUID1
    waiting room
    ࢀর
    Ωʔ TTL
    UUID1 600
    ॻ͖ࠐΈ
    Client
    

    View Slide

54. ੑೳධՁ
    

    View Slide

55. Cookie࢖͑ΔϕϯνϚʔΧʔॻ͍ͨ
    https://github.com/pyama86/ngx-smart-ratelimit/tree/master/bench
    CPU Apple M1 MAX
    
    ϝϞϦ 64G
    
    10ฒྻ 10,000ϦΫΤετ
    
    keepaliveͳ͠
    ଌఆ؀ڥ
    ݁Ռ
    5000 request / sec
    
    ฏۉ0.2msecͰԠ౴Մೳ
    

    View Slide

56. ੩తίϯςϯπ഑৴ͱͷൺֱ
    ݁Ռ
    8000 request / sec
    
    ฏۉ0.125msec
    0.125msec/request ÷ 0.2msec/request = 1.6
    

    View Slide

57. ·ͱΊ
    

    View Slide

58. ΋͏͋ͱ͸ҿञͷΈɺ
    
    
    ҿञͦ͜͢΂ͯʂʂʂ
    

    View Slide

59. શํҐ࠾༻͍ͯ͠·͢ɻ
    

    ͜ͷ͋ͱҿञͨ͠ΒɺԿͰ΋࿩͠·͢ɻ
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    

    View Slide