Goで実装された高速な 仮想待合室サーバの実装と詳解

by Kazuhiko Yamashita

Slide 1

Slide 1 text

GoͰ࣮૷͞Εͨߴ଎ͳ   Ծ૝଴߹ࣨαʔόͷ࣮૷ͱৄղ ʙے೑৯ಊɺૣ͘෱Ԭʹ΋Ͱ͖ͯ΄͍͠ΜͰ͚͢Ͳฤʙ

Slide 2

Slide 2 text

ࢁԼ࿨඙!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ ($1ɺ"JS fl PXɺ1VC4VCɺ%BUB fl PX

Slide 3

Slide 3 text

ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

౰೔݊΋͋Γ·͢ʂʂ1

Slide 9

Slide 9 text

αʔϏεͱࣄۀ෦੍ ˞ଞʹ΋ΤϯδχΞϦϯά෦໳ɺάϧʔϓɾνʔϜ͕͋Δ͕આ໌ͷ؆ུԽͷͨΊਤ͔Β͸লུ ٕज़෦ίʔϙϨʔτΤϯδχΞϦϯάάϧʔϓ ٕज़෦σʔλج൫νʔϜ $50ࣨݚڀ։ൃνʔϜ ࣛࣇౡΤϯδχΞϦϯάνʔϜ ηΩϡϦςΟରࡦࣨ σβΠϯࣨ

Slide 10

Slide 10 text

43&ͱͯ͠ͷ੹຿Λෛ͍ͬͯΔϝϯόʔ

Slide 11

Slide 11 text

ٕज़ελοΫ Platform Datastore Monitor On-premises,Google Cloud,AWS Orchastration OpenStack,Kubernetes RDS,MySQL,PostgreSQL Mackerel,Prometheus,Datadog https://tech.pepabo.com/tech-stack/

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

ࠓ೔࿩͢͜ͱ γεςϜߏ੒֓ཁ ൵ܶ Ծ૝଴߹࣮ࣨ૷ৄղ ๻ͷ༮গظͷՄѪ͍ࣸਅ100બ

Slide 14

Slide 14 text

γεςϜߏ੒֓ཁ OpenStack Managed LB Backend APIs nginx proxy role proxy roleͰෳ਺ͷछྨͷapp roleʹϓϩΩγ nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role

Slide 15

Slide 15 text

൵ܶ

Slide 16

Slide 16 text

൵ܶ͸ى͖΍͍͢ ͓٬༷ͷݶఆ঎඼ൢചͳͲʹΑΔେྔΞΫηε ΧʔτϘοτʹΑΔෆਖ਼ͳେྔΞΫηε DDOS߈ܸ

Slide 17

Slide 17 text

൵ܶ͸  ๷͛Δͷ͔ʁ

Slide 18

Slide 18 text

ΤΠοʂʂʂ

Slide 19

Slide 19 text

ʮϨʔτϦϛοτΛ ͔͚ΔͱͲ͏ͳΔʁʯ

Slide 20

Slide 20 text

஌ΒΜͷ͔ʁ

Slide 21

Slide 21 text

ΤΠοʂʂʂ ঎඼͕ ചΕͳ͘ͳΔͷͩ

Slide 22

Slide 22 text

ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ 1. େྔΞΫηε͕ൃੜ͢Δ 2. ϨʔτϦϛοτΛ͔͚Δ 3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ 4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍ 5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍ 6. ձࣾͷച্্͕͕Βͳ͍ 7. Զͷڅྉ্͕͕Βͳ͍

Slide 23

Slide 23 text

ৄղϨʔτϦϛοτʹΑΔܦࡁଛࣦ 1. େྔΞΫηε͕ൃੜ͢Δ 2. ϨʔτϦϛοτΛ͔͚Δ 3. ঎඼ʹΞΫηε͕Ͱ͖ͳ͍ͷͰചΕͳ͘ͳΔ 4. ঎඼͕ചΕͳ͍ͱ͓٬༷γϣοϓͷച্্͕͕Βͳ͍ 5. ͓٬༷ͷച্্͕͕Βͳ͍ͱΧϥʔϛʔͷച্্͕͕Βͳ͍ 6. ձࣾͷച্্͕͕Βͳ͍ 7.Զͷڅྉ্͕͕Βͳ͍!!!௒ࠔΔ!!!

Slide 24

Slide 24 text

–Kazuhiko Yamashita 2023. “Ծ૝଴߹ࣨͱ͸څྉͰ͋Δ.”

Slide 25

Slide 25 text

Ծ૝଴߹ࣨ ઌணॱʹྲྀྔ੍ޚ͠ͳ͕Β  αΠτ΁Ҋ಺͢Δ͜ͱͰ ߴෛՙ࣌Ͱ΋҆ఆͨ͠ΞΫηε͕Մೳ

Slide 26

Slide 26 text

Ծ૝଴߹ࣨ OpenStack Managed LB Backend APIs ଴߹ࣨAPIΛ։ൃͨ͠ nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role nginx app unicorn   php-fpm app role proxy role nginx waiting room

Slide 27

Slide 27 text

ৄղ଴߹ࣨγεςϜ

Slide 28

Slide 28 text

Ծ૝଴߹ࣨ֓ཁ proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room ڐՄϦετ Site A 200 Site B ... Site C ... ΫϥΠΞϯτʹ଴ͪ൪߸Λ࠾൪ͯ͠ɺڐՄϦετͷ൪߸ΑΓ খ͍͞൪߸Λ͍࣋ͬͯΕ͹Ұఆ࣌ؒΞΫηε͕Մೳ Client A ΫϥΠΞϯτ൪߸:100 ΞΫηεڐՄ Client B ΫϥΠΞϯτ൪߸:300 ଴߹ࣨ΁

Slide 29

Slide 29 text

ৄղ଴߹ࣨγεςϜ ଴߹ࣨىಈܖػ ଴ͪ൪߸ͷ࠾൪ ڐՄΫϥΠΞϯτͷ൑ఆ

Slide 30

Slide 30 text

଴߹ࣨىಈܖػ location / { limit_req zone=example nodelay; limit_req_status 512; error_page 512 =200 @waitingroom; root /var/www/nginx; } ϨʔτϦϛοτʹ֘౰ͨ͠Β  ಛఆͷΤϯυϙΠϯτʹ  ϦμΠϨΫτ OHJOYͷSBUFMJNJUNPEVMFΛར༻͠ɺ ա৒ΞΫηεΛݕ஌

Slide 31

Slide 31 text

଴߹ࣨىಈܖػ location @waitingroom { mruby_rewrite_handler enable_waitingroom.rb; proxy_pass origin_backend/$request_uri; internal; } location ~ ^/queues { proxy_pass waitingroom; internal; } ଴߹ࣨAPI͸mrubyͰ  ίʔϧͯ͠ɺ଴߹ࣨΛ  ىಈ͢Δ

Slide 32

Slide 32 text

଴߹ࣨىಈܖػ 1.ϊϯϒϩοΩϯάʹϦΫΤετ 2.੍ݶதͰॱ൪଴ͪঢ়ଶͳΒ503 3.੍ݶ͞Εͯͳ͚Ε͹όοΫΤϯυ΁ϓϩΩγ

Slide 33

Slide 33 text

ngx_http_subrequest location / { # subrequest } location /subreqest { puts "hello" } # http://localhost/ => hello ͋ΔϩέʔγϣϯʹདྷͨϦΫΤετΛ ଞͷϩέʔγϣϯʹϦΫΤετͨ݁͠ՌͰ Ԡ౴͢ΔΑ͏ͳ࢖͍ํ

Slide 34

Slide 34 text

Nginx::Async::HTTP.sub_request ϦΫΤετΛ౤͛ΔͱϒϩοΫΛ։์͠ɺ Ϩεϙϯε࣌ʹίʔϧόοΫΛड͚औΓ࠶։ SFRVFTU SFRVFTU NSVCZ SFTQPOTF SFTQPOTF NSVCZ TVCSFRFTU 3FRVFTU 3FTQPOTF

Slide 35

Slide 35 text

଴߹ࣨىಈܖػ location @waitingroom { mruby_rewrite_handler enable_waitingroom.rb; proxy_pass origin_backend/$request_uri; internal; } location ~ ^/queues { proxy_pass waitingroom; internal; } /queuesΛ࣮ߦ͍ͯ͠Δ  API͸nginxΛϒϩοΫ͠ͳ͍

Slide 36

Slide 36 text

଴߹ࣨඇ੍ݶ࣌ͷϦΫΤετભҠ Waiting Room nginx Client Origin 1.ϦΫΤετ 2.ىಈ͞Ε͍ͯΔ͔ʁ 3.ະىಈ͔  ڐՄࡁΈΫϥΠΞϯτ 4.ϦΫΤετ 5.Ϩεϙϯε 6.Ϩεϙϯε

Slide 37

Slide 37 text

଴߹੍ࣨݶ࣌ͷϦΫΤετભҠ Waiting Room nginx Client Origin 1.ϦΫΤετ 2.ىಈ͞Ε͍ͯΔ͔ʁ 3.ىಈத͔ͭɺ  ະڐՄΫϥΠΞϯτ 4.503Ϩεϙϯε ଴߹ࣨىಈঢ়ଶͷͱ͖͸ϨʔτϦϛοτʹؔ܎ͳ͘   Ұఆ࣌ؒɺ଴߹ࣨʹҊ಺͢Δ

Slide 38

Slide 38 text

଴߹ࣨAPIͷ֓ཁ (PͰ࣮૷͞Εͨ)551αʔό https://github.com/pyama86/ngx_waitingroom go on-memory cache redis query query response response ଟ૚ͷߴ଎ͳσʔλετΞΛར༻ͯ͠ɺߴ଎ʹ࣮ߦ

Slide 39

Slide 39 text

଴߹ࣨىಈ 4&5/9 &91*3& ;"%% ܭࢉྔ:O(1) Ωʔྫ: a.com  SET if Not eXistsͷུͰɺΩʔ͕ଘࡏ͠ͳ͍৔߹ͷΈ஋Λઃఆ͢Δɻ Ωʔ͕ଘࡏ͢Δ৔߹ʹ1ɺΩʔ͕ͳ͍৔߹ʹ0Λฦ٫͢ΔͷͰɺϩοΫͷΑ͏ʹར༻Ͱ͖Δɻ BDPNͰ଴߹ࣨΛىಈ͢Δ৔߹ ܭࢉྔ: O(log(N)) Ωʔྫ: queue-domains  ϝϯόʔ+είΞͰϋογϡͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻܭࢉྔ΋௿͘ɺॏෳഉআͰ͖ΔͷͰɺ ଴߹͕ࣨىಈ͍ͯ͠ΔυϝΠϯͷϦετʹར༻͍ͯ͠Δɻ ܭࢉྔ:O(1) Ωʔྫ: a.com  TTLΛઃఆ͢ΔAPIɻυϝΠϯ໊ΛΩʔʹઃఆ͢Δ͜ͱͰɺ଴߹ࣨͷىಈ࣌ؒΛ TTLΛ༻੍͍ͯޚ͢Δ͜ͱ͕Ͱ͖Δɻྫ͑͹TTL=300ʹ͢Δͱ5෼ؒ͸଴߹͕ࣨىಈɻ

Slide 40

Slide 40 text

σʔλߏ଄ key value queue-domains [a.com, b.com, c.com] a.com 200 b.com 300 c.com 400 ͳΔ΂͘υϝΠϯΛى఺ʹ0 ͰΞΫηεͰ͖Δߏ଄ ZADD SETNX

Slide 41

Slide 41 text

଴ͪ൪߸ͷ࠾൪

Slide 42

Slide 42 text

଴ͪ൪߸ͷ࠾൪ Client 1.ϦΫΤετ 2.1001൪Λ෷͍ग़͠ waiting room ࠾൪:1000 ڐՄࡁΈ൪߸:200 ΫϥΠΞϯτ͸൪·Ͱ࠾൪͞Ε͍ͯΔ ൪·Ͱ͸ڐՄࡁΈͳͷͰɺΫϥΠΞϯτ͕଴ͪߦྻʹଘࡏ ൪Λ࣍ͷΫϥΠΞϯτʹ෷͍ग़͍ͨ͠

Slide 43

Slide 43 text

଴ͪ൪߸ͷ࠾൪ Client ݱ୅ͷ8͸ଟॏͰϦΫΤετ͕ߦΘΕΔͨΊɺ ϦΫΤετ͝ͱͰ͸ͳ͘ɺΫϥΠΞϯτ͝ͱʹ   ޮ཰Α͘࠾൪͢Δʹ͸޻෉͕ඞཁ waiting room index.html main.css main.js main.jpg ޻෉͠ͳ͍ͱ1ΫϥΠΞϯτʹ   4ͭ࠾൪ͯ͠͠·͏

Slide 44

Slide 44 text

଴ͪ൪߸ͷ࠾൪ ·ͣ͸࠾൪࣌ࠁΛ෷͍ग़͠ϖʔδʹϦμΠϨΫτͯ͠ɺ   KBWBTDSJQUͰ਺ඵ͓͖ʹϙʔϦϯά͠   ࠾൪࣌ࠁʹͳͬͨΒ࠾൪͢Δ͜ͱͰଟॏ࠾൪Λ੍ݶ 1.index.html,main.cs.. 2.redirect 503.hml ΫϥΠΞϯτ൪߸: NULL ΫϥΠΞϯτID: NULL ࠾൪࣌ࠁ: 10ඵޙ waiting room client 3.̐ඵ͝ͱʹ503.html΁ϦΫΤετ ΫϥΠΞϯτ൪߸: 1 ΫϥΠΞϯτID: UUID ࠾൪࣌ࠁ: DONE 4.࠾൪

Slide 45

Slide 45 text

଴ͪ൪߸ͷ࠾൪ ΫϥΠΞϯτͷঢ়ଶ͸҉߸Խͨ͠$PPLJFʹอଘ ΫϥΠΞϯτ൪߸ ΫϥΠΞϯτID ΫϥΠΞϯτ൪߸ൃߦ࣌ࠁ αʔόͰ࠾൪͍ͯ͠Δ௨͠൪߸ ΫϥΠΞϯτࣝผࢠ(UUID) ΫϥΠΞϯτ൪߸Λൃߦ͢Δ࣌ࠁɻUnixTimeͰ֨ೲ͞Ε͓ͯΓɺ γεςϜશମͰ࣌ࠁ͕େ͖ͣ͘ΕΔ͜ͱ͸ ૝ఆ͍ͯ͠ͳ͍

Slide 46

Slide 46 text

଴ͪ൪߸ͷ࠾൪ */$3 &91*3& ܭࢉྔ O(1) Ωʔྫ:a.com  Χ΢ϯλɻࢦఆͨ͠ΩʔΛΞτϛοΫʹΠϯΫϦϝϯτͰ͖Δɻ FYBNQMFDPNͰ଴߹ࣨΛىಈ͢Δ৔߹ ܭࢉྔ O(1) Ωʔྫ: a.com  લड़ͷ௨Γɻ

Slide 47

Slide 47 text

͜͜·Ͱ࿩ͨ͜͠ͱ • ଴߹ࣨ͸nginxͷϨʔτϦϛοτ౸ୡΛܖػʹىಈ͠ɺࢦఆͨ࣌ؒ͠ىಈ͠ଓ͚Δ • ΫϥΠΞϯτ͸ΫοΩʔʹΫϥΠΞϯτ൪߸΍ΫϥΠΞϯτIDΛ࣋ͭ • ΫϥΠΞϯτ൪߸ͷ෷͍ग़͠͸ɺRedisͷINCRͰΧ΢ϯτΞοϓͨ͠൪߸͕෷͍ग़͞ΕΔ ڐՄ൪߸Ϧετ a.com 200 b.com 300 c.com 400 ࠾൪Ϧετ a.com 500 b.com 600 c.com 700 αʔόαΠυσʔλετΞ ΫϥΠΞϯτCookie ΫϥΠΞϯτ൪߸ 100 ΫϥΠΞϯτID UUID ࠾൪࣌ࠁ 1986/04/10

Slide 48

Slide 48 text

ڐՄΫϥΠΞϯτͷ൑ఆ

Slide 49

Slide 49 text

ڐՄ൪߸ͷߋ৽ proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room proxy role nginx waiting room ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ҰͭͷڐՄϦετΛෳ਺ͷQSPYZSPMF͕ࢀর͢Δ৔߹ʹɺ ͲͷΑ͏ʹߋ৽͢Δ͔ʁ 1෼͝ͱʹ200ΫϥΠΞϯτͣͭ ڐՄ͢Δͱ͍͏Α͏ͳ   ੍ޚΛ͍ͨ͠ 200→400→600....

Slide 50

Slide 50 text

զʑʹ͸goroutine͕͋Δ͡Όͳ͍͔ waiting room http server access controller ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ࢦఆִؒͰϦετΛࢀর͠ɺڐՄϦετΛߋ৽͢Δ goroutineΛىಈ͢Δ ߋ৽

Slide 51

Slide 51 text

ڐՄ൪߸Ϧετͷഉଞ੍ޚ waiting room http server access controller ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 SETNXΛ׆༻͢Δ͜ͱͰඞͣυϝΠϯ͋ͨΓͷ  ߋ৽ϓϩηε͸ඞͣҰͭʹͳΔ waiting room http server access controller SETNX ok SETNX ng

Slide 52

Slide 52 text

ڐՄ൪߸ͷߋ৽ ;3"/(& &91*3& 4&5/9 ܭࢉྔ: O(log(N)+M) Ωʔྫ queue-domains  ଴߹͕ࣨىಈ͞Ε͍ͯΔ͢΂ͯͷυϝΠϯΛϦετͰऔಘ͢Δɻ HPSPVUJOFͰࢦఆඵ਺ײִؒͰϧʔϓ͢Δ ܭࢉྔ:O(1) Ωʔྫ: a.com_lock ϩοΫΛऔΕͨproxy roleͷΈ͕ɺڐՄ൪߸Λߋ৽Մೳɻ ෳ਺ͷproxy role͕ಉ࣌ʹߋ৽͢Δ͜ͱΛഉଞ੍ޚ͢Δɻ ܭࢉྔ O(1) Ωʔྫ:a.com_lock ࣍ճͷڐՄ൪߸ߋ৽࣌ؒ·Ͱͷ஋ΛTTLʹઃఆ͢Δ͜ͱͰϩοΫ͕औΒΕͨ··ʹͳΔͨΊ ߋ৽ִؒΛTTLͰ੍ޚͰ͖Δɻ60ඵִؒͰڐՄ൪߸Λߋ৽͍ͨ͠ͳΒ60ඵΛઃఆɻ 4&5&9 ܭࢉྔ O(1)Ωʔྫ:a.com ݱࡏͷڐՄ൪߸ʹ௥ՃڐՄ͍ͨ͠ΫϥΠΞϯτ਺Λ௥Ճͯ͠ɺTTLͱͱ΋ʹॻ͖ࠐΈɻ

Slide 53

Slide 53 text

ΞΫηεڐՄ ڐՄ൪߸Ϧετ Site A 200 Site B 1000 Site C 440 ΫϥΠΞϯτͷΞΫηεΛڐՄͨ͠৔߹͸ɺ ΫϥΠΞϯτIDΛΩʔʹRedisʹอଘ͠ɺ TTLͷظؒΞΫηεڐՄ Site A ΫϥΠΞϯτ൪߸:100 ΫϥΠΞϯτID: UUID1 waiting room ࢀর Ωʔ TTL UUID1 600 ॻ͖ࠐΈ Client

Slide 54

Slide 54 text

ੑೳධՁ

Slide 55

Slide 55 text

Cookie࢖͑ΔϕϯνϚʔΧʔॻ͍ͨ https://github.com/pyama86/ngx-smart-ratelimit/tree/master/bench CPU Apple M1 MAX ϝϞϦ 64G 10ฒྻ 10,000ϦΫΤετ keepaliveͳ͠ ଌఆ؀ڥ ݁Ռ 5000 request / sec ฏۉ0.2msecͰԠ౴Մೳ

Slide 56

Slide 56 text

੩తίϯςϯπ഑৴ͱͷൺֱ ݁Ռ 8000 request / sec ฏۉ0.125msec 0.125msec/request ÷ 0.2msec/request = 1.6

Slide 57

Slide 57 text

·ͱΊ

Slide 58

Slide 58 text

΋͏͋ͱ͸ҿञͷΈɺ ҿञͦ͜͢΂ͯʂʂʂ

Slide 59

Slide 59 text

શํҐ࠾༻͍ͯ͠·͢ɻ   ͜ͷ͋ͱҿञͨ͠ΒɺԿͰ΋࿩͠·͢ɻ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU