Erlang/OTP で作るリアルタイムサーバー

Slide 1

Slide 1 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ϦΞϧλΠϜαʔόʔ ERLANG/OTP Ͱ࡞Δ 1 ೥݄೔ HVNJ*ODਗ਼ਫ༎ޗ ERLANG & ELIXIR FEST 2019

Slide 2

Slide 2 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 2 ձࣾ঺հ

Slide 3

Slide 3 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 3 ձࣾ঺հ ઃཱ ೥݄೔ ࠃ಺ࢠձࣾ ւ֎ࢠձࣾ ࣄۀ಺༰ ϞόΠϧήʔϜࣄۀ ࣾ ࣾ 93ࣄۀ 73 "3 .3ͳͲ ࣾ ࣾ ϒϩοΫνΣʔϯࣄۀ ࣾ ઃཱ ೥݄೔ ࠃ಺ࢠձࣾ ւ֎ࢠձࣾ ࣄۀ಺༰ ϞόΠϧήʔϜࣄۀ ࣾ ࣾ 93ࣄۀ 73 "3 .3ͳͲ ࣾ ࣾ ϒϩοΫνΣʔϯࣄۀ ࣾ ˞೥݄ݱࡏ ࣄۀྫϞόΠϧήʔϜࣄۀͷ໊ࣗࣾٛ഑৴ͷҰ෦ ͳͲʜ

Slide 4

Slide 4 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣗݾ঺հ ▸ ਗ਼ਫ༎ޗ!ZBNJPOQ ▸ 2011೥ ೖࣾ ▸ 2015೥ R&D΁ 4 ήʔϜαʔόʔͷ։ൃ ڞ௨ج൫ͷ։ൃ ϦΞϧλΠϜαʔόʔͷ։ൃ ڞ௨ج൫։ൃɾӡ༻νʔϜϚωʔδϟʔ ࣍ੈ୅ڞ௨ج൫ͷ։ൃ

Slide 5

Slide 5 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 5 ࠓ೔࿩͢ࣄ ϦΞϧλΠϜαʔόʔ Λ Erlang/OTP Ͱ ϓϩτίϧ͔Β ࡞ͬͨ࿩

Slide 6

Slide 6 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 6 ࠓ೔࿩͢ࣄ ϦΞϧλΠϜ αʔόʔͱ͸

Slide 7

Slide 7 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 7 1 VS 1 ͷϚϧνରઓ

Slide 8

Slide 8 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ήʔϜͷ தͰ͸ 8

Slide 9

Slide 9 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 9 1 VS 1 ͷϚϧνରઓ ΩϟϥΫλʔ͸2ମ?

Slide 10

Slide 10 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 10 1 VS 1 ͷϚϧνରઓ ίϯϐϡʔλʔ͸2୆͋Δ

Slide 11

Slide 11 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 11 1 VS 1 ͷϚϧνରઓ ίϯϐϡʔλʔ͸2୆͋Δ COPY COPY ΩϟϥΫλʔ͸4ମ ௨৴ ௨৴

Slide 12

Slide 12 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ೋ୆ͷίϯϐϡʔλʔͰ௨৴͢Δඞཁ͕͋Δ ▸ ࣗ෼ͷΩϟϥΛಈ͔ͨ͠Β͙͢ʹ$01:ʹ఻Θͬͯཉ͍͠ ▸ ૬खͷ$01:ʹμϝʔδΛ༩͑ͨΒ͙͢ʹΦϦδφϧʹ఻Θͬͯཉ͍͠ ▸ ߴස౓௨৴ ▸ εϚϗ͸ͦͷ··Ͱ͸௚઀௨৴͕೉͍͜͠ͱ͕ଟ͍ 12

Slide 13

Slide 13 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 13 ϦΞϧλΠϜαʔόʔͱ͸ 1VC4VC 2 VS 2 ͷରઓήʔϜ

Slide 14

Slide 14 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ήʔϜҎ֎ʹ΋ 14

Slide 15

Slide 15 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 15 ͓ख఻͍ͷ࣮ྫ ϚΠΫϩαʔϏε܈ 0DVMVT4UPSF 'BDFCPPL ʜ "1* ೝূ ՝ۚ ഑৴ 6OJUZ IUUQTWBSLDPKQ ϓϥοτϑΥʔϜ܈ 1SPYZ 1VC4VC ഑৴ελδΦ ήʔϜαʔό

Slide 16

Slide 16 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 16 ΞδΣϯμ ͳͥϦΞϧλΠϜαʔόʔ͔ ઃܭ Pub/SubϞσϧ ϓϩτίϧ

Slide 17

Slide 17 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 17 ΞδΣϯμ ͳͥϦΞϧλΠϜαʔόʔ͔ ઃܭ Pub/SubϞσϧ ϓϩτίϧ

Slide 18

Slide 18 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͜Ε·ͰͷιʔγϟϧήʔϜ ▸ ඇಉظίϛϡχέʔγϣϯ ▸ ಉ࣌ʹ༡ΜͰ͍ͳ͍ͷʹҰॹʹ༡ΜͰ͍Δײ֮ ▸ χίχίಈը ▸ ٕज़త੍໿ ▸ ϞόΠϧ8FCಛ༗ͷ࢓༷ ▸ ॲཧೳྗɾόοςϦʔ ▸ ճઢ଎౓ɾྉۚʢύέࢮʣ 18

Slide 19

Slide 19 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ΧδϡΞϧήʔϜ͔ΒΑΓೱີͳήʔϜମݧ΁ ▸ ΑΓڧ͍࿈ܞɾҰମײ ▸ ຊ౰ͷಉ࣌ϓϨΠ ▸ χίχίੜ์ૹ ▸ ΞϓϦԽ ▸ ୺຤ੑೳɾωοτϫʔΫ؀ڥͷܶతͳ޲্ 19

Slide 20

Slide 20 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 20 ϦΫΤετ/Ϩεϙϯε Ϟσϧ 1. Request 2. Response Client Server ࠓ΋࢖͍ͬͯΔ)551௨৴

Slide 21

Slide 21 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 21 ϦΫΤετ/Ϩεϙϯε Ϟσϧ Client Server 1.Request αʔόʔͰى͖ͨ͜ͱΛ ࠓ͙͢ʹ఻͑ΒΕͳ͍

Slide 22

Slide 22 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE Կ͕໰୊͔ʁ ▸ ௨৴͸Ϣʔβʔଆ͔Β͔͠։࢝Ͱ͖ͳ͍ʢ4FSWFS͔Β։࢝Ͱ͖ͳ͍ʣ ▸ ߴස౓ͷ௨৴͕೉͍͠ʢຖඵճ௨৴͍ͨ͠ʣ ▸ Φʔόʔϔου ▸ ϨΠϯςϯγ 22

Slide 23

Slide 23 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE Server Client SYN SYN+ACK ACK ࠷ॳʹ Connection Λཱ֬͢Δ 23 TCP CONNECTION Λ௚઀࢖͏

Slide 24

Slide 24 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE Server Client Connection ͕͋ΔݶΓࣗ༝ͳλΠϛϯάͰ௨৴Ͱ͖Δ 24 TCP CONNECTION Λ௚઀࢖͏

Slide 25

Slide 25 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 25 HTTPͰ΋TCP͸࢖͍ͬͯΔ͕ Response Λฦͨ͠Β Connection Λഁغ͢Δ  ʢεςʔτϨεʣ Server Client

Slide 26

Slide 26 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ϦΞϧλΠϜ௨৴ͷϝϦοτ ▸ ४උ͑͞Ͱ͖Ε͹ ͲͪΒ͔ΒͰ΋ૹΓ࢝ΊΒΕΔ ▸ Ұճ͋ͨΓͷ௨৴ίετ͕খ͍͞ ▸ ඵؒճͱ͔ૹΔ͜ͱ͕Ͱ͖Δ ▸ ஗Ԇ͕গͳ͍ 26

Slide 27

Slide 27 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ϦΞϧλΠϜ௨৴ͷσϝϦοτ ▸ ෛՙ͕ߴ͍ɾෛՙ෼ࢄ͠ʹ͍͘ ▸ ঢ়ଶΛϝϞϦ্ʹ࣋ͭͷͰো֐ʹऑ͍ ▸ Τϥʔύλʔϯ͕ଟ͘ίʔυ͕ෳࡶʹͳΓ͕ͪ ▸ ୺຤ͷόοςϦʔফඅ͕ܹ͍͠ ௨৴ස౓ʹґଘ 27

Slide 28

Slide 28 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ୺຤ؒͷ௚઀௨৴ 11 ͸ʁ ▸ ಠྗͰ௚઀ͭͳ͕Δͷ͸೉͍͠ ▸ ૝ఆ͞ΕΔ΄΅શͯͷ୺຤͸/"5ͷԼʹ͋Δ ▸ 8J'J؀ڥ΋ؚΊΔͱɺͲΕ͚ͩؤுͬͯ΋ܨ͕Βͳ͍୺຤͕͋Δ ▸ ୆਺͕૿͑ͨ࣌ͷશମͷ௨৴ྔ͕രൃ͢Δ ▸ αʔόʔαΠυϩδοΫ͕͋Δ৔߹ʹθϩ͔Β։ൃ͕ඞཁ 28

Slide 29

Slide 29 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 29 ϑϧϝογϡ

Slide 30

Slide 30 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 30 ϑϧϝογϡ

Slide 31

Slide 31 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 31 ελʔ

Slide 32

Slide 32 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 32 ελʔ

Slide 33

Slide 33 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͳͥطଘͷ੡඼ͷ࠾༻Ͱ͸ͳ͔͔ͬͨ ▸ 8JOEPXT4FSWFS͸ զʑʹͱͬͯ ӡ༻ίετ͕ߴ͍ ▸ ػೳաଟ ▸ ήʔϜͰͷ௨৴ʹಛ༗ͳ࢓༷΁ͷରԠ ▸ མͱͨ͘͠ͳ͍ɻ౔೔ʹ৸͍ͯΒΕΔγεςϜ͕ཉ͔ͬͨ͠ ▸ طଘγεςϜͱͷ૬ੑ 33

Slide 34

Slide 34 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 34 طଘͷ࢓૊Έͱ݌՞͠ͳ͍γϯϓϧͳγεςϜ͕ཉ͍͠ ELB App Job MQ Batch log 1VC4VC

Slide 35

Slide 35 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 35 ΞδΣϯμ ϦΞϧλΠϜαʔόʔͱ͸ Pub/SubϞσϧ ϓϩτίϧ ઃܭ

Slide 36

Slide 36 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ίϯηϓτ ▸ ౔؅ͱͯ͠ͷ໾ׂʹప͢Δ ▸ εςʔτϑϧɺσʔλϨε ▸ ඼࣭ॏࢹ ▸ ҆ఆͯ͠ܨ͕Γɺͦͦ͜͜଎͘ɺམͪͳ͍ɻ ▸ 5$1ϝΠϯ 36

Slide 37

Slide 37 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ౔؅ʹప͢Δ ▸ ίωΫγϣϯΛ࣋ͭʹঢ়ଶ͕ଟ͘ͳΔ ▸ ωοτϫʔΫϓϩάϥϛϯάͷେ൒͸ΤϥʔϋϯυϦϯά ▸ ࣋ͭ΂͖ঢ়ଶ͸ۃྗݮΒ͢ ▸ ΞοϓσʔτͷՄೳੑΛݮΒ͢ ▸ ੾அʹͭͳ͕Δ 37

Slide 38

Slide 38 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE εςʔτϑϧɾσʔλϨε ▸ ঢ়ଶʢεςʔτʣ͸͕࣋ͭɺσʔλ͸࣋ͨͳ͍ɻ ▸ ӬଓԽ͸͠ͳ͍ ▸ ඞཁͳσʔλ͸֎෦ʹ໰͍߹ΘͤPS௨஌ FYೝূ ▸ ىಈͨ͠Β͙͢࢖͑Δঢ়ଶ 38

Slide 39

Slide 39 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 39 ೝূ App 1. game start API 2. IP:Port, Token, Topic 3. Connect Token 4. Token 5. OK 6. OK 1VC4VC

Slide 40

Slide 40 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ඼࣭ॏࢹ ▸ υΩϡϝϯτ ▸ ҉໧ͷϧʔϧΛ࡞Βͳ͍ ▸ 4QIJOYΛ࠾༻ QBDLFUEJBH ▸ ෛՙࢼݧ ▸ ੑೳΛอূ͢Δ 40 ▸ ࣗಈςετ ▸ σάϨͤ͞ͳ͍ ▸ &OEUP&OEςετʹͩ͜ΘΔ ▸ λΠϜΞ΢τ ▸ ಉ࣌ॲཧ

Slide 41

Slide 41 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͳͥTCP͔ʁ ▸ /"5௒͑ ▸ ౸ୡอূ ▸ ॱংอূ ▸ ᫔᫓੍ޚ ▸ ήʔϜʹΑͬͯ͸5$1ͷΈͰ΋े෼ͳ৔߹΋ଟ͍ λʔϯ੍ͳͲ 41

Slide 42

Slide 42 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE UDP ▸ ͋͘·Ͱ5$1ͷαϒͱͯ͠ͷ6%1 ▸ ౸ୡอূͳ͠ɾॱংอূͳ͠ɾ᫔᫓੍ޚͳ͠ ▸ ඼࣭ͷྑ͘ͳ͍ճઢͳͲͰͱʹ͔͘ૣ͘ૹΓ͍ͨ৔߹ ▸ ॏཁͳ΍ΓͱΓ͸5$1Ͱߦ͏ 42

Slide 43

Slide 43 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͳͥ ERLANG/OTP ͳͷ͔ ▸ ಈతܕ෇͚ ▸ ༏Εͨ଱ো֐ੑ ▸ ฒߦॲཧɾ෼ࢄॲཧʹڧ͍ 43

Slide 44

Slide 44 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ܰྔϓϩηε ▸ ىಈ଎౓ɾ࢖༻ϝϞϦ͕௒ܰྔͳϓϩηε ▸ 04ͷϓϩηεɾεϨουͱ͸ҧ͏ ▸ ҰॠͰ্ཱ͕ͪΓɺ͘͝Θ͔ͣͳϝϞϦ͔͠࢖༻͠ͳ͍ ▸ ΠϯελϯεΛ࡞Δײ֮ͰϓϩηεΛ࡞Δ ▸ ਺ສɺ਺ेສͷϓϩηε૸Δ ▸ ϓϩηεؒͰ͸ϝοηʔδ௨৴͕Ͱ͖Δ 44

Slide 45

Slide 45 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE CRASHͯ͠΋ྡͷϓϩηεʹӨڹ͕ͳ͍ ▸ Τϥʔ͕ىͬͨ͜ϓϩηε͸ࢮ͵ ▸ ϓϩηε͕ࢮΜͩࣄ͸؂ࢹ͍ͯ͠Δผͷϓϩηεʹରॲͤ͞Δ ▸ ࠶ىಈ΍ޙ࢝຤Λͨ͠Γɺ࿈ܞ͢Δ΄͔ͷϓϩηεͷॲஔΛͨ͠Γ ▸ Ϋϥελͷ৔߹ɺ؂ࢹϓϩηε͸ผͷϚγϯͰಈ͍͍ͯͯ΋໰୊ͳ͍ ▸ ਖ਼ৗܥͱҟৗܥͷॲཧ͕෼཭͞ΕΔ 45

Slide 46

Slide 46 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣾ಺ʹಋೖ࣮੷͕͋ͬͨ ▸ ೝূɾ՝ۚɾςΩετݕূɺͦΕͧΕͷϚΠΫϩαʔϏε ▸ ਺೥ؒͷ࿈ଓՔಇ࣮੷ ▸ શΞϓϦ͔ΒͷϦΫΤετΛ਺୆Ͱॲཧ͍ͯ͠Δ 46

Slide 47

Slide 47 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ARK ArkPubSub Server 47 ໋໊

Slide 48

Slide 48 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 48 ΞδΣϯμ ࣮૷ Pub/SubϞσϧ ͳͥϦΞϧλΠϜαʔόʔ͔ ઃܭ

Slide 49

Slide 49 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE PUB/SUB(ग़൛/ߪಡ)Ϟσϧͱ͸ ▸ ඇಉظϝοηʔδϯάϞσϧͷҰछ ▸ 4VCTDSJCF ߪಡ ͍ͯ͠ΔϢʔβʔʹϝοηʔδΛ1VCMJTI ग़൛ ͢Δ ▸ 4VCଆ͸5PQJDͱ͍͏୯ҐͰߪಡ͢Δ ▸ 1VCଆ͸5PQJDʹϝοηʔδΛૹΔ ▸ 1VCଆ͸4VCଆΛҙࣝ͢Δඞཁ͕ͳ͘ɺૄ݁߹ʹͳΔ 49

Slide 50

Slide 50 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 50 /CHATROOM/AAA A B C D PubSub Subscribe

Slide 51

Slide 51 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 51 /CHATROOM/AAA A B C D PubSub Publish

Slide 52

Slide 52 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 52 /CHATROOM/AAA /TOPIC/BBB A B C D PubSub Subscribe

Slide 53

Slide 53 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 53 A B C D PubSub Publish /CHATROOM/AAA /TOPIC/BBB

Slide 54

Slide 54 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ROOM Ϟσϧͱͷҧ͍ ▸ 4VCTDSJCFS͕ͷ5PQJDʹ΋1VCMJTIͰ͖Δ ▸ 1VCMJTIFS͕5PQJDΛ4VCTDSJCF͍ͯ͠Δඞཁ͸ͳ͍ ▸ 3PPNͷ؅ཧ ࡞੒࡟আ ͕ඞཁͳ͍ 54

Slide 55

Slide 55 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣮ࡍͷήʔϜͰͷ࢖ΘΕํ Room/1234 A B C PubSub શһ΁ͷϝοηʔδ͸શһ༻ͷ Topic ΁

Slide 56

Slide 56 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣮ࡍͷήʔϜͰͷ࢖ΘΕํ Room/1234 Room/1234/User/A A B C PubSub Room/1234/User/B Room/1234/User/C ݸਓ΁ͷϝοηʔδ͸ݸਓ༻ͷ Topic ΁

Slide 57

Slide 57 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣮ࡍͷήʔϜͰͷ࢖ΘΕํ Room/1234 Room/1234/User/A A B C PubSub Room/1234/User/B Room/1234/User/C ࢀՃऀ͸ શମ༻ Topic ͱ ݸਓ༻ Topic Λ Subscribe ͢Δ Room/1234/User/D D

Slide 58

Slide 58 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࣮૷ ▸ HQSPDΛར༻ ▸ 5PQJDͷ࣮ମ͸&54ʹ֨ೲ͞ΕͨTVCTDSJCF͍ͯ͠ΔϓϩηεͷҰཡ ▸ 4VCTDSJCF͢Δͱ4FTTJPO͕5PQJDΛΩʔʹࣗ෼ͷ1*%Λॻ͖ࠐΉ ▸ 1VCMJTIଆ͸1VCMJTIઌ5PQJDͰ&54͔Β1*%ͷҰཡΛऔಘ ▸ ҰཡʹΉ͚ͯϝοηʔδૹ৴Λߦ͏ 58

Slide 59

Slide 59 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 59 SUBSCRIBEͷ࣮૷ ranch ErlangVM

Slide 60

Slide 60 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 60 SUBSCRIBEͷ࣮૷ ranch ErlangVM session <0.99.0>

Slide 61

Slide 61 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 61 SUBSCRIBEͷ࣮૷ ranch ErlangVM session <0.99.0> Topic/AA <0.80.0> ETS

Slide 62

Slide 62 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 62 SUBSCRIBEͷ࣮૷ ranch session <0.99.0> ErlangVM Write <0.99.0> Topic/AA <0.80.0> ETS

Slide 63

Slide 63 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 63 SUBSCRIBEͷ࣮૷ ranch ErlangVM Topic/AA <0.80.0>, <0.99.0> session <0.99.0> ETS

Slide 64

Slide 64 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ׬ྃ! 64

Slide 65

Slide 65 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 65 PUBLISHͷ࣮૷ ranch session <0.99.0> ErlangVM link Topic/AA <0.80.0>, <0.99.0> Read session <0.80.0> Send ETS

Slide 66

Slide 66 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ׬ྃ! 66

Slide 67

Slide 67 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 67 ΞδΣϯμ ࣮૷ ͳͥϦΞϧλΠϜαʔόʔ͔ ઃܭ Pub/SubϞσϧ

Slide 68

Slide 68 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 68 ϓϩηεσβΠϯ ranch session (system process) session_udp ErlangVM link link

Slide 69

Slide 69 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 69 ϓϩηεσβΠϯ ranch sessions_sup (simple_one_for_one) arkps_sup (one_for_one) session_udp_sup (simple_one_for_one) ErlangVM udp_sup (one_for_one) gen_udp session (system process) session_udp link link

Slide 70

Slide 70 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 70 ϓϩηεσβΠϯ ranch session (system process) session_udp sessions_sup (simple_one_for_one) arkps_sup (one_for_one) session_udp_sup (simple_one_for_one) ErlangVM udp_sup (one_for_one) gen_udp link link

Slide 71

Slide 71 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ϓϩτίϧํ਑ ▸ ਺CJUΛ੾Γ٧ΊΔΑΓγϯϓϧͰ͋Δ͜ͱ ▸ ඞཁे෼ͳ֦ுੑ ▸ γϦΞϥΠζɾύʔε͕͠΍͍͢ ▸ &SMBOH͸όΠφϦʹڧ͍ͷͰ஫ҙ͢Δ 71

Slide 72

Slide 72 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ϓϩτίϧ֓ཁ ▸ 5$1Λ࠾༻ ▸ 5-7ϕʔεͷϓϩτίϧߏ଄ ▸ ࠷খݶͷݻఆϔομͱ֦ு෦෼ͷ5-7Խ ▸ CZUFT୯Ґ 72

Slide 73

Slide 73 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 73 ݻఆϔομ 1 bit 16 32 Type (ϝοηʔδͷछྨ) Length (શମͷ௕͞) SenderTimestamp (ૹ৴ऀ࣌ؒ) ※ޙड़ ReceiverTimesamp (ड৴ऀ࣌ؒ) ※ޙड़ Payload (ૹ৴಺༰)

Slide 74

Slide 74 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 74 PAYLOAD ΋࿈ଓͨ͠ TLV ͷܗΛͱΔ 1 16 32 SectionType SectionLength Value

Slide 75

Slide 75 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͭ·Γ 75

Slide 76

Slide 76 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 76 TLVϕʔε 1 16 32 Type Length SenderTimestamp ReceiverTimesamp SectionType SectionLength Value SectionType SectionLength Value

Slide 77

Slide 77 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͜ͷߏ଄ʹΑΔϝϦοτ ▸ ֦ு͠΍͍͢ ▸ ࣮૷͕ൺֱత༰қ ▸ όϦσʔγϣϯ͕༰қ 77

Slide 78

Slide 78 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE RTTܭଌ ▸ Ϣʔβʔαʔόʔͷճઢঢ়ଶΛ೺Ѳ͍͍ͯͨ͠ ▸ ڧ੍੾அͳͲ ▸ ௨৴࣌ʹࣗ෼ͷλΠϚʔͱॲཧ࣌ؒΛՃࢉͨ͠૬खͷλΠϚʔΛૹΔ 78

Slide 79

Slide 79 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 79 ݻఆϔομ 1 bit 16 32 Type (ϝοηʔδͷछྨ) Length (શମͷ௕͞) SenderTimestamp (ૹ৴ऀ࣌ؒ) ※ޙड़ ReceiverTimesamp (ड৴ऀ࣌ؒ) ※ޙड़ Payload (ૹ৴಺༰)

Slide 80

Slide 80 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 80 RTTܭଌ Client Timer Timer 1ms 1000ms Sender 1 ड৴͔Βૹ৴·Ͱʹ NT͔͔ͬͨͷͰ ड͚औͬͨʹΛ଍͢ Receiver 301 1300ms 321ms ͜ͷ࣌఺Ͱࣗ෼ͷλΠϚʔ͕ ͳͷͰΛҾ͍ͯ355͸NT PubSub 300ms

Slide 81

Slide 81 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 81 RTTܭଌ Client Timer Timer 1 1000 Sender Receiver 1 0 ※ ˞ॳճ͸૬खͷ5JNFS͕ Θ͔Βͳ͍ͷͰ ૹ৴·ͰʹNT͔͔ͬͨͷͰ ड͚औͬͨʹΛ଍͢ Sender Receiver 1300 301 1300 321 ͜ͷ࣌఺Ͱࣗ෼ͷλΠϚʔ͕ ͳͷͰ355͸NT 421 ૹ৴·ͰʹNT͔͔ͬͨͷͰ ड͚औͬͨʹΛ଍͢ Sender Receiver 421 1400 1420 ͜ͷ࣌఺Ͱࣗ෼ͷλΠϚʔ͕ ͳͷͰ355͸NT 521 1520 ͜ͷ࣌఺Ͱࣗ෼ͷλΠϚʔ͕ ͳͷͰ355͸NT Sender Receiver 521 1500 ड৴͔Βૹ৴·ͰʹNT͔͔ͬͨͷͰ ड͚औͬͨʹΛ଍͢ PubSub

Slide 82

Slide 82 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ෛՙࢼݧ 82

Slide 83

Slide 83 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͳͥෛՙࢼݧΛߦ͏͔ ▸ ݶքੑೳΛ஌Δ ▸ ϘτϧωοΫͷൃݟ ▸ εέʔϧ͢Δ͜ͱͷ֬ೝ ▸ ඞཁ༰ྔͷܭࢉ ▸ ίετࢼࢉ 83

Slide 84

Slide 84 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE LOCUSTΛ࠾༻ ▸ 1ZUIPO੡෼ࢄෛՙࢼݧπʔϧ ▸ ͜ͷͨΊʹϑϧػೳͷ1ZUIPOΫϥΠΞϯτΛ։ൃ ▸ ઀ଓଆ͸ͲΜͳʹؤுͬͯ΋ݶք͕͘ΔͷͰ਺Ͱղܾ ▸ εϙοτΠϯελϯε͍҆ ▸ NMBSHFͷεϙοτΠϯελϯε͕UNFEJVNΦϯσϚϯυΑΓ͔҆ͬͨ 84

Slide 85

Slide 85 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 85 Locust Slaves x8 Locust Master m4.2xlarge CPU: 8 RAM: 32GB m4.xlarge CPU: 4 RAM: 16GB PubSub

Slide 86

Slide 86 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 86 ϝοηʔδ਺ܭଌ ਓ෦԰Ͱճ഑৴͢Δͱ ഑৴໋ྩϝοηʔδ֤୺຤΁ϝοηʔδ ߹ܭϝοηʔδ PubSub શһ͕1ඵʹ1ճ഑৴͢Δͱ 8ਓ x 8ϝοηʔδ = ຖඵ64ϝοηʔδ

Slide 87

Slide 87 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 87 user/room 8 message/user/sec 0.2 waittime[ms]5000 rooms 100 1000 2000 3000 3500 3750 (૝ఆ) users 800 8000 16000 24000 28000 30000 (૝ఆ) message/sec 1280 12800 25600 38400 44800 48000 (࣮ଌ) message/sec 1270 12783 25560 38328 44787 21054 (Server)CPU [%] 66.5 360.4 404.7 560.9 635.8 444.4 (Server)RAM [%] 0.6 2.4 3.7 6.2 7.2 16.5 RTT Med [ms] 1 2 5 12 19 34 RTT Avg [ms] 14 11 12 14 27 1260 RTT Max [ms] 44 49 75 356 536 62798 End to End Med [ms] 1 3 8 18 39 460 End to End Avg [ms] 2 12 13 21 50 3017 End to End Max [ms] 747 738 754 1003 1315 100979 66.5 360.4 404.7 560.9 635.8 444.4 1270 12783 25560 38328 44787 21054 0 100 200 300 400 500 600 700 800 0 10000 20000 30000 40000 50000 60000 70000 80000 800 8000 16000 24000 28000 30000 8user/room 1msg/5s (Server)CPU [%] () message/sec

Slide 88

Slide 88 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 88 user/room 8 message/user/sec 1 waittime[ms]1000 rooms 100 400 800 900 1000 (૝ఆ) users 800 3200 6400 7200 8000 (૝ఆ) message/sec 6400 25600 51200 57600 64000 (࣮ଌ) message/sec 6360 25527 48195 53240 25154 (Server)CPU [%] 275.8 618.3 709.3 774.7 766.7 (Server)RAM [%] 0.5 1.1 2.7 3 11.2 RTT Med [ms] 1 1 5 12 5 RTT Avg [ms] 10 7 24 40 494 RTT Max [ms] 45 48 491 1100 34295 End to End Med [ms] 2 11 36 80 310 End to End Avg [ms] 10 13 54 120 970 End to End Max [ms] 554 550 981 2923 44120 275.8 618.3 709.3 774.7 766.7 6360 25527 48195 53240 25154 0 100 200 300 400 500 600 700 800 0 10000 20000 30000 40000 50000 60000 70000 80000 800 3200 6400 7200 8000 8user/room 1msg/s (Server)CPU [%] () message/sec

Slide 89

Slide 89 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 89 user/room 8 message/user/sec 15 waittime[ms] 66.6666666666667 rooms 10 50 70 80 90 100 (૝ఆ) users 80 400 560 640 720 800 (૝ఆ) message/sec 9600 48000 67200 76800 86400 96000 (࣮ଌ) message/sec 9096 46047 58600 60764 63539 26390 (Server)CPU [%] 229.8 694.7 776.6 775.6 787.9 786.2 (Server)RAM [%] 0.4 0.5 0.6 0.7 0.7 8 RTT Med [ms] 38 2 7 10 15 24 RTT Avg [ms] 30 14 15 17 23 194 RTT Max [ms] 45 67 93 122 522 20686 End to End Med [ms] 19 19 24 33 46 73 End to End Avg [ms] 18 21 27 37 50 293 End to End Max [ms] 618 631 657 718 1231 39470 229.8 694.7 776.6 775.6 787.9 786.2 9096 46047 58600 60764 63539 26390 0 100 200 300 400 500 600 700 800 0 10000 20000 30000 40000 50000 60000 70000 80000 80 400 560 640 720 800 8user/room 15msg/s (Server)CPU [%] () message/sec

Slide 90

Slide 90 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE TCP Ͱͷੑೳ͸ 5 ສ MESSAGE / SEC 90

Slide 91

Slide 91 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE UDPͷεϧʔϓοτ͕TCPͷ1/5໰୊ ▸ αϒػೳͱͯ͠6%1Ͱͷૹ৴ػೳΛ͚ͭͯࢼݧΛ͕ͨ͠ສTͰͳ͍ ▸ 7ʮສT͸͍͚Δʯ 91

Slide 92

Slide 92 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE FPROF ͰϓϩϑΝΠϦϯά ▸ TFOE͕஗͍ ▸ QPSU@DPNNBOEͰඇಉظʹͨ͠Βղܾ ▸ HFO@VEQUDQTFOE͕ϘτϧωοΫͳͱ͖ʹ΍Δ͜ͱ  IUUQRJJUBDPNNVSVSVJUFNTCFCCBDFC 92

Slide 93

Slide 93 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 93 user/room 8 message/user/sec 0.2 waittime[ms]5000 rooms 100 2000 3750 6250 7500 8750 (૝ఆ) users 800 16000 30000 50000 60000 70000 (૝ఆ) message/sec 1280 25600 48000 80000 96000 112000 (࣮ଌ) message/sec 1281 25639 48124 78742 94555 97731 (Server)CPU [%] 46.9 312.2 426.8 625 723.2 718 (Server)RAM [%] 2 6.8 9.8 14 17.3 18.7 1281 25639 48124 78742 94555 97731 0 100 200 300 400 500 600 700 800 0 20000 40000 60000 80000 100000 120000 140000 160000 800 16000 30000 50000 60000 70000 8user/room 1msg/5s (Server)CPU [%] () message/sec

Slide 94

Slide 94 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 94 user/room 8 message/user/sec 1 waittime[ms]1000 rooms 100 400 800 1000 1200 1400 1600 (૝ఆ) users 800 3200 6400 8000 9600 11200 12800 (૝ఆ) message/sec 6400 25600 51200 64000 76800 89600 102400 (࣮ଌ) message/sec 6356 25472 51202 63890 75912 88143 99033 (Server)CPU [%] 139.4 269.1 425 500.3 625.8 699.2 765.3 (Server)RAM [%] 2 3.6 5 5 5.2 5.2 6.6 6356 25472 51202 63890 75912 88143 99033 0 100 200 300 400 500 600 700 800 0 20000 40000 60000 80000 100000 120000 140000 160000 800 3200 6400 8000 9600 11200 12800 8user/room 1msg/s (Server)CPU [%] () message/sec

Slide 95

Slide 95 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 95 user/room 8 message/user/sec 15 waittime[ms] 66.6666666666667 rooms 10 50 70 80 90 100 110 120 130 140 (૝ఆ) users 80 400 560 640 720 800 880 960 1040 1120 (૝ఆ) message/sec 9600 48000 67200 76800 86400 96000 105600 115200 124800 134400 (࣮ଌ) message/sec 8912 45967 64648 74035 83211 91581 99596 106775 112889 116739 (Server)CPU [%] 161.5 374.2 489.7 535.8 582.1 632.1 666.4 703.7 726.9 784.7 (Server)RAM [%] 2.1 2.3 2.3 2.3 2.3 2.2 1.8 2.6 3 3.4 8912 45967 64648 74035 83211 91581 99596 106775 112889 116739 0% 100% 200% 300% 400% 500% 600% 700% 800% 0 20000 40000 60000 80000 100000 120000 140000 10 50 70 80 90 100 110 120 130 140 8user/room 15msg/s (Server)CPU [%] () message/sec

Slide 96

Slide 96 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 10 ສ MESSAGE / SEC Λୡ੒ 96

Slide 97

Slide 97 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ෛՙࢼݧ ▸ ࠷ۙߦͳͬͨ࠶ࢼݧͰ͸ෛՙࢼݧπʔϧ͔Βࣗ࡞ͯ͠ߦ͍·ͨ͠ ▸ ৽ଔ೥໨ͷΤϯδχΞ͕ҰਓͰ୲౰ ▸ ඵؒສϝοηʔδΛ͞͹͚ʂ  ϦΞϧλΠϜରઓότϧήʔϜΛࢧ͑Δෛՙࢼݧͷ௅ઓʲ%FWFMPQFST#PPTUʳ ▸ IUUQTDPEF[JOFKQBSUJDMFEFUBJM 97

Slide 98

Slide 98 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE HWো֐΁ͷରԠ 98

Slide 99

Slide 99 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 99 HEART BEAT App PubSub PubSub PubSub HTTP API A 2019-01-01 00:00:00 alive B 2019-01-01 00:00:00 alive C 2019-01-01 00:00:00 alive -4 sec -4 sec -4 sec ఆظత(5ඵʹ1ճ)ʹAPIαʔόʔͷAPIΛݺͼग़͠ RDBMSʹ࠷ऴݺͼग़͠೔࣌, εςʔλεΛอଘ

Slide 100

Slide 100 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 100 HEART BEAT App PubSub PubSub PubSub HTTP API A 2019-01-01 00:00:00 dead B 2019-01-01 01:00:00 alive C 2019-01-01 01:00:00 alive A B C -3604 sec -4 sec -4 sec αʔόʔো֐

Slide 101

Slide 101 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 101 HEART BEAT App PubSub PubSub PubSub HTTP API A 2019-01-01 00:00:00 dead B 2019-01-01 01:00:00 alive C 2019-01-01 01:00:00 alive A B C -3604 sec -4 sec -4 sec ௨৴ো֐

Slide 102

Slide 102 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 102 HEART BEAT App PubSub PubSub PubSub HTTP API A 2019-01-01 00:00:00 dead B 2019-01-01 01:00:00 alive C 2019-01-01 01:00:00 alive A B C -3604 sec -4 sec -4 sec ௨৴ো֐͔Β෮چ

Slide 103

Slide 103 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 103 HEART BEAT App PubSub PubSub PubSub HTTP API A 2019-01-01 00:00:00 dead B 2019-01-01 01:00:00 alive C 2019-01-01 01:00:00 alive A B C -3604 sec -4 sec -4 sec Ұ౓ࢮ๢൑ఆͨ͠αʔόʔͷ෮ؼ͸ ઈରʹೝΊͳ͍ ௨৴ো֐͔Β෮چ

Slide 104

Slide 104 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE લఏ ▸ "SL144FSWFS͸ಉҰαʔόʔʹܨ͕͍ͬͯΔϢʔβʔಉ࢜Ͱ͔͠௨৴Ͱ͖ͳ͍ ▸ ΫϥελϦϯά΋ߟྀ͕ͨ͠ ▸ ෳࡶੑ ▸ ஗Ԇ 104

Slide 105

Slide 105 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 105 ͳͥ෮ؼΛೝΊͳ͍͔ App Connection 1. heartbeat 2. OK 1VC4VC 1VC4VC Server A Server B RoomA Server A RoomB Server B

Slide 106

Slide 106 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 106 ͳͥ෮ؼΛೝΊͳ͍͔ App Connection 1. heartbeat 2. OK 1VC4VC 1VC4VC Server A Server B RoomA Server A RoomB Server B

Slide 107

Slide 107 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 107 ͳͥ෮ؼΛೝΊͳ͍͔ App 1. RoomA Server? 1VC4VC 2. Server B IP, Port … 1VC4VC Server A Server B RoomA Server B RoomB Server B Connection

Slide 108

Slide 108 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ΊͰͨ͠ΊͰͨ͠… 108

Slide 109

Slide 109 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 109 ͳͥ෮ؼΛೝΊͳ͍͔ App Connection 1. heartbeat 2. OK 1VC4VC 1VC4VC Server A Server B RoomA Server A RoomB Server B

Slide 110

Slide 110 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 110 ͳͥ෮ؼΛೝΊͳ͍͔ App Connection 1. heartbeat 2. OK 1VC4VC 1VC4VC Server A Server B RoomA Server A RoomB Server B 1. RoomA Server? 2. Server B IP, Port …

Slide 111

Slide 111 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 111 ͳͥ෮ؼΛೝΊͳ͍͔ App Connection 1. heartbeat 2. OK 1VC4VC 1VC4VC Server A Server B RoomA Server B RoomB Server B 1. RoomA Server? 2. Server B IP, Port … Connection ಉ͡αʔόʔʹ͍Δ΂͖Ϣʔβʔ͕ ผͷαʔόʔʹ

Slide 112

Slide 112 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE όʔδϣϯΞοϓ ▸ ࠓܨ͍Ͱ͍Δਓ ▸ ΫϥΠΞϯτͷόʔδϣϯYαʔόʔͷόʔδϣϯ ▸ ৽͍͠αʔόʔΛͨͯͯɺ"QQͰͷ৽ن઀ଓͷҊ಺ઌΛมߋ ▸ طଘͷ઀ଓ͕͍ͳ͘ͳͬͨΒམͱͯ͠ऴྃ 112

Slide 113

Slide 113 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ROOM WORKER ▸ 3PPN৘ใΛήʔϜϩδοΫ·ͰؚΊͯ໘౗ΛΈΔ ▸ ࠷ॳʹ4VCTDSJCF͢Δ ▸ ϒϩʔΧʔ͔Β͸ΫϥΠΞϯτѻ͍ 113

Slide 114

Slide 114 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE 114 PubSub App Battle ॏཁͳ৘ใ͸HTTP API ϦΞϧλΠϜͷ΍ΓऔΓ͸ Ark

Slide 115

Slide 115 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ࡞ͬͯΈͯ ▸ ࢥͬͨΑΓ଎౓͕ग़ͨ ▸ σάϨͤ͞ͳ͍ࣄͷॏཁ͞ ▸ ӡ༻Λ࠷༏ઌͷ࢟੎ 115

Slide 116

Slide 116 text

$PQZSJHIU $ HVNJ*OD"MM3JHIUT3FTFSWFE ͋Γ͕ͱ͏ ͍͟͝·ͨ͠ 116