DAU 100 万人突破！急成長を支える Shadowverse のインフラ技術

by Cygames

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

ΞδΣϯμ 4IBEPXWFSTFΛࢧ͑ ΔΠ ϯϑϥٕज़ εέʔϥϒϧͳϩάऩूج൫ େن໛؀ڥγεςϜͱͭͷνϟ Ϩϯδ ೔5#ͷϩάऩूج൫

Slide 4

Slide 4 text

4IBEPXWFSTFΛࢧ͑ Δ Π ϯϑϥٕज़

Slide 5

Slide 5 text

Slide 6

Slide 6 text

େن໛؀ڥ

Slide 7

Slide 7 text

"QQMJDBUJPO4FSWFST Node.js Nginx 6TFS$MJFOU "QQMJDBUJPO4FSWFST Apache / PHP %BUBCBTF4FSWFST MySQL HTTPS WebSocket Elastic Load Balancing Elastic Load Balancing $BTIF4FSWFST Memcached Redis

Slide 8

Slide 8 text

ͭͷνϟ Ϩϯδ શσʔλͷอଘͱσʔλϩʔυ ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε /HJOYʹΑ Δಠ࣮ࣗ૷ͷ-ϩʔυόϥϯα ରઓγεςϜͷεέʔϧΞ΢ τ

Slide 9

Slide 9 text

૒ํ޲Ϧ ΞϧλΠ Ϝ௨৴ ࣋ଓత઀ଓͷཱ֬ 8FC4PDLFU ରઓ૬खʹ Ϧ ΞϧλΠ Ϝʹ σʔλ൓ө

Slide 10

Slide 10 text

Ϧ ΞϧλΠ Ϝ௨৴ͷαʔόߏ੒ ΫϥΠ Ξϯ τ͸3PPN*%Ͱ άϧʔϓԽ͞ΕΔ αʔό͸ϝ οηʔδΛ  ΫϥΠ Ξϯ τʹڞ༗͢Δ  ࢓૊ΈΛ࣋ͭ ෼ࢄϦ ΞϧλΠ Ϝ௨৴ͷ  ߏ੒ͷݕ౼ Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) RoomID 1 RoomID 2 4FSWFS

Slide 11

Slide 11 text

1VC4VCϝ οηʔδϯάϞσϧ 1VC4VCϞσϧબఆ ࣮૷͕༰қ ୯Ұো֐఺ ʢ410'ʣ 1VC4VC͕Ϙ τϧωο Ϋʹ εέʔϧΞ΢ τͷ՝୊ ແఀࢭͷมߋ͕ߦ͑ͳ͍ Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) RoomID 1 /PEF RoomID 2 /PEF RoomID 1 RoomID 3 /PEF RoomID 2 3FEJT1VC4VC RoomID 1 RoomID 3 Client D (RoomID 3) Layer4 LB(Elastic Load Balancing)

Slide 12

Slide 12 text

ϧʔϜ*%ରԠ-ϩʔυόϥϯαͷ։ൃ 410'ͷഉআ ϊ ʔυ୆਺Λಈతʹมߋ Ϛ ονϯάϩδο Ϋ Λ࣮૷ ಉҰϧʔϜ*%Λ  ಉҰαʔόʹಈతϧʔς Ο ϯά Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) Lua Nginx 1 Layer4 LB(Elastic Load Balancing) Lua Application Load Balancer Cluster Lua Nginx N Client D (RoomID 3) Nginx 2 … Room 1 Node 1 Room 2 Room 3 Node 3 Node 2 Dynamic Routing

Slide 13

Slide 13 text

/HJOY-ϩʔυόϥϯαͱ-VBεΫ Ϧϓ τ /HJOY 8FC4PDLFU௨৴ ʢ૒ํ޲௨৴ʣ ͷϩʔυόϥϯα -VBϞδϡʔϧʹΑ Γϩδο ΫػೳΛ࣮૷ -VBεΫ Ϧϓ τ ϔο μʹ෇༩͞ΕͨϧʔϜ*%Λݩʹಈత෼ࢄ 8FC"1*ʹΑ Γແఀࢭʹઃఆมߋ

Slide 14

Slide 14 text

෼ࢄϩδο Ϋͷ࣮૷ ෼ࢄϩδο Ϋ ϧʔϜ*%ͷϋο γϡ஋ͱ/PEFαʔό਺ͷ৒༨ࢉ ʢNPEʣ Ͱ ઀ଓઌϊ ʔυ͕ܾఆ͞ΕΔ $POTJTUFOU)BTIJOH ʢϋο γϡܭࢉʣ ϊ ʔυͷ௥Ճ ɾ ࡟আΛ࠷খݶͷมߋͰϚ ο ϐϯά

Slide 15

Slide 15 text

ສηο γϣϯ ಉ࣌ରઓ਺ສୡ੒

Slide 16

Slide 16 text

ͭͷνϟ Ϩϯδ શσʔλͷอଘͱσʔλϩʔυ ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε /HJOYʹΑ Δಠ࣮ࣗ૷ͷ-ϩʔυόϥϯα ରઓγεςϜͷεέʔϧΞ΢ τ

Slide 17

Slide 17 text

ϦϓϨΠػೳ ࠷৽݅ͷότϧΛ  ϦϓϨΠ ͢Δ શରઓσʔλͷอଘ շదͳԠ౴଎౓

Slide 18

Slide 18 text

(BNF4ZTUFN 3FQMBZ4ZTUFN ϦϓϨΠσʔλͷอଘͱ࠶ੜ Amazon S3 Redis SQS Web Servers  (PHP) Worker Servers  (PHP)

Slide 19

Slide 19 text

ϦϓϨΠػೳͷσʔλ

Slide 20

Slide 20 text

ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε ࣮੷ͷ๛෋ͳ"84αʔϏεΛ࠾༻ 424ʹΑ ΔඇಉظԽͰػೳ෼཭ "NB[PO4ͷ҆ఆͨ͠σʔλऔಘ ຊ൪؀ڥʹ͓͚Δ௕ظؒͷݕূ Ϣʔβʹ͸ݟͤͳ͍ඇެ։Ϧ Ϧʔε

Slide 21

Slide 21 text

· ͱΊ ୯Ұো֐఺ͷͳ͍֦ுՄೳͳαʔόߏ੒ ϐʔΫ࣌ͷ௨৴ྔ ɾ ΞΫηε͕ଟ͘ɺ ༧ଌࠔ೉ ରઓγεςϜΛಠٕࣗज़Ͱ࣮૷ େ༰ྔσʔλͷอଘɺ σʔλ׬શੑͷ֬อ "NB[PO3%4 "NB[PO4Λར༻

Slide 22

Slide 22 text

εέʔϥϒϧͳ  ϩάऩूج൫

Slide 23

Slide 23 text

ιʔγϟϧήʔϜͱ ϩά $4ରԠ ,1*ࢦඪͷऔಘ γεςϜෆ۩߹ͷௐࠪ

Slide 24

Slide 24 text

σʔλͷछྨͱྔ ςΩε τϩά ΞΫηεϩάɺ Τϥʔϩάɺ ό τϧϩάɺ 42-ϩάʜ %BUBCBTF*OTFSUϩά ՝ۚϩάɺ ΞΠ ςϜϩάɺ ߦಈϩά

Slide 25

Slide 25 text

˞4IBEPXWFSTFҎ֎΋ؚΉ

Slide 26

Slide 26 text

ςΩε τϩάऩूͷ՝୊ ϩάऔΓ ͜΅͠ ҰߦͷαΠ ζ੍ݶ όο ϑ ΝΩϡʔҲΕ தܧαʔόͷෛՙ ϩάσʔλ૿େʹΑ Δ  εέʔϧΞ΢ τͷ՝୊ Web Server 1 Error log KPI log Log Proxy 3 4UPSBHF"SSBZ Web Server 2 Web Server N SQL log Log Proxy 2 Access log Log Proxy 1

Slide 27

Slide 27 text

.Z42-Π ϯαʔ τ ϩάͷ՝୊ Ϩεϙϯε஗Ԇ .Z42-*OTFSUॲཧͷ஗Ԇ ϩάσʔλ૿େ ϨϓϦέʔγϣϯ஗Ԇ Web Server 1 Action log Web Server 2 Web Server N Item log MySQL Server Payment log

Slide 28

Slide 28 text

ಠࣗϩάసૹΤʔδΣϯ τͷ։ൃ "NB[PO4ʹϩάΛू໿ ̎ͭͷϩάసૹΤʔδΣϯ τ Λ։ൃ 4UBMLFSςΩε τϩάͷసૹ 1PSUFS%#*OTFSUϩάͷඇಉظసૹ ߏ଄Խσʔλͷసૹͱ424ϝ οηʔδ࡞੒

Slide 29

Slide 29 text

4UBMLFSͱ1PSUFS औΓ ͜΅͠๷ࢭͱ࠶઀ଓ సૹͰ͖ͳ͔ͬͨϩά͸શͯϩʔΧϧʹอଘ &YQPOFOUJBM#BDLP⒎Ϧ τϥΠִؒͷௐ੔ +JUUFS ʢ͹Β͖ͭʣ Λಋೖ͠ɺ ࠶઀ଓཁٻΛ෼ࢄ

Slide 30

Slide 30 text

4UBMLFSͱ1PSUFS "NB[PO4ͷύϑ ΥʔϚϯε޲্ ઌ಄จࣈʹϋο γϡϓϨϑ Ο ο Ϋ ε௥Ճ͠ɺ   JOEFYΛෳ਺ύʔς Ο γϣϯʹ෼ࢄ ྫ ɿ 4CVDLFU/MpMF໊

Slide 31

Slide 31 text

"TZODISPOPVT-PH 4ZTUFN (BNF 4ZTUFN CVTJOFTT 4ZTUFN 1PSUFSͱඇಉظ%#ϩάγεςϜ Worker Servers (Amazon EC2) Log Datebase (Amazon Aurora) σʔλղੳ CSରԠ Porter Web Servers Amazon S3 SQS TSV

Slide 32

Slide 32 text

"844΁ϩάू໿ Kibana Google BigQuery Mackerel Amazon Aurora Redshift Porter/Stalker Amazon S3

Slide 33

Slide 33 text

Slide 34

Slide 34 text

ϩά౷ܭͱ άϥϑඳը )551εςʔλεͷΞΫηε਺

Slide 35

Slide 35 text

ώʔ τϚοϓ ΞΫηεϩάͱ࠲ඪ৘ใΛϚο ϐϯά

Slide 36

Slide 36 text

.BDLFSFM "QBDIFϨεϙϯελΠ ϜඵҎ্ͷ෼෍

Slide 37

Slide 37 text

"NB[PO4ͱ֎෦πʔϧͷ࿈ܞ εέʔϥϒϧʹ  Π ϯελϯεΛ૿ݮ ϩάͷྔ͸ϐʔΫ࣌ͱ  ࠷খͷ࣌Ͱ໿ഒͷࠩ αʔόϨεͰ  ӡ༻޻਺Λܰݮ Google BigQuery Porter/Stalker Amazon S3

Slide 38

Slide 38 text

1.Put Event Trigger 2.Create Message 3.C ount M essages &$4Λར༻ͨ͠εέʔϥϒϧͳϩά࿈ܞ Google Cloud Storage Google BigQuery Auto Scaling 4.Run Task 6.Save Objects SQL Query Cron Job Amazon S3 Amazon Lamdba Amazon Lamdba SQS Amazon EC2 Container Service(ECS) Ϋϥελͷ$16࢖༻཰ʹԠͯ͡ Π ϯελϯε਺Λ૿ݮ 5.Get Files ϩάྔʹԠͯ͡ίϯςφىಈ਺Λ૿ݮ

Slide 39

Slide 39 text

ͭͷεέʔϦϯάϙϦ γͷ࿈ಈ -BNCEB'VODUJPO ϩάྔʹԠͯ͡ɺ   ίϯςφىಈ਺Λ্͛Δ &$"VUP4DBMJOH ىಈதίϯςφ਺͕  ্ঢʹΑ Γ$16࢖༻཰্ঢ &$εέʔϦϯάϙϦ γʹΑ Γ  $POUBJOFS*OTUBODF૿Ճ &$4Ϋϥελʔͷ$16࢖༻཰ $POUBJOFS*OTUBODFىಈ਺

Slide 40

Slide 40 text

ίε τ࡟ݮͱࣗಈԽͷ࣮ݱ αʔόϨεʺίε τ࡟ݮ -BNCEB&$4ʹΑ Γ ίε τ Λʹ࡟ݮ ίϯςφٕज़ͷ࠾༻ -BNCEBͱ&$4ͷద༻ൣғ όονॲཧɺ Π ϕϯ τ τ ϦΨʔ͸-BNCEB ৗ࣮࣌ߦ͢Δॲཧ͸&$4

Slide 41

Slide 41 text

"NB[PO"UIFOBͷར༻ 1.Put Event Trigger 2.Create Message 3.C ount M essages Auto Scaling 4.Run Task Cron Job Amazon S3 Amazon Lamdba Amazon Lamdba SQS Amazon EC2 Container Service(ECS) 5.Get Files SQL Query Amazon S3 Amazon Athena 6.Save Objects Google Cloud Storage Google BigQuery SQL Query

Slide 42

Slide 42 text

"NB[PO"UIFOBͷར༻ ύεม׵ʹΑ Δύʔς Ο γϣϯԽ 4UBMLFSͷ4ύεܗࣜ "NB[PO"UIFOBͷ4ύεܗࣜ S3://athena-examples/l2xa-<ﬁle>.20150101 s3://athena-examples/ex/2015/01/01/<ﬁle>

Slide 43

Slide 43 text

· ͱΊ ϩάू໿ʹΑ Γσʔλ׆༻͕ଅਐ ඇಉظԽΛߦ͍ϨεϙϯεΛվળ αʔόϨε ͱࣗಈԽͰίε τ ͱ޻਺࡟ݮ

Slide 44

Slide 44 text

$ZHBNFTΠ ϯϑϥ͕ େ੾ʹ͍ͯ͠Δ ͜ ͱ

Slide 45

Slide 45 text

౰ͨ Γલͷ͜ ͱ Λ  ౰ͨ Γલʹ΍Δ

Slide 46

Slide 46 text

Π ϯϑϥͷϨΠ ϠʔͰ  ՝୊Λղܾ͢Δ

Slide 47

Slide 47 text

ίΞٕज़͸  ࣗ෼ͨͪͰ࣮૷͢Δ

Slide 48

Slide 48 text

5IBOLZPV