DAU 100 万人突破! 急成長を支える Shadowverse のインフラ技術

510ec964f5d26c2724c883fd7b671e3d?s=47 Cygames
June 02, 2017

DAU 100 万人突破! 急成長を支える Shadowverse のインフラ技術

2017/06/01 Game Tech Session ~AWS Summit Tokyo 2017~

510ec964f5d26c2724c883fd7b671e3d?s=128

Cygames

June 02, 2017
Tweet

Transcript

  1. 

  2.  

  3.  ΞδΣϯμ 4IBEPXWFSTFΛࢧ͑ ΔΠ ϯϑϥٕज़ εέʔϥϒϧͳϩάऩूج൫ େن໛؀ڥγεςϜͱͭͷνϟ Ϩϯδ ೔5#ͷϩάऩूج൫

  4.  4IBEPXWFSTFΛࢧ͑ Δ Π ϯϑϥٕज़

  5. 

  6.  େن໛؀ڥ

  7.  "QQMJDBUJPO4FSWFST Node.js Nginx 6TFS$MJFOU "QQMJDBUJPO4FSWFST Apache / PHP %BUBCBTF4FSWFST

    MySQL HTTPS WebSocket Elastic Load Balancing Elastic Load Balancing $BTIF4FSWFST Memcached Redis
  8.  ͭͷνϟ Ϩϯδ શσʔλͷอଘͱσʔλϩʔυ ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε /HJOYʹΑ Δಠ࣮ࣗ૷ͷ-ϩʔυόϥϯα ରઓγεςϜͷεέʔϧΞ΢ τ

  9. ૒ํ޲Ϧ ΞϧλΠ Ϝ௨৴ ࣋ଓత઀ଓͷཱ֬ 8FC4PDLFU  ରઓ૬खʹ Ϧ ΞϧλΠ Ϝʹ

    σʔλ൓ө 
  10. Ϧ ΞϧλΠ Ϝ௨৴ͷαʔόߏ੒ ΫϥΠ Ξϯ τ͸3PPN*%Ͱ άϧʔϓԽ͞ΕΔ αʔό͸ϝ οηʔδΛ
 ΫϥΠ

    Ξϯ τʹڞ༗͢Δ
 ࢓૊ΈΛ࣋ͭ ෼ࢄϦ ΞϧλΠ Ϝ௨৴ͷ
 ߏ੒ͷݕ౼  Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) RoomID 1 RoomID 2 4FSWFS
  11. 1VC4VCϝ οηʔδϯάϞσϧ 1VC4VCϞσϧબఆ ࣮૷͕༰қ ୯Ұো֐఺ ʢ410'ʣ  1VC4VC͕Ϙ τϧωο Ϋʹ

    εέʔϧΞ΢ τͷ՝୊ ແఀࢭͷมߋ͕ߦ͑ͳ͍  Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) RoomID 1 /PEF RoomID 2 /PEF RoomID 1 RoomID 3 /PEF RoomID 2 3FEJT1VC4VC RoomID 1 RoomID 3 Client D (RoomID 3) Layer4 LB(Elastic Load Balancing)
  12. ϧʔϜ*%ରԠ-ϩʔυόϥϯαͷ։ൃ 410'ͷഉআ ϊ ʔυ୆਺Λಈతʹมߋ Ϛ ονϯάϩδο Ϋ Λ࣮૷ ಉҰϧʔϜ*%Λ
 ಉҰαʔόʹಈతϧʔς

    Ο ϯά  Client A (RoomID 1) Client B (RoomID 2) Client C (RoomID 1) Lua Nginx 1 Layer4 LB(Elastic Load Balancing) Lua Application Load Balancer Cluster Lua Nginx N Client D (RoomID 3) Nginx 2 … Room 1 Node 1 Room 2 Room 3 Node 3 Node 2 Dynamic Routing
  13.  /HJOY-ϩʔυόϥϯαͱ-VBεΫ Ϧϓ τ /HJOY 8FC4PDLFU௨৴ ʢ૒ํ޲௨৴ʣ ͷϩʔυόϥϯα -VBϞδϡʔϧʹΑ Γϩδο

    ΫػೳΛ࣮૷ -VBεΫ Ϧϓ τ ϔο μʹ෇༩͞ΕͨϧʔϜ*%Λݩʹಈత෼ࢄ 8FC"1*ʹΑ Γແఀࢭʹઃఆมߋ
  14.  ෼ࢄϩδο Ϋͷ࣮૷ ෼ࢄϩδο Ϋ ϧʔϜ*%ͷϋο γϡ஋ͱ/PEFαʔό਺ͷ৒༨ࢉ ʢNPEʣ Ͱ ઀ଓઌϊ

    ʔυ͕ܾఆ͞ΕΔ $POTJTUFOU)BTIJOH ʢϋο γϡܭࢉʣ  ϊ ʔυͷ௥Ճ ɾ ࡟আΛ࠷খݶͷมߋͰϚ ο ϐϯά
  15.  ສηο γϣϯ ಉ࣌ରઓ਺ສୡ੒

  16.  ͭͷνϟ Ϩϯδ શσʔλͷอଘͱσʔλϩʔυ ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε /HJOYʹΑ Δಠ࣮ࣗ૷ͷ-ϩʔυόϥϯα ରઓγεςϜͷεέʔϧΞ΢ τ

  17. ϦϓϨΠػೳ ࠷৽݅ͷότϧΛ
 ϦϓϨΠ ͢Δ શରઓσʔλͷอଘ շదͳԠ౴଎౓ 

  18.  (BNF4ZTUFN 3FQMBZ4ZTUFN ϦϓϨΠσʔλͷอଘͱ࠶ੜ Amazon S3 Redis SQS Web Servers


    (PHP) Worker Servers
 (PHP)
  19.  ϦϓϨΠػೳͷσʔλ

  20.  ϦϓϨΠػೳͷ೥຤Ϧ Ϧʔε ࣮੷ͷ๛෋ͳ"84αʔϏεΛ࠾༻ 424ʹΑ ΔඇಉظԽͰػೳ෼཭ "NB[PO4ͷ҆ఆͨ͠σʔλऔಘ ຊ൪؀ڥʹ͓͚Δ௕ظؒͷݕূ Ϣʔβʹ͸ݟͤͳ͍ඇެ։Ϧ Ϧʔε

  21.  · ͱΊ ୯Ұো֐఺ͷͳ͍֦ுՄೳͳαʔόߏ੒ ϐʔΫ࣌ͷ௨৴ྔ ɾ ΞΫηε͕ଟ͘ɺ ༧ଌࠔ೉ ରઓγεςϜΛಠٕࣗज़Ͱ࣮૷ େ༰ྔσʔλͷอଘɺ

    σʔλ׬શੑͷ֬อ "NB[PO3%4 "NB[PO4Λར༻
  22.  εέʔϥϒϧͳ
 ϩάऩूج൫

  23.  ιʔγϟϧήʔϜͱ ϩά $4ରԠ ,1*ࢦඪͷऔಘ γεςϜෆ۩߹ͷௐࠪ

  24.  σʔλͷछྨͱྔ ςΩε τϩά ΞΫηεϩάɺ Τϥʔϩάɺ ό τϧϩάɺ 42-ϩάʜ %BUBCBTF*OTFSUϩά

    ՝ۚϩάɺ ΞΠ ςϜϩάɺ ߦಈϩά
  25.  ˞4IBEPXWFSTFҎ֎΋ؚΉ

  26. ςΩε τϩάऩूͷ՝୊ ϩάऔΓ ͜΅͠ ҰߦͷαΠ ζ੍ݶ όο ϑ ΝΩϡʔҲΕ தܧαʔόͷෛՙ

    ϩάσʔλ૿େʹΑ Δ
 εέʔϧΞ΢ τͷ՝୊  Web Server 1 Error log KPI log Log Proxy 3 4UPSBHF"SSBZ Web Server 2 Web Server N SQL log Log Proxy 2 Access log Log Proxy 1
  27. .Z42-Π ϯαʔ τ ϩάͷ՝୊ Ϩεϙϯε஗Ԇ .Z42-*OTFSUॲཧͷ஗Ԇ ϩάσʔλ૿େ ϨϓϦέʔγϣϯ஗Ԇ  Web

    Server 1 Action log Web Server 2 Web Server N Item log MySQL Server Payment log
  28.  ಠࣗϩάసૹΤʔδΣϯ τͷ։ൃ "NB[PO4ʹϩάΛू໿ ̎ͭͷϩάసૹΤʔδΣϯ τ Λ։ൃ 4UBMLFSςΩε τϩάͷసૹ 1PSUFS%#*OTFSUϩάͷඇಉظసૹ

    ߏ଄Խσʔλͷసૹͱ424ϝ οηʔδ࡞੒
  29.  4UBMLFSͱ1PSUFS औΓ ͜΅͠๷ࢭͱ࠶઀ଓ సૹͰ͖ͳ͔ͬͨϩά͸શͯϩʔΧϧʹอଘ &YQPOFOUJBM#BDLP⒎Ϧ τϥΠִؒͷௐ੔ +JUUFS ʢ͹Β͖ͭʣ Λಋೖ͠ɺ

    ࠶઀ଓཁٻΛ෼ࢄ
  30.  4UBMLFSͱ1PSUFS "NB[PO4ͷύϑ ΥʔϚϯε޲্ ઌ಄จࣈʹϋο γϡϓϨϑ Ο ο Ϋ ε௥Ճ͠ɺ

    
 JOEFYΛෳ਺ύʔς Ο γϣϯʹ෼ࢄ ྫ ɿ 4CVDLFU/MpMF໊
  31.  "TZODISPOPVT-PH 4ZTUFN (BNF 4ZTUFN CVTJOFTT 4ZTUFN 1PSUFSͱඇಉظ%#ϩάγεςϜ Worker Servers

    (Amazon EC2) Log Datebase (Amazon Aurora) σʔλղੳ CSରԠ Porter Web Servers Amazon S3 SQS TSV
  32.  "844΁ϩάू໿ Kibana Google BigQuery Mackerel Amazon Aurora Redshift Porter/Stalker

    Amazon S3
  33. 

  34.  ϩά౷ܭͱ άϥϑඳը )551εςʔλεͷΞΫηε਺

  35.  ώʔ τϚοϓ ΞΫηεϩάͱ࠲ඪ৘ใΛϚο ϐϯά

  36.  .BDLFSFM "QBDIFϨεϙϯελΠ ϜඵҎ্ͷ෼෍

  37. "NB[PO4ͱ֎෦πʔϧͷ࿈ܞ εέʔϥϒϧʹ
 Π ϯελϯεΛ૿ݮ ϩάͷྔ͸ϐʔΫ࣌ͱ
 ࠷খͷ࣌Ͱ໿ഒͷࠩ αʔόϨεͰ
 ӡ༻޻਺Λܰݮ  Google

    BigQuery Porter/Stalker Amazon S3
  38.  1.Put Event Trigger 2.Create Message 3.C ount M essages

    &$4Λར༻ͨ͠εέʔϥϒϧͳϩά࿈ܞ Google Cloud Storage Google BigQuery Auto Scaling 4.Run Task 6.Save Objects SQL Query Cron Job Amazon S3 Amazon Lamdba Amazon Lamdba SQS Amazon EC2 Container Service(ECS) Ϋϥελͷ$16࢖༻཰ʹԠͯ͡ Π ϯελϯε਺Λ૿ݮ 5.Get Files ϩάྔʹԠͯ͡ίϯςφىಈ਺Λ૿ݮ
  39. ͭͷεέʔϦϯάϙϦ γͷ࿈ಈ -BNCEB'VODUJPO ϩάྔʹԠͯ͡ɺ 
 ίϯςφىಈ਺Λ্͛Δ &$"VUP4DBMJOH ىಈதίϯςφ਺͕
 ্ঢʹΑ Γ$16࢖༻཰্ঢ

    &$εέʔϦϯάϙϦ γʹΑ Γ
 $POUBJOFS*OTUBODF૿Ճ  &$4Ϋϥελʔͷ$16࢖༻཰ $POUBJOFS*OTUBODFىಈ਺
  40.  ίε τ࡟ݮͱࣗಈԽͷ࣮ݱ αʔόϨεʺίε τ࡟ݮ -BNCEB&$4ʹΑ Γ ίε τ Λʹ࡟ݮ

    ίϯςφٕज़ͷ࠾༻ -BNCEBͱ&$4ͷద༻ൣғ όονॲཧɺ Π ϕϯ τ τ ϦΨʔ͸-BNCEB ৗ࣮࣌ߦ͢Δॲཧ͸&$4
  41.  "NB[PO"UIFOBͷར༻ 1.Put Event Trigger 2.Create Message 3.C ount M

    essages Auto Scaling 4.Run Task Cron Job Amazon S3 Amazon Lamdba Amazon Lamdba SQS Amazon EC2 Container Service(ECS) 5.Get Files SQL Query Amazon S3 Amazon Athena 6.Save Objects Google Cloud Storage Google BigQuery SQL Query
  42. "NB[PO"UIFOBͷར༻ ύεม׵ʹΑ Δύʔς Ο γϣϯԽ 4UBMLFSͷ4ύεܗࣜ "NB[PO"UIFOBͷ4ύεܗࣜ  S3://athena-examples/l2xa-<file>.20150101 s3://athena-examples/ex/2015/01/01/<file>

  43.  · ͱΊ ϩάू໿ʹΑ Γσʔλ׆༻͕ଅਐ ඇಉظԽΛߦ͍ϨεϙϯεΛվળ αʔόϨε ͱࣗಈԽͰίε τ ͱ޻਺࡟ݮ

  44.  $ZHBNFTΠ ϯϑϥ͕ େ੾ʹ͍ͯ͠Δ ͜ ͱ

  45.  ౰ͨ Γલͷ͜ ͱ Λ
 ౰ͨ Γલʹ΍Δ

  46.  Π ϯϑϥͷϨΠ ϠʔͰ
 ՝୊Λղܾ͢Δ

  47.  ίΞٕज़͸
 ࣗ෼ͨͪͰ࣮૷͢Δ

  48.  5IBOLZPV