モンストを支えるインフラの今とこれから

dots. Conference Spring 2016
ゲーム開発の裏側
http://eventdots.jp/event/580344

Transcript

1. ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β
   2016.3.1 dots. Conference Spring 2016
   ήʔϜ։ൃͷཪଆ
   mixi, Inc. XFLAG™ STUDIO
   @isaoshimizu
   

   View Slide

2. About Me
   • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu
   • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ
   • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ
   • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β

   εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁
   2
   

   View Slide

3. ϞϯελʔετϥΠΫ
   • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε
   • iOS൛, Android൛Λఏڙ
   • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ
   • ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։
   • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ
   3
   

   View Slide

4. ͔͜͜Β͸ϞϯελʔετϥΠΫ
   ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢
   4
   

   View Slide

5. ਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ
   5
   

   View Slide

6. 6
   Application
   300+ Servers
   Resque
   50 Servers
   Redis
   30 Servers
   Memcached
   200+ Servers
   CDN Traffic
   Max 270Gbps
   API Traffic
   Avg 1Gbps
   TURN
   40 Servers
   MariaDB
   250+ Servers
   Internal Traffic
   Max 20Gbps
   

   View Slide

7. Πϯϑϥߏ੒
   7
   

   View Slide

8. 8
   DC1 DC2
   Main Main/Backup Main/Dev
   

   View Slide

9. ෳ਺DCͱAmazon VPC
   • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒
   • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ
   • ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ
   • ৄ͍͠ղઆ

   “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”

   http://xflag.com/blog/infradb/internal_traffic_volume.html
   9
   

   View Slide

10. ֤ڌ఺ͷ໾ׂ
    10
    

    View Slide

11. 11
    DC1 DC2
    Main Main/Backup Main/Dev
    Application
    MariaDB
    Redis
    Resque
    Memcached
    MariaDB
    Redis
    Memcached
    TURN Server
    ։ൃ༻Πϯελϯε܈
    Application
    Resque
    

    View Slide

12. ֤ڌ఺ͷ໾ׂ
    • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ
    • ຊ൪ͷRedisɺMariaDBɺMemcached
    • DC1ʹMaster/Backupͷηοτ
    • DC2ʹBackup
    • Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ
    • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ
    12
    

    View Slide

13. ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ
    13
    

    View Slide

14. 14
    DC1 DC2 AWS
    TURN Server (EC2)
    CDN (CloudFront, Akamai)
    API API
    LB LB
    Internet
    Client
    

    View Slide

15. ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ
    • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁
    • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ
    • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚
    • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2 + Elastic IPʣ
    • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ
    15
    

    View Slide

16. σϓϩΠ
    16
    

    View Slide

17. 17
    DC1 DC2
    Deploy Server
    S3 Bucket
    tarball
    S3 PUT
    GitHub
    ssh
    Code
    Internet
    

    View Slide

18. σϓϩΠ
    • Capistrano 2
    
    • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ
    
    • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli
    
    • GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒
    
    • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։
    
    • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ
    18
    

    View Slide

19. • capistrano-rbenv https://github.com/yyuu/capistrano-rbenv
    
    • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ
    
    • Ruby 2.2.4ಋೖࡁΈ
    19
    

    View Slide

20. ϓϩϏδϣχϯά
    20
    

    View Slide

21. 21
    DC1 DC2
    Deploy Server
    S3 Bucket
    GitHub
    ssh
    Chef Cookbook
    Ubuntu Package Mirror
    Internet
    deb Package
    deb Package
    

    View Slide

22. ϓϩϏδϣχϯά
    • ChefʹΑΔϓϩϏδϣχϯά͕جຊ
    • capistrano-paratrooper-chef

    https://github.com/tk0miya/capistrano-paratrooper-chef
    • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏
    • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ
    22
    

    View Slide

23. • Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ
    • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ
    • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ
    • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ
    23
    

    View Slide

24. • S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ
    • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ
    • aptlyͰߏங http://www.aptly.info/
    • ݸਓͷ։ൃϚγϯͰ΋Vagrant + VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ
    • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen
    24
    

    View Slide

25. ෛՙରࡦ
    25
    

    View Slide

26. ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘…
    26
    

    View Slide

27. ෛՙରࡦ
    • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ
    • Applicationαʔόͷ૿ઃ

    12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40 coreʣͷϚγϯ͕جຊߏ੒
    • ΫΤϦվળ΍ΩϟογϡରԠ

    ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ
    • Memcachedͷ૿ઃ
    27
    

    View Slide

28. Memcachedͷߏ੒
    28
    

    View Slide

29. Memcachedͷߏ੒
    • Memcached͸2ϓʔϧߏ੒
    • 1ϓʔϧ͋ͨΓ໿100୆
    • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ
    • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ
    • ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ

    DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores
    • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ
    29
    

    View Slide

30. 2ϓʔϧߏ੒ͷMemcached

    ೖΕସ͑ํ๏
    30
    

    View Slide

31. 31
    App
    M
    App App
    M
    M M
    M M
    M M
    M M
    M M
    set get set
    Pool 1 Pool 2
    App App App
    M M
    M M
    M M
    set get
    Pool 1 Pool 2
    ௨ৗ࣌ ఀࢭɾ૿ઃ
    M M
    શΫϦΞ͓ͯ͘͠
    M M
    M M
    M M
    Pool2΁ͷsetΛఀࢭ
    M Memcachedϗετ
    

    View Slide

32. 32
    App
    M
    App App
    M
    M M
    M M
    set get
    set
    Pool 1 Pool 2
    ੾Γସ͑
    M M
    M M
    M M
    M M
    App App App
    set get
    Pool 1 Pool 2
    Ωϟογϡ͋ͨͨΊ
    set
    M M
    M M
    M M
    M M
    M M
    M M
    M M
    ͠͹Β͋ͨͨ͘ΊΔ
    

    View Slide

33. ϩʔυόϥϯαʔ
    33
    

    View Slide

34. ϩʔυόϥϯαʔ
    • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻
    • A10 Networks AX2500
    • RESTful APIʹΑΔૢ࡞͕Մೳ
    • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ
    • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ
    34
    

    View Slide

35. 35
    nginx
    unicorn
    nginx
    unicorn
    nginx
    unicorn
    nginx
    unicorn
    Memcached΍MariaDBͳͲ
    Internet
    A10 ϩʔυόϥϯαʔ
    Applicationαʔό
    

    View Slide

36. ϞχλϦϯάɾ؂ࢹ
    36
    

    View Slide

37. 37
    CloudForecast
    • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ
    • ໿1,000୆ͷαʔό͕ܭଌର৅
    • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ
    https://github.com/kazeburo/cloudforecast
    

    View Slide

38. 38
    GrowthForecast
    • DCϥοΫͷిྗ஋ͷάϥϑԽ
    • Fluentd Monitoring AgentͷάϥϑԽ
    http://kazeburo.github.io/GrowthForecast/
    

    View Slide

39. 39
    • ͢΂ͯͷαʔόͷ؂ࢹ
    • 12,000Λ௒͑Δ؂ࢹ߲໨
    • ಠࣗͷϓϥάΠϯΛ௥Ճ
    • PagerDutyͱͷ࿈ܞ
    • ౰൪ͷεέδϡʔϦϯά
    • ΤεΧϨʔγϣϯϙϦγʔ
    • Slack΁ͷ௨஌
    https://www.nagios.org/
    

    View Slide

40. 40
    • nginx, ApplicationͷϩάΛ

    FluentdΛ࢖ͬͯElasticsearch΁సૹ
    • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ
    • Τϥʔස౓ͷܭଌ
    • ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ
    • Slow QueryͷՄࢹԽ
    • CloudTrailϩάͷՄࢹԽ
    https://www.elastic.co/
    

    View Slide

41. 41
    nginx
    unicorn
    Amazon S3
    GrowthForecast
    LTSV Log
    Application Servers
    ϩάసૹϑϩʔ
    Monitoring Agent
    Response Time
    Status Code
    Monitoring Agent
    Log
    Log
    td-agent 2.3.x
    Max 500Mbps Max 500Mbps
    

    View Slide

42. ೔ৗͷπʔϧ
    42
    

    View Slide

43. 43
    • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ
    • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ
    • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ
    • ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ

    http://gihyo.jp/magazine/SD/archive/2016/201601
    • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ

    ͢΂ͯGitHub্Ͱ؅ཧ
    • pull requestsϕʔεͷ։ൃɺϨϏϡʔ
    • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ
    • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ
    • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ
    • Hubotͱͷ࿈ܞ
    

    View Slide

44. ͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ
    44
    

    View Slide

45. • ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ
    • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍
    • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ
    • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ
    • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ
    • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ
    • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ
    45
    

    View Slide

46. • MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ
    • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ
    • ؂ࢹɺϞχλϦϯά؀ڥͷվળ
    • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ
    • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋
    • Consulͱͷ࿈ܞ
    46
    

    View Slide

47. • ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ
    • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠
    • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ
    • Πϯϑϥςετͷޮ཰Խ
    • Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠
    • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ
    47
    

    View Slide

48. ࠷ޙʹ
    48
    

    View Slide

49. ΤϯδχΞืूதʂ
    49
    http://xflag.com/recruit/career/engineer.html
    

    View Slide

50. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    50
    

    View Slide