$30 off During Our Annual Pro Sale. View Details »

モンストを支えるインフラの今とこれから

 モンストを支えるインフラの今とこれから

dots. Conference Spring 2016
ゲーム開発の裏側
http://eventdots.jp/event/580344

Isao Shimizu

March 01, 2016
Tweet

More Decks by Isao Shimizu

Other Decks in Technology

Transcript

  1. ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β
    2016.3.1 dots. Conference Spring 2016
    ήʔϜ։ൃͷཪଆ
    mixi, Inc. XFLAG™ STUDIO
    @isaoshimizu

    View Slide

  2. About Me
    • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu
    • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ
    • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ
    • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β

    εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁
    2

    View Slide

  3. ϞϯελʔετϥΠΫ
    • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε
    • iOS൛, Android൛Λఏڙ
    • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ
    • ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։
    • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ
    3

    View Slide

  4. ͔͜͜Β͸ϞϯελʔετϥΠΫ
    ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢
    4

    View Slide

  5. ਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ
    5

    View Slide

  6. 6
    Application
    300+ Servers
    Resque
    50 Servers
    Redis
    30 Servers
    Memcached
    200+ Servers
    CDN Traffic
    Max 270Gbps
    API Traffic
    Avg 1Gbps
    TURN
    40 Servers
    MariaDB
    250+ Servers
    Internal Traffic
    Max 20Gbps

    View Slide

  7. Πϯϑϥߏ੒
    7

    View Slide

  8. 8
    DC1 DC2
    Main Main/Backup Main/Dev

    View Slide

  9. ෳ਺DCͱAmazon VPC
    • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒
    • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ
    • ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ
    • ৄ͍͠ղઆ

    “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”

    http://xflag.com/blog/infradb/internal_traffic_volume.html
    9

    View Slide

  10. ֤ڌ఺ͷ໾ׂ
    10

    View Slide

  11. 11
    DC1 DC2
    Main Main/Backup Main/Dev
    Application
    MariaDB
    Redis
    Resque
    Memcached
    MariaDB
    Redis
    Memcached
    TURN Server
    ։ൃ༻Πϯελϯε܈
    Application
    Resque

    View Slide

  12. ֤ڌ఺ͷ໾ׂ
    • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ
    • ຊ൪ͷRedisɺMariaDBɺMemcached
    • DC1ʹMaster/Backupͷηοτ
    • DC2ʹBackup
    • Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ
    • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ
    12

    View Slide

  13. ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ
    13

    View Slide

  14. 14
    DC1 DC2 AWS
    TURN Server (EC2)
    CDN (CloudFront, Akamai)
    API API
    LB LB
    Internet
    Client

    View Slide

  15. ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ
    • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁
    • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ
    • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚
    • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2 + Elastic IPʣ
    • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ
    15

    View Slide

  16. σϓϩΠ
    16

    View Slide

  17. 17
    DC1 DC2
    Deploy Server
    S3 Bucket
    tarball
    S3 PUT
    GitHub
    ssh
    Code
    Internet

    View Slide

  18. σϓϩΠ
    • Capistrano 2

    • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ

    • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli

    • GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒

    • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։

    • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ
    18

    View Slide

  19. • capistrano-rbenv https://github.com/yyuu/capistrano-rbenv

    • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ

    • Ruby 2.2.4ಋೖࡁΈ
    19

    View Slide

  20. ϓϩϏδϣχϯά
    20

    View Slide

  21. 21
    DC1 DC2
    Deploy Server
    S3 Bucket
    GitHub
    ssh
    Chef Cookbook
    Ubuntu Package Mirror
    Internet
    deb Package
    deb Package

    View Slide

  22. ϓϩϏδϣχϯά
    • ChefʹΑΔϓϩϏδϣχϯά͕جຊ
    • capistrano-paratrooper-chef

    https://github.com/tk0miya/capistrano-paratrooper-chef
    • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏
    • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ
    22

    View Slide

  23. • Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ
    • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ
    • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ
    • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ
    23

    View Slide

  24. • S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ
    • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ
    • aptlyͰߏங http://www.aptly.info/
    • ݸਓͷ։ൃϚγϯͰ΋Vagrant + VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ
    • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen
    24

    View Slide

  25. ෛՙରࡦ
    25

    View Slide

  26. ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘…
    26

    View Slide

  27. ෛՙରࡦ
    • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ
    • Applicationαʔόͷ૿ઃ

    12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40 coreʣͷϚγϯ͕جຊߏ੒
    • ΫΤϦվળ΍ΩϟογϡରԠ

    ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ
    • Memcachedͷ૿ઃ
    27

    View Slide

  28. Memcachedͷߏ੒
    28

    View Slide

  29. Memcachedͷߏ੒
    • Memcached͸2ϓʔϧߏ੒
    • 1ϓʔϧ͋ͨΓ໿100୆
    • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ
    • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ
    • ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ

    DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores
    • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ
    29

    View Slide

  30. 2ϓʔϧߏ੒ͷMemcached

    ೖΕସ͑ํ๏
    30

    View Slide

  31. 31
    App
    M
    App App
    M
    M M
    M M
    M M
    M M
    M M
    set get set
    Pool 1 Pool 2
    App App App
    M M
    M M
    M M
    set get
    Pool 1 Pool 2
    ௨ৗ࣌ ఀࢭɾ૿ઃ
    M M
    શΫϦΞ͓ͯ͘͠
    M M
    M M
    M M
    Pool2΁ͷsetΛఀࢭ
    M Memcachedϗετ

    View Slide

  32. 32
    App
    M
    App App
    M
    M M
    M M
    set get
    set
    Pool 1 Pool 2
    ੾Γସ͑
    M M
    M M
    M M
    M M
    App App App
    set get
    Pool 1 Pool 2
    Ωϟογϡ͋ͨͨΊ
    set
    M M
    M M
    M M
    M M
    M M
    M M
    M M
    ͠͹Β͋ͨͨ͘ΊΔ

    View Slide

  33. ϩʔυόϥϯαʔ
    33

    View Slide

  34. ϩʔυόϥϯαʔ
    • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻
    • A10 Networks AX2500
    • RESTful APIʹΑΔૢ࡞͕Մೳ
    • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ
    • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ
    34

    View Slide

  35. 35
    nginx
    unicorn
    nginx
    unicorn
    nginx
    unicorn
    nginx
    unicorn
    Memcached΍MariaDBͳͲ
    Internet
    A10 ϩʔυόϥϯαʔ
    Applicationαʔό

    View Slide

  36. ϞχλϦϯάɾ؂ࢹ
    36

    View Slide

  37. 37
    CloudForecast
    • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ
    • ໿1,000୆ͷαʔό͕ܭଌର৅
    • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ
    https://github.com/kazeburo/cloudforecast

    View Slide

  38. 38
    GrowthForecast
    • DCϥοΫͷిྗ஋ͷάϥϑԽ
    • Fluentd Monitoring AgentͷάϥϑԽ
    http://kazeburo.github.io/GrowthForecast/

    View Slide

  39. 39
    • ͢΂ͯͷαʔόͷ؂ࢹ
    • 12,000Λ௒͑Δ؂ࢹ߲໨
    • ಠࣗͷϓϥάΠϯΛ௥Ճ
    • PagerDutyͱͷ࿈ܞ
    • ౰൪ͷεέδϡʔϦϯά
    • ΤεΧϨʔγϣϯϙϦγʔ
    • Slack΁ͷ௨஌
    https://www.nagios.org/

    View Slide

  40. 40
    • nginx, ApplicationͷϩάΛ

    FluentdΛ࢖ͬͯElasticsearch΁సૹ
    • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ
    • Τϥʔස౓ͷܭଌ
    • ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ
    • Slow QueryͷՄࢹԽ
    • CloudTrailϩάͷՄࢹԽ
    https://www.elastic.co/

    View Slide

  41. 41
    nginx
    unicorn
    Amazon S3
    GrowthForecast
    LTSV Log
    Application Servers
    ϩάసૹϑϩʔ
    Monitoring Agent
    Response Time
    Status Code
    Monitoring Agent
    Log
    Log
    td-agent 2.3.x
    Max 500Mbps Max 500Mbps

    View Slide

  42. ೔ৗͷπʔϧ
    42

    View Slide

  43. 43
    • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ
    • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ
    • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ
    • ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ

    http://gihyo.jp/magazine/SD/archive/2016/201601
    • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ

    ͢΂ͯGitHub্Ͱ؅ཧ
    • pull requestsϕʔεͷ։ൃɺϨϏϡʔ
    • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ
    • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ
    • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ
    • Hubotͱͷ࿈ܞ

    View Slide

  44. ͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ
    44

    View Slide

  45. • ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ
    • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍
    • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ
    • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ
    • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ
    • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ
    • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ
    45

    View Slide

  46. • MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ
    • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ
    • ؂ࢹɺϞχλϦϯά؀ڥͷվળ
    • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ
    • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋
    • Consulͱͷ࿈ܞ
    46

    View Slide

  47. • ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ
    • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠
    • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ
    • Πϯϑϥςετͷޮ཰Խ
    • Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠
    • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ
    47

    View Slide

  48. ࠷ޙʹ
    48

    View Slide

  49. ΤϯδχΞืूதʂ
    49
    http://xflag.com/recruit/career/engineer.html

    View Slide

  50. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    50

    View Slide