モンストを支えるインフラの今とこれから

dots. Conference Spring 2016
ゲーム開発の裏側
http://eventdots.jp/event/580344

Transcript

1. ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™ STUDIO @isaoshimizu
2. About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β
 εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁ 2
3. ϞϯελʔετϥΠΫ • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ • ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ 3
4. ͔͜͜Β͸ϞϯελʔετϥΠΫ ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢ 4
5. ਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5
6. 6 Application 300+ Servers Resque 50 Servers Redis 30 Servers Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps
7. Πϯϑϥߏ੒ 7
8. 8 DC1 DC2 Main Main/Backup Main/Dev
9. ෳ਺DCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒ • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ • ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ • ৄ͍͠ղઆ
 “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”
 http://xflag.com/blog/infradb/internal_traffic_volume.html 9
10. ֤ڌ఺ͷ໾ׂ 10
11. 11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque
12. ֤ڌ఺ͷ໾ׂ • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup • Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12
13. ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13
14. 14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai) API API LB LB Internet Client
15. ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁ • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚ • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2 + Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ 15
16. σϓϩΠ 16
17. 17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT GitHub ssh Code Internet
18. σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli • GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒ • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։ • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18
19. • capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19
20. ϓϩϏδϣχϯά 20
21. 21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef Cookbook Ubuntu Package Mirror Internet deb Package deb Package
22. ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef
 https://github.com/tk0miya/capistrano-paratrooper-chef • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏ • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ 22
23. • Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ 23
24. • S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰ΋Vagrant + VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24
25. ෛՙରࡦ 25
26. ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘… 26
27. ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ
 12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40 coreʣͷϚγϯ͕جຊߏ੒ • ΫΤϦվળ΍ΩϟογϡରԠ
 ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ • Memcachedͷ૿ઃ 27
28. Memcachedͷߏ੒ 28
29. Memcachedͷߏ੒ • Memcached͸2ϓʔϧߏ੒ • 1ϓʔϧ͋ͨΓ໿100୆ • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ • ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ
 DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29
30. 2ϓʔϧߏ੒ͷMemcached
 ೖΕସ͑ํ๏ 30
31. 31 App M App App M M M M M M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2΁ͷsetΛఀࢭ M Memcachedϗετ
32. 32 App M App App M M M M M set get set Pool 1 Pool 2 ੾Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠͹Β͋ͨͨ͘ΊΔ
33. ϩʔυόϥϯαʔ 33
34. ϩʔυόϥϯαʔ • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ 34
35. 35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn Memcached΍MariaDBͳͲ Internet A10 ϩʔυόϥϯαʔ Applicationαʔό
36. ϞχλϦϯάɾ؂ࢹ 36
37. 37 CloudForecast • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ • ໿1,000୆ͷαʔό͕ܭଌର৅ • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ https://github.com/kazeburo/cloudforecast
38. 38 GrowthForecast • DCϥοΫͷిྗ஋ͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/
39. 39 • ͢΂ͯͷαʔόͷ؂ࢹ • 12,000Λ௒͑Δ؂ࢹ߲໨ • ಠࣗͷϓϥάΠϯΛ௥Ճ • PagerDutyͱͷ࿈ܞ • ౰൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slack΁ͷ௨஌ https://www.nagios.org/
40. 40 • nginx, ApplicationͷϩάΛ
 FluentdΛ࢖ͬͯElasticsearch΁సૹ • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ • Τϥʔස౓ͷܭଌ • ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/
41. 41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps
42. ೔ৗͷπʔϧ 42
43. 43 • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ • ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ
 http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ
 ͢΂ͯGitHub্Ͱ؅ཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ
44. ͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ 44
45. • ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍ • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ 45
46. • MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ • ؂ࢹɺϞχλϦϯά؀ڥͷվળ • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋ • Consulͱͷ࿈ܞ 46
47. • ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠ • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ • Πϯϑϥςετͷޮ཰Խ • Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ 47
48. ࠷ޙʹ 48
49. ΤϯδχΞืूதʂ 49 http://xflag.com/recruit/career/engineer.html
50. ͋Γ͕ͱ͏͍͟͝·ͨ͠ 50