モンストを支えるインフラの今とこれから

Transcript

1. 1.

   ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™

   STUDIO @isaoshimizu
2. 2.

   About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ

   • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β
 εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁ 2
3. 3.

   ϞϯελʔετϥΠΫ • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ •

   ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ 3
4. 4.

   ͔͜͜Β͸ϞϯελʔετϥΠΫ ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢ 4

5. 5.

   ਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5

6. 6.

   6 Application 300+ Servers Resque 50 Servers Redis 30 Servers

   Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps
7. 7.

   Πϯϑϥߏ੒ 7

8. 8.

   8 DC1 DC2 Main Main/Backup Main/Dev

9. 9.

   ෳ਺DCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒ • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ •

   ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ • ৄ͍͠ղઆ
 “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”
 http://xflag.com/blog/infradb/internal_traffic_volume.html 9
10. 10.

    ֤ڌ఺ͷ໾ׂ 10

11. 11.

    11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque

    Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque
12. 12.

    ֤ڌ఺ͷ໾ׂ • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup •

    Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12
13. 13.

    ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13

14. 14.

    14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai)

    API API LB LB Internet Client
15. 15.

    ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁ • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚ • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2

    + Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ 15
16. 16.

    σϓϩΠ 16

17. 17.

    17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT

    GitHub ssh Code Internet
18. 18.

    σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli •

    GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒ • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։ • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18
19. 19.

    • capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19

20. 20.

    ϓϩϏδϣχϯά 20

21. 21.

    21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef

    Cookbook Ubuntu Package Mirror Internet deb Package deb Package
22. 22.

    ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef
 https://github.com/tk0miya/capistrano-paratrooper-chef • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏ • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ

    22
23. 23.

    • Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ 23

24. 24.

    • S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰ΋Vagrant +

    VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24
25. 25.

    ෛՙରࡦ 25

26. 26.

    ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘… 26

27. 27.

    ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ
 12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40

    coreʣͷϚγϯ͕جຊߏ੒ • ΫΤϦվળ΍ΩϟογϡରԠ
 ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ • Memcachedͷ૿ઃ 27
28. 28.

    Memcachedͷߏ੒ 28

29. 29.

    Memcachedͷߏ੒ • Memcached͸2ϓʔϧߏ੒ • 1ϓʔϧ͋ͨΓ໿100୆ • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ •

    ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ
 DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29
30. 30.

    2ϓʔϧߏ੒ͷMemcached
 ೖΕସ͑ํ๏ 30

31. 31.

    31 App M App App M M M M M

    M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2΁ͷsetΛఀࢭ M Memcachedϗετ
32. 32.

    32 App M App App M M M M M

    set get set Pool 1 Pool 2 ੾Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠͹Β͋ͨͨ͘ΊΔ
33. 33.

    ϩʔυόϥϯαʔ 33

34. 34.

    ϩʔυόϥϯαʔ • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ

    • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ 34
35. 35.

    35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn Memcached΍MariaDBͳͲ

    Internet A10 ϩʔυόϥϯαʔ Applicationαʔό
36. 36.

    ϞχλϦϯάɾ؂ࢹ 36

37. 37.

    37 CloudForecast • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ • ໿1,000୆ͷαʔό͕ܭଌର৅ • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ https://github.com/kazeburo/cloudforecast

38. 38.

    38 GrowthForecast • DCϥοΫͷిྗ஋ͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/

39. 39.

    39 • ͢΂ͯͷαʔόͷ؂ࢹ • 12,000Λ௒͑Δ؂ࢹ߲໨ • ಠࣗͷϓϥάΠϯΛ௥Ճ • PagerDutyͱͷ࿈ܞ •

    ౰൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slack΁ͷ௨஌ https://www.nagios.org/
40. 40.

    40 • nginx, ApplicationͷϩάΛ
 FluentdΛ࢖ͬͯElasticsearch΁సૹ • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ • Τϥʔස౓ͷܭଌ •

    ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/
41. 41.

    41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers

    ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps
42. 42.

    ೔ৗͷπʔϧ 42

43. 43.

    43 • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ •

    ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ
 http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ
 ͢΂ͯGitHub্Ͱ؅ཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ
44. 44.

    ͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ 44

45. 45.

    • ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍ • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ

    • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ 45
46. 46.

    • MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ • ؂ࢹɺϞχλϦϯά؀ڥͷվળ

    • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋ • Consulͱͷ࿈ܞ 46
47. 47.

    • ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠ • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ • Πϯϑϥςετͷޮ཰Խ •

    Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ 47
48. 48.

    ࠷ޙʹ 48

49. 49.

    ΤϯδχΞืूதʂ 49 http://xflag.com/recruit/career/engineer.html

50. 50.

    ͋Γ͕ͱ͏͍͟͝·ͨ͠ 50