モンストを支えるインフラの今とこれから

by Isao Shimizu

Slide 1

Slide 1 text

ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™ STUDIO @isaoshimizu

Slide 2

Slide 2 text

About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 2011೥8݄ΑΓגࣜձࣾϛΫγΟ • ݱࡏ͸ΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • ໿2೥લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β  εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻΁ 2

Slide 3

Slide 3 text

ϞϯελʔετϥΠΫ • 2013೥10݄10೔ ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰ୺຤Ͱॏෳμ΢ϯϩʔυ͞Εͨ਺͸ؚ·ͣʣ • ೔ຊɺ୆࿷ɺؖࠃɺ๺ถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ഑৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ਺100ສຊಥഁ 3

Slide 4

Slide 4 text

͔͜͜Β͸ϞϯελʔετϥΠΫ ೔ຊ൛ʹݶఆͨ͠಺༰ʹͳΓ·͢ 4

Slide 5

Slide 5 text

਺஋ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5

Slide 6

Slide 6 text

6 Application 300+ Servers Resque 50 Servers Redis 30 Servers Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps

Slide 7

Slide 7 text

Πϯϑϥߏ੒ 7

Slide 8

Slide 8 text

8 DC1 DC2 Main Main/Backup Main/Dev

Slide 9

Slide 9 text

ෳ਺DCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃ؀ڥʣͰߏ੒ • ֤DC͸TY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτ઀ଓ • ֤DCؒ͸ยܥো֐࣌ʹ͓͍ͯ΋20Gbps·Ͱ଱͑ΒΕΔઃܭ • ৄ͍͠ղઆ  “ڌ఺৑௕ͨ͠࿩ʢ಺෦τϥϑΟοΫϘϦϡʔϜฤʣ”  http://xﬂag.com/blog/infradb/internal_trafﬁc_volume.html 9

Slide 10

Slide 10 text

֤ڌ఺ͷ໾ׂ 10

Slide 11

Slide 11 text

11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque

Slide 12

Slide 12 text

֤ڌ఺ͷ໾ׂ • ຊ൪ͷApplicationɺResque͸DC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup • Backup͸DC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰ͸ຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12

Slide 13

Slide 13 text

ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13

Slide 14

Slide 14 text

14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai) API API LB LB Internet Client

Slide 15

Slide 15 text

ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏ੒ʣ΁ • ϩʔυόϥϯαʔ΁ͷΞΫηε͸DNSϥ΢ϯυϩϏϯʢ2ͭͷԾ૝IPʣ • ϩʔυόϥϯαʔ͔Β֤ڌ఺ͷApplicationαʔό΁ৼΓ෼͚ • AWSͰ͸ϚϧνϓϨΠ༻ʹTURN ServerʢEC2 + Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamai΋ซ༻ɻ 15

Slide 16

Slide 16 text

σϓϩΠ 16

Slide 17

Slide 17 text

17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT GitHub ssh Code Internet

Slide 18

Slide 18 text

σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli • GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞੒ • ApplicationαʔόͰ͸S3ʹΞοϓϩʔυ͞ΕͨtarballΛμ΢ϯϩʔυɺల։ • GitHub΁ͷ઀ଓ਺੍ݶରࡦʢgithub.com΁େྔʹ઀ଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18

Slide 19

Slide 19 text

• capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷ഑෍ɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19

Slide 20

Slide 20 text

ϓϩϏδϣχϯά 20

Slide 21

Slide 21 text

21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef Cookbook Ubuntu Package Mirror Internet deb Package deb Package

Slide 22

Slide 22 text

ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef  https://github.com/tk0miya/capistrano-paratrooper-chef • ෺ཧϚγϯͷॳظηοτΞοϓʢ࠷௿ݶʣ͸AnsibleΛ࢖͏ • OSΫϦʔϯΠϯετʔϧ௚ޙͷঢ়ଶͰ͸ɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ 22

Slide 23

Slide 23 text

• Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞੒ɺར༻͍ͯ͠Δ • ಠࣗCookbook͸ɺXFLAGશମͷΠϯϑϥͰ΋ڞ௨ͯ͠࢖͑ΔΑ͏ʹ • ͢΂ͯͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻ͯ͠΋໰୊͕ى͖ͳ͍Α͏ʹ 23

Slide 24

Slide 24 text

• S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷো֐ରࡦʣ • ioDriveͳͲͷυϥΠό΋ஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰ΋Vagrant + VirtualBox + ChefͰߏ੒Ͱ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24

Slide 25

Slide 25 text

ෛՙରࡦ 25

Slide 26

Slide 26 text

ࠓ೥ʹೖ͔ͬͯΒ΋ߴෛՙ͕ଓ͘… 26

Slide 27

Slide 27 text

ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧ෼ׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ  12-20 core ͳCPU x 2ʢHyper-Threading׵ࢉͰ24-40 coreʣͷϚγϯ͕جຊߏ੒ • ΫΤϦվળ΍ΩϟογϡରԠ  ioDrive(ioMemory)͔ΒSSD΁ͷεέʔϧμ΢ϯͷ࣮ݱ • Memcachedͷ૿ઃ 27

Slide 28

Slide 28 text

Memcachedͷߏ੒ 28

Slide 29

Slide 29 text

Memcachedͷߏ੒ • Memcached͸2ϓʔϧߏ੒ • 1ϓʔϧ͋ͨΓ໿100୆ • Ϛγϯ1୆͋ͨΓͷϝϞϦׂΓ౰ͯ26GBʢϝϞϦ32GB౥ࡌͷϚγϯʣ • 1ϓʔϧશମͰ໿2.6TBͷ༰ྔΛ΋ͭ • ྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ  DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠Ͱϓʔϧͷ੾Γସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29

Slide 30

Slide 30 text

2ϓʔϧߏ੒ͷMemcached  ೖΕସ͑ํ๏ 30

Slide 31

Slide 31 text

31 App M App App M M M M M M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2΁ͷsetΛఀࢭ M Memcachedϗετ

Slide 32

Slide 32 text

32 App M App App M M M M M set get set Pool 1 Pool 2 ੾Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠͹Β͋ͨͨ͘ΊΔ

Slide 33

Slide 33 text

ϩʔυόϥϯαʔ 33

Slide 34

Slide 34 text

ϩʔυόϥϯαʔ • Applicationαʔόͷલஈ͸ΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ • ࣗࣾ։ൃͷGo੡ͷCLIπʔϧʢa10-cliʣ • Applicationαʔόͷ௥Ճɺ࡟আ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞ΢τ͕Ͱ͖Δ 34

Slide 35

Slide 35 text

35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn Memcached΍MariaDBͳͲ Internet A10 ϩʔυόϥϯαʔ Applicationαʔό

Slide 36

Slide 36 text

ϞχλϦϯάɾ؂ࢹ 36

Slide 37

Slide 37 text

37 CloudForecast • ͢΂ͯͷαʔόͷϝτϦΫεऔಘ • ໿1,000୆ͷαʔό͕ܭଌର৅ • ༷ʑͳಠࣗͷϝτϦΫεΛ௥Ճ https://github.com/kazeburo/cloudforecast

Slide 38

Slide 38 text

38 GrowthForecast • DCϥοΫͷిྗ஋ͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/

Slide 39

Slide 39 text

39 • ͢΂ͯͷαʔόͷ؂ࢹ • 12,000Λ௒͑Δ؂ࢹ߲໨ • ಠࣗͷϓϥάΠϯΛ௥Ճ • PagerDutyͱͷ࿈ܞ • ౰൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slack΁ͷ௨஌ https://www.nagios.org/

Slide 40

Slide 40 text

40 • nginx, ApplicationͷϩάΛ  FluentdΛ࢖ͬͯElasticsearch΁సૹ • APIຖͷॲཧ࣌ؒ΍ϨεϙϯελΠϜͷܭଌ • Τϥʔස౓ͷܭଌ • ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/

Slide 41

Slide 41 text

41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps

Slide 42

Slide 42 text

೔ৗͷπʔϧ 42

Slide 43

Slide 43 text

43 • ίϛϡχέʔγϣϯ͸SlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃ؀ڥ΁ͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ • ৄ͘͠͸Software Design 2016೥1݄߸ͷChatOpsಛूʹ  http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυ΋Πϯϑϥߏங༻ͷCookbookͳͲ  ͢΂ͯGitHub্Ͱ؅ཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈ͸Slack΁௨஌͞ΕΔ • GitHubͰpull requests͕࡞੒͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜ੒ɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ

Slide 44

Slide 44 text

͜Ε͔Β΍͍͖͍ͬͯͨ͜ͱ 44

Slide 45

Slide 45 text

• ίʔυͷվળɺεέʔϧΞ΢τɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓ͸Ͱ͖Δ͚ͩ΍Γͨ͘ͳ͍ • ݁Ռతʹεέʔϧμ΢ϯʹͭͳ͕Δͱϕετ • ͜ΕΒ͸ैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτ΢ΣΞͷܧଓతΞοϓσʔτ • Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣ΁ͷҠߦ • Kernelɺϛυϧ΢ΣΞͷΞοϓσʔτ͸ੵۃతʹ࣮ࢪ 45

Slide 46

Slide 46 text

• MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋Ε͹஌Γ͍ͨ • ؂ࢹɺϞχλϦϯά؀ڥͷվળ • ෛՙ΍ҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ΋͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ΋ • Consulͱͷ࿈ܞ 46

Slide 47

Slide 47 text

• ࣾ಺ͷ৽نҊ݅Ͱ͸Hashicorpࣾ੡πʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢Δͷ͸ίετ΍ϦεΫ͕ߴ͍ͷͰಋೖ༧ఆ͸ͳ͠ • ෦෼తʹπʔϧར༻ʹΑΔޮ཰Խ͸͓͜ͳ͍͖͍ͬͯͨ • Πϯϑϥςετͷޮ཰Խ • Docker΍ServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺ΋ͬͱγϯϓϧʹ 47