大規模になったサービスでやるべき基本的なこと

by Isao Shimizu

Slide 1

Slide 1 text

େن໛ʹͳͬͨαʔϏεͰ΍Δ΂͖جຊతͳ͜ͱ ήʔϜ։ൃ෦ SREάϧʔϓ ਗ਼ਫ ܄ @isaoshimizu 2017.2.7 ʲDMM GAMESओ࠵ʂʳʮෳࡶɾେن໛webαʔϏεΛࢧ͑Δٕज़ษڧձʯ XFLAG STUDIO

Slide 2

Slide 2 text

About me • ਗ਼ਫ ܄ @isaoshimizu • 2011.8-2014.3 SNS mixiͷӡ༻ • XFLAG ελδΦ • 2014.4-2016.6 αʔόʔΤϯδχΞ • 2016.7- SRE ओʹࠃ಺޲͚ϞϯελʔετϥΠΫΛࢧ͑Δ͓࢓ࣄ • ଞʹ΋ϞϯετελδΞϜɺϒϥφΠDASHͳͲ • ޷͖ͳ΋ͷ • LinuxɺMySQLɺnginxɺMemcachedͳͲͷϛυϧ΢ΣΞɺGoɺΫϥϑτϏʔϧ 2

Slide 3

Slide 3 text

௚ۙͷൃදࢿྉ • 2016.3.1 ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β  https://speakerdeck.com/isaoshimizu/monsutowozhi-eruinhurafalsejin- tokorekara • 2017.1.30 SREάϧʔϓ͕Ͱ͖ͯ͜ͷ൒೥ؒ΍͖ͬͯͨ͜ͱ  https://speakerdeck.com/isaoshimizu/sregurupugadekitekofalseban-nian- jian-yatutekitakoto 3

Slide 4

Slide 4 text

αʔϏεͷن໛͕େ͖͘ͳΔͱ͍͏͜ͱ

Slide 5

Slide 5 text

αʔϏεͷن໛͕େ͖͘ͳΔͱ͍͏͜ͱ • ΞΫηε͕૿͑Δ • ෛՙͱͷઓ͍͕࢝·Δ • αʔό͕૿͑Δ • ؔΘΔਓ΋૿͑Δ 5

Slide 6

Slide 6 text

ෛՙΛՄࢹԽ͢Δ

Slide 7

Slide 7 text

؂ࢹɾϞχλϦϯά • ʮͳΜͱͳ͘ෛՙߴͦ͏ʯͱ͍͏ঢ়ଶ͸ͱͯ΋ةݥ • ศརͳπʔϧ͕૿͖͍͑ͯͯΔͷͰͪΌΜͱ׆༻͢Δ • Nagios, CloudForecast, Cacti, Ganglia, Munin, Sensu, Zabbix, Kurado, Prometheus • Mackerel, Datadog, CloudWatch, NewRelic, PagerDuty, Pingdom • Fluentd+Elasticsearch+KibanaͰϩά͔ΒάϥϑΛੜ੒͢Δͷ΋ޮՌత • άϥϑ͸ຖ೔֬ೝ͢ΔɻཧղͰ͖ͳ͍άϥϑ͸ੜ੒ɾදࣔͯ͠΋ҙຯ͕ͳ͍ɻ • ຖ݄ෛՙঢ়گΛαϚϥΠζ͓ͯ͘͠ͱৼΓฦΓ΍ࠓޙͷରࡦํ਑͕࡞Γ΍͍͢ • ݟ͑ͳ͍΋ͷɺଌΕͳ͍΋ͷ͸ϦεΫ • αʔϏε࿈ಈͷاը΍Πϕϯτ͸ૣΊʹڞ༗ͯ͠උ͑Δ 7

Slide 8

Slide 8 text

εέʔϧΞοϓ͔εέʔϧΞ΢τ͢΂͖͔

Slide 9

Slide 9 text

εέʔϧΞοϓ or εέʔϧΞ΢τ • εέʔϧΞοϓ • ੑೳΛ্͛ͯෛՙରࡦ • εέʔϧͰ͖Δ্ݶ͕ܾ·ͬͯ͠·͏ɻ͞ΒͳΔෛՙ૿΁ͷରԠ͸ݫ͍͠ɻ • PCI-ExpressͳioMemory΍NVMe SSDΛ࢖ͬͯ΋μϝͳ࣌͸μϝɻ • εέʔϧΞ΢τ • LB഑ԼͷΞϓϦέʔγϣϯαʔό͸૿΍͢͜ͱͰCPUϦιʔεΛෛՙ෼ࢄ • DB΁ͷ઀ଓͱΞΫηε͕૿͑ΔͷͰཁ஫ҙ • DBʹ͓͍ͯ͸ɺεέʔϧΞ΢τͰ͖Δͱɺத௕ظతͳෛՙରࡦͱͯ͠༗ޮ • Ұ࣌తͳෛՙ૿΁ͷରࡦͰ͋Ε͹εέʔϧΞοϓ΋ޮՌత 9

Slide 10

Slide 10 text

τϥϑΟοΫͷ૿Ճ

Slide 11

Slide 11 text

τϥϑΟοΫͷ૿Ճ • Ϋϥ΢υͷ৔߹ • ΠϯελϯελΠϓʹΑ্ͬͯݶ͕ҟͳΔʢϕϯνϚʔΫͱ͓ͬͯ͘ͱ҆৺ʣ • ࣗࣾDCΛ࢖ͬͯΔ৔߹ • NICͱεΠονͷଳҬ্ݶʢαʔόʔ͕ͲͷϥοΫʹ͋Δ͔Λҙࣝʣ • ճઢͷ৑௕ੑʢํܥ͕མͪͯ΋ڐ༰Ͱ͖ΔଳҬઃܭʣ • NICͷ৑௕ੑ • Bonding: ίωΫγϣϯ୯ҐͰ෼ࢄ͞ΕΔͷͰNICx2Ͱ2ഒͷଳҬͱ͸ݶΒͳ͍ • TCPͷ࠶ૹճ਺ͷࢹ֮Խ • ಛʹΫϥ΢υͰ͸ϨΠςϯγ΍ଳҬͷมԽ͕͋ΔͷͰཁ஫ҙ 11

Slide 12

Slide 12 text

αʔόʔ/Πϯελϯε਺ͷ૿Ճ

Slide 13

Slide 13 text

αʔόʔ/Πϯελϯε਺ͷ૿Ճ • σϓϩΠɺϓϩϏδϣχϯάͷޮ཰ԽɺࣗಈԽ • Capistrano, AWS CodeDeploy, Stretcher + Consul • Chef, Ansible, Puppet, Itamae • ϕʔεͱͳΔΠϝʔδʢAMIͳͲʣߏஙͷࣗಈԽ • Packerศར • ୆਺ʹґଘ͠ͳ͍ΦϖϨʔγϣϯ͕ཧ૝ܥ 13

Slide 14

Slide 14 text

σʔλϕʔε ΫΤϦ਺ͷ૿Ճ

Slide 15

Slide 15 text

σʔλϕʔε ΫΤϦ਺ͷ૿Ճ • ϘτϧωοΫ͸READ͔WRITE͔ • σʔλϕʔεɺςʔϒϧͷ୯ҐͰαʔόΛ෼ׂ • γϟʔσΟϯάʢID΍ϋογϡ஋ͷ৒༨Ͱ෼ׂʣ • ΩϟογϡʢMemcached, RedisͳͲʣͷ׆༻ • ετϨʔδͷIOݶքͷݟۃΊʢΧλϩάεϖοΫΛ৴༻ͤͣܭଌ ͢Δʣ • tmp table, file sortͳͲ͕ൃੜ͍ͯ͠ͳ͍͔ • Dirty Page΍waitͷൃੜස౓Λάϥϑ͔Β஌Δ • όοϑΝϓʔϧར༻཰ͷঢ়گ 15

Slide 16

Slide 16 text

σʔλϕʔε αΠζͷ૿Ճ

Slide 17

Slide 17 text

σʔλϕʔε αΠζͷ૿Ճ • Ұ౓ᷓΕͯ͠·͏ͱରॲ͕ࠔ೉ • ςʔϒϧͷσʔλɺΠϯσοΫεαΠζ • όοϑΝʔϓʔϧͷΩϟύγςΟ • ϝϞϦΛ૿΍ͯ͠ϓʔϧαΠζΛ޿͛Δͱ͍͏ख • όΠφϦϩάͷ૿Ճɻexpire_logs_daysͷௐ੔ʢMySQLͷ৔߹ʣɻ • AUTO INCREMENTͷ্ݶ஋ʹ஫ҙʢint͸໿21ԯɺbigint͸໿922ژʣ • intͩͱςʔϒϧʹΑͬͯ͸͋ͬͱ͍͏ؒʹୡͯ͠͠·͏ͷͰ஫ҙ • ᮢ஋Λઃఆͯ͠؂ࢹ͓ͯ͘͠ͱ҆৺ 17

Slide 18

Slide 18 text

ਓһͷ૿Ճ

Slide 19

Slide 19 text

ਓһͷ૿Ճ • ΞΧ΢ϯτ • Ϋϥ΢υͷΞΧ΢ϯτɺLinuxϢʔβʔΞΧ΢ϯτɺGitHubͷΞΧ΢ϯτ • AWSͷ৔߹ɺCloudTrailΛ࢖ͬͨ؂ࠪϩά • LDAPͳͲͰϩάΠϯϢʔβʔɺϗετͷ੍ݶʢηΩϡϦςΟͷ୲อʣ • ϓϩδΣΫτ΁৽ͨʹδϣΠϯˍൈ͚Δϝϯόʔ΁ͷରԠʢνΣοΫϦετͷςϯϓϨʔτʣ • Ξϥʔτ • ౰൪੍ɺΤεΧϨʔγϣϯํ๏ͳͲͷཱ֬ɺPagerDutyͷ׆༻ • ϊ΢ϋ΢΍खॱͷڞ༗ • Wikiͷ׆༻ʢྫ: GitHub Wiki, Qiita Team, Crowiʣ • ߏங΍ো֐ൃੜ࣌ͷखॱɺϧʔϧΛυΩϡϝϯτԽʢΠϯγσϯτൃੜ࣌ʹྫྷ੩ʹରॲͰ͖ΔΑ͏ʹʣ • 4 eyesͰͷ࡞ۀ֬ೝͯ͠ࣄނ๷ࢭʢྫ: DNSߋ৽ɺTerminateɺService StopͳͲʣ • ChatOpsͰ࡞ۀͰ͖Δ͜ͱΛ૿΍͢ 19

Slide 20

Slide 20 text

·ͱΊ

Slide 21

Slide 21 text

·ͱΊ • ن໛ʹ͔͔ΘΒͣجຊΛ͔ͬ͠Γͱ • ى͖͍ͯΔ͜ͱΛਖ਼͘͠೺Ѳ͢Δ͜ͱɺΠϝʔδͰ͖Δ͜ͱ • ن໛ʹൺྫͯ͠࡞ۀྔ͕૿͑ͳ͍Α͏ʹ͢Δ͜ͱ • ਓ਺͕૿͑ͨͱ͖ʹ࡞ۀෛՙ͕෼ࢄ͞ΕΔΑ͏ʹϊ΢ϋ΢΍खॱΛల։͢Δ͜ͱ • େن໛͡Όͳͯ͘΋΍ͬͨ΄͏͕͍͍͜ͱ͸ଟ͍ • ஥ؒืूத https://xflag.com/recruit/ 21

Slide 22

Slide 22 text

Thank you!