Upgrade to Pro — share decks privately, control downloads, hide ads and more …

大規模になったサービスでやるべき基本的なこと

Isao Shimizu
February 07, 2017

 大規模になったサービスでやるべき基本的なこと

2017年2月7日
【DMM GAMES主催!】「複雑・大規模webサービスを支える技術勉強会」

Isao Shimizu

February 07, 2017
Tweet

More Decks by Isao Shimizu

Other Decks in Technology

Transcript

  1. େن໛ʹͳͬͨαʔϏεͰ΍Δ΂͖جຊతͳ͜ͱ
    ήʔϜ։ൃ෦ SREάϧʔϓ ਗ਼ਫ ܄ @isaoshimizu
    2017.2.7 ʲDMM GAMESओ࠵ʂʳʮෳࡶɾେن໛webαʔϏεΛࢧ͑Δٕज़ษڧձʯ
    XFLAG STUDIO

    View Slide

  2. About me
    • ਗ਼ਫ ܄ @isaoshimizu
    • 2011.8-2014.3 SNS mixiͷӡ༻
    • XFLAG ελδΦ
    • 2014.4-2016.6 αʔόʔΤϯδχΞ
    • 2016.7- SRE ओʹࠃ಺޲͚ϞϯελʔετϥΠΫΛࢧ͑Δ͓࢓ࣄ
    • ଞʹ΋ϞϯετελδΞϜɺϒϥφΠDASHͳͲ
    • ޷͖ͳ΋ͷ
    • LinuxɺMySQLɺnginxɺMemcachedͳͲͷϛυϧ΢ΣΞɺGoɺΫϥϑτϏʔϧ
    2

    View Slide

  3. ௚ۙͷൃදࢿྉ
    • 2016.3.1 ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β

    https://speakerdeck.com/isaoshimizu/monsutowozhi-eruinhurafalsejin-
    tokorekara
    • 2017.1.30 SREάϧʔϓ͕Ͱ͖ͯ͜ͷ൒೥ؒ΍͖ͬͯͨ͜ͱ

    https://speakerdeck.com/isaoshimizu/sregurupugadekitekofalseban-nian-
    jian-yatutekitakoto
    3

    View Slide

  4. αʔϏεͷن໛͕େ͖͘ͳΔͱ͍͏͜ͱ

    View Slide

  5. αʔϏεͷن໛͕େ͖͘ͳΔͱ͍͏͜ͱ
    • ΞΫηε͕૿͑Δ
    • ෛՙͱͷઓ͍͕࢝·Δ
    • αʔό͕૿͑Δ
    • ؔΘΔਓ΋૿͑Δ
    5

    View Slide

  6. ෛՙΛՄࢹԽ͢Δ

    View Slide

  7. ؂ࢹɾϞχλϦϯά
    • ʮͳΜͱͳ͘ෛՙߴͦ͏ʯͱ͍͏ঢ়ଶ͸ͱͯ΋ةݥ
    • ศརͳπʔϧ͕૿͖͍͑ͯͯΔͷͰͪΌΜͱ׆༻͢Δ
    • Nagios, CloudForecast, Cacti, Ganglia, Munin, Sensu, Zabbix, Kurado,
    Prometheus
    • Mackerel, Datadog, CloudWatch, NewRelic, PagerDuty, Pingdom
    • Fluentd+Elasticsearch+KibanaͰϩά͔ΒάϥϑΛੜ੒͢Δͷ΋ޮՌత
    • άϥϑ͸ຖ೔֬ೝ͢ΔɻཧղͰ͖ͳ͍άϥϑ͸ੜ੒ɾදࣔͯ͠΋ҙຯ͕ͳ͍ɻ
    • ຖ݄ෛՙঢ়گΛαϚϥΠζ͓ͯ͘͠ͱৼΓฦΓ΍ࠓޙͷରࡦํ਑͕࡞Γ΍͍͢
    • ݟ͑ͳ͍΋ͷɺଌΕͳ͍΋ͷ͸ϦεΫ
    • αʔϏε࿈ಈͷاը΍Πϕϯτ͸ૣΊʹڞ༗ͯ͠උ͑Δ
    7

    View Slide

  8. εέʔϧΞοϓ͔εέʔϧΞ΢τ͢΂͖͔

    View Slide

  9. εέʔϧΞοϓ or εέʔϧΞ΢τ
    • εέʔϧΞοϓ
    • ੑೳΛ্͛ͯෛՙରࡦ
    • εέʔϧͰ͖Δ্ݶ͕ܾ·ͬͯ͠·͏ɻ͞ΒͳΔෛՙ૿΁ͷରԠ͸ݫ͍͠ɻ
    • PCI-ExpressͳioMemory΍NVMe SSDΛ࢖ͬͯ΋μϝͳ࣌͸μϝɻ
    • εέʔϧΞ΢τ
    • LB഑ԼͷΞϓϦέʔγϣϯαʔό͸૿΍͢͜ͱͰCPUϦιʔεΛෛՙ෼ࢄ
    • DB΁ͷ઀ଓͱΞΫηε͕૿͑ΔͷͰཁ஫ҙ
    • DBʹ͓͍ͯ͸ɺεέʔϧΞ΢τͰ͖Δͱɺத௕ظతͳෛՙରࡦͱͯ͠༗ޮ
    • Ұ࣌తͳෛՙ૿΁ͷରࡦͰ͋Ε͹εέʔϧΞοϓ΋ޮՌత
    9

    View Slide

  10. τϥϑΟοΫͷ૿Ճ

    View Slide

  11. τϥϑΟοΫͷ૿Ճ
    • Ϋϥ΢υͷ৔߹
    • ΠϯελϯελΠϓʹΑ্ͬͯݶ͕ҟͳΔʢϕϯνϚʔΫͱ͓ͬͯ͘ͱ҆৺ʣ
    • ࣗࣾDCΛ࢖ͬͯΔ৔߹
    • NICͱεΠονͷଳҬ্ݶʢαʔόʔ͕ͲͷϥοΫʹ͋Δ͔Λҙࣝʣ
    • ճઢͷ৑௕ੑʢํܥ͕མͪͯ΋ڐ༰Ͱ͖ΔଳҬઃܭʣ
    • NICͷ৑௕ੑ
    • Bonding: ίωΫγϣϯ୯ҐͰ෼ࢄ͞ΕΔͷͰNICx2Ͱ2ഒͷଳҬͱ͸ݶΒͳ͍
    • TCPͷ࠶ૹճ਺ͷࢹ֮Խ
    • ಛʹΫϥ΢υͰ͸ϨΠςϯγ΍ଳҬͷมԽ͕͋ΔͷͰཁ஫ҙ
    11

    View Slide

  12. αʔόʔ/Πϯελϯε਺ͷ૿Ճ

    View Slide

  13. αʔόʔ/Πϯελϯε਺ͷ૿Ճ
    • σϓϩΠɺϓϩϏδϣχϯάͷޮ཰ԽɺࣗಈԽ
    • Capistrano, AWS CodeDeploy, Stretcher + Consul
    • Chef, Ansible, Puppet, Itamae
    • ϕʔεͱͳΔΠϝʔδʢAMIͳͲʣߏஙͷࣗಈԽ
    • Packerศར
    • ୆਺ʹґଘ͠ͳ͍ΦϖϨʔγϣϯ͕ཧ૝ܥ
    13

    View Slide

  14. σʔλϕʔε ΫΤϦ਺ͷ૿Ճ

    View Slide

  15. σʔλϕʔε ΫΤϦ਺ͷ૿Ճ
    • ϘτϧωοΫ͸READ͔WRITE͔
    • σʔλϕʔεɺςʔϒϧͷ୯ҐͰαʔόΛ෼ׂ
    • γϟʔσΟϯάʢID΍ϋογϡ஋ͷ৒༨Ͱ෼ׂʣ
    • ΩϟογϡʢMemcached, RedisͳͲʣͷ׆༻
    • ετϨʔδͷIOݶքͷݟۃΊʢΧλϩάεϖοΫΛ৴༻ͤͣܭଌ ͢Δʣ
    • tmp table, file sortͳͲ͕ൃੜ͍ͯ͠ͳ͍͔
    • Dirty Page΍waitͷൃੜස౓Λάϥϑ͔Β஌Δ
    • όοϑΝϓʔϧར༻཰ͷঢ়گ
    15

    View Slide

  16. σʔλϕʔε αΠζͷ૿Ճ

    View Slide

  17. σʔλϕʔε αΠζͷ૿Ճ
    • Ұ౓ᷓΕͯ͠·͏ͱରॲ͕ࠔ೉
    • ςʔϒϧͷσʔλɺΠϯσοΫεαΠζ
    • όοϑΝʔϓʔϧͷΩϟύγςΟ
    • ϝϞϦΛ૿΍ͯ͠ϓʔϧαΠζΛ޿͛Δͱ͍͏ख
    • όΠφϦϩάͷ૿Ճɻexpire_logs_daysͷௐ੔ʢMySQLͷ৔߹ʣɻ
    • AUTO INCREMENTͷ্ݶ஋ʹ஫ҙʢint͸໿21ԯɺbigint͸໿922ژʣ
    • intͩͱςʔϒϧʹΑͬͯ͸͋ͬͱ͍͏ؒʹୡͯ͠͠·͏ͷͰ஫ҙ
    • ᮢ஋Λઃఆͯ͠؂ࢹ͓ͯ͘͠ͱ҆৺
    17

    View Slide

  18. ਓһͷ૿Ճ

    View Slide

  19. ਓһͷ૿Ճ
    • ΞΧ΢ϯτ
    • Ϋϥ΢υͷΞΧ΢ϯτɺLinuxϢʔβʔΞΧ΢ϯτɺGitHubͷΞΧ΢ϯτ
    • AWSͷ৔߹ɺCloudTrailΛ࢖ͬͨ؂ࠪϩά
    • LDAPͳͲͰϩάΠϯϢʔβʔɺϗετͷ੍ݶʢηΩϡϦςΟͷ୲อʣ
    • ϓϩδΣΫτ΁৽ͨʹδϣΠϯˍൈ͚Δϝϯόʔ΁ͷରԠʢνΣοΫϦετͷςϯϓϨʔτʣ
    • Ξϥʔτ
    • ౰൪੍ɺΤεΧϨʔγϣϯํ๏ͳͲͷཱ֬ɺPagerDutyͷ׆༻
    • ϊ΢ϋ΢΍खॱͷڞ༗
    • Wikiͷ׆༻ʢྫ: GitHub Wiki, Qiita Team, Crowiʣ
    • ߏங΍ো֐ൃੜ࣌ͷखॱɺϧʔϧΛυΩϡϝϯτԽʢΠϯγσϯτൃੜ࣌ʹྫྷ੩ʹରॲͰ͖ΔΑ͏ʹʣ
    • 4 eyesͰͷ࡞ۀ֬ೝͯ͠ࣄނ๷ࢭʢྫ: DNSߋ৽ɺTerminateɺService StopͳͲʣ
    • ChatOpsͰ࡞ۀͰ͖Δ͜ͱΛ૿΍͢
    19

    View Slide

  20. ·ͱΊ

    View Slide

  21. ·ͱΊ
    • ن໛ʹ͔͔ΘΒͣجຊΛ͔ͬ͠Γͱ
    • ى͖͍ͯΔ͜ͱΛਖ਼͘͠೺Ѳ͢Δ͜ͱɺΠϝʔδͰ͖Δ͜ͱ
    • ن໛ʹൺྫͯ͠࡞ۀྔ͕૿͑ͳ͍Α͏ʹ͢Δ͜ͱ
    • ਓ਺͕૿͑ͨͱ͖ʹ࡞ۀෛՙ͕෼ࢄ͞ΕΔΑ͏ʹϊ΢ϋ΢΍खॱΛల։͢Δ͜ͱ
    • େن໛͡Όͳͯ͘΋΍ͬͨ΄͏͕͍͍͜ͱ͸ଟ͍
    • ஥ؒืूத https://xflag.com/recruit/
    21

    View Slide

  22. Thank you!

    View Slide