Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SRE Lounge 20180117

46557bd7d7383bb6bee229a21d112b4a?s=47 abnoumaru
January 17, 2018
5.2k

SRE Lounge 20180117

クローズな勉強会で話した資料を社外向けにしたものです

46557bd7d7383bb6bee229a21d112b4a?s=128

abnoumaru

January 17, 2018
Tweet

Transcript

  1. MSP ʹ͓͚ΔτΠϧ࡟ݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB

    Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
  2. ࣗݾ঺հ • ৽ଔ2೥໨ೖࣾ6೥໨ • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •

    ͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
  3. ϋʔτϏʔπ • 24࣌ؒ365೔ͷ༗ਓ؂ࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻Λ୅ߦʢϝσΟΞɺ EC

    αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷ໾ׂ • ӡ༻άϧʔϓ 24࣌ؒ365೔ͷো֐ରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲౰ऀʣ • ٕज़։ൃࣨ ࣾ಺αʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾ෍ڭɺࣾ಺શମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺ͸ͳ͍ • SRE ຊΛʢ࿨༁ͯ͠ʣྠಡ͢Δ෦׆͸͋Δ • ΍͍ͬͯΔ͜ͱ͸ඇৗʹ͍ۙͷͰؔ৺͸ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
  4. Managed Service Provider • αʔϏεʹ෇ਵͯ͠ൃੜ͢Δۀ຿Λ·ͱΊͯ੥͚ෛ͏ҰछͷΞ΢τιʔγϯά • αʔόӡ༻ • αʔόͷख഑ •

    OS΍ϛυϧ΢ΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷ؂ࢹ • ো֐ରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365೔ͷి࿩΍ϝʔϧʹΑΔαϙʔτ • ։ൃऀͰ͸ͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ΋͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
  5. Site Reliability Engineering • ιϑτ΢ΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧ࡟ݮ •

    Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
  6. MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (

    @abnoumaru ) 6
  7. SRE ͷओͳৼΔ෣͍͔ΒݟͨϋʔτϏʔπ • ιϑτ΢ΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ͸୲౰Ͱ͸ͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͸͋Δ • ࣾ಺ͷγεςϜվળ • γεςϜΤϯδχΞϦϯά

    • ओʹ΍ͬͯΔ • τΠϧ࡟ݮ • ӡ༻ͷݱ৔͸τΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
  8. Site Reliability Engineering • ιϑτ΢ΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧ࡟ݮ •

    ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
  9. τΠϧ αʔϏεʹؔ܎ͷ͋Δ࡞ۀͷ಺ɺҎԼͷ͍ͣΕ͔ʹ౰ͯ͸·Δ΋ͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ

    • ௕ظతՁ஋Λ࣋ͨͳ͍ • αʔϏε੒௕ʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
  10. τΠϧͷྫ • ଟ਺͋Δ • ఆྫλεΫ • ؂ࢹઃఆ • υΩϡϝϯτ࡞੒ •

    … • Nagios ͔Β௨஌͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • Ξϥʔτ࡟ݮʹτΠϧ࡟ݮͱ΋ݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
  11. τΠϧͰ͸ͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫε௥Ճ • …

    • ਓؒͷ൑அΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭ൑அͷ಺༰΋ϓϩάϥϜͰॲཧͰ͖Δ৔߹͸ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏৔߹͸γεςϜ΍Ξϥʔτͷઃܭ͔Βݟ௚͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
  12. τΠϧΛूΊΔ • ೔ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ

    • ۀ຿ؔ࿈ͷ৔߹͸ࢥཱ͍ͬͨਓ͕΍ͬͯΔ • ֤࡞ۀͷ࣌ؒɾස౓ͷચ͍ग़͠ • ӡ༻ۀ຿ͷଟ͍ӡ༻άϧʔϓ͸࡞ۀͷ৘ใΛચ͍ग़ͨ͠ • ༏ઌ౓΍Ͳͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
  13. ೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack

    ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
  14. ೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack

    ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
  15. Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦԰ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ΋͘͠͸ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST

    • ӡ༻ͱΤϯδχΞͰײ૝΍஌ࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ಺༰͸ bot ͔Β Redmine ʹొ࿥͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
  16. ೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack

    ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
  17. ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়໨ࢹͰ֬ೝ • ਺͸೔ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔࿩͢ •

    SRE ຊͰݴ͑͹ 12. Troubleshooting ͷτϦΞʔδʹ౰ͯ͸·Γͦ͏ • ఆ࣌֬ೝ • ໰͍߹Θͤ΍ӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
  18. HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18

  19. HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19

  20. ۀ຿ͷτΠϧ࡟ݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌ౓͸ͳ͍ͷͰجຊతʹ͸͢΂ͯଈ࣌಺༰͸֬ೝʢࢲͷνʔϜ͸ʣ • ׂΓࠐΈ • ಺༰ʹΑͬͯௐ੔͸͋Δ •

    ස౓͕ߴ͍৔߹͸ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash ΍ Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟ਺ʢ࢖͍·Θͤͳ͍͜ͱ͕͋Δͷ͸ͭΒ͍ʣ • ස౓ͷߴ͍ࣾ಺ͷ࡞ۀ • ιϑτ΢ΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
  21. ؂ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔ౓ɺߏ੒ͷҟͳΔαʔό 2000 ୆͕૬ख • Nagios ʴ

    Cacti ΛखಈͰߋ৽ • ෳ੡Ͱαʔό͕૿͑ΔΑΓॳΊͯ৮Δ؀ڥ΁ͷ؂ࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ೺Ѳ΍؀ڥ΁ͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱ؅ཧ͞Ε͍ͯΔ • ৑௕ԽͷͨΊʹ༻ҙ͞Εͨෳ਺ͷ Nagios Λߋ৽ • ࣾ಺Ͱ؂ࢹઃఆπʔϧΛ࡞੒͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
  22. happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷ؂ࢹ • Nagios 2 ୆ʢʴςετ༻

    1 ୆ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯ؅ཧɺ ຊ൪αʔό΁ίϛοτɺҰ୆ͣͭϩάΠϯͯ͠൓ө • ݱࡏͷ؂ࢹ • Nagios 2 ୆ʢʴςετ༻ 1 ୆ʣʴ Grafana ʢ஋ͷऔಘ͸ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔ౓ੜ੒ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪؀ڥ΁൓ө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
  23. ·ͱΊ • SRE ຊͷ಺༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧ࡟ݮࣄྫΛ঺հ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏Λ࢖ͬͯ஍ಓʹτΠϧΛ࡟ݮ͍ͯ͠Δ

    • ײ૝ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε͹ SRE ຊͷ಺༰ͱ೔ʑͷۀ຿Λݟൺ΂ͯΈΔͱSRE Β͍͠ৼΔ෣͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23