Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SRE Lounge 20180117

abnoumaru
January 17, 2018
6.3k

SRE Lounge 20180117

クローズな勉強会で話した資料を社外向けにしたものです

abnoumaru

January 17, 2018
Tweet

Transcript

  1. MSP ʹ͓͚ΔτΠϧ࡟ݮ
    גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ
    Ѩ෦ وথʢ @abnoumaru ʣ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1

    View full-size slide

  2. ࣗݾ঺հ
    • ৽ଔ2೥໨ೖࣾ6೥໨
    • ΠϯϑϥΤϯδχΞ
    • झຯ
    • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46
    • ͜͜࠷ۙྑ͘৮Δ
    • Plesk ɺKubernetes
    • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2

    View full-size slide

  3. ϋʔτϏʔπ
    • 24࣌ؒ365೔ͷ༗ਓ؂ࢹͷ MSP ࣄۀ
    • ओʹ Web γεςϜͷӡ༻Λ୅ߦʢϝσΟΞɺ EC αΠτɺήʔϜɺϝʔϧɾɾɾʣ
    • ϋʔτϏʔπͷΤϯδχΞͷ໾ׂ
    • ӡ༻άϧʔϓ 24࣌ؒ365೔ͷো֐ରԠɺӡ༻ɾอक࡞ۀɺݚम
    • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲౰ऀʣ
    • ٕज़։ൃࣨ ࣾ಺αʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾ෍ڭɺࣾ಺શମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ
    • SRE ͱ͍͏෦ॺ͸ͳ͍
    • SRE ຊΛʢ࿨༁ͯ͠ʣྠಡ͢Δ෦׆͸͋Δ
    • ΍͍ͬͯΔ͜ͱ͸ඇৗʹ͍ۙͷͰؔ৺͸ߴ͍
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3

    View full-size slide

  4. Managed Service Provider
    • αʔϏεʹ෇ਵͯ͠ൃੜ͢Δۀ຿Λ·ͱΊͯ੥͚ෛ͏ҰछͷΞ΢τιʔγϯά
    • αʔόӡ༻
    • αʔόͷख഑
    • OS΍ϛυϧ΢ΣΞͷΠϯετʔϧɾઃఆ
    • ӡ༻தͷ؂ࢹ
    • ো֐ରԠ
    • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά
    • 24࣌ؒ365೔ͷి࿩΍ϝʔϧʹΑΔαϙʔτ
    • ։ൃऀͰ͸ͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ΋͋Δ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4

    View full-size slide

  5. Site Reliability Engineering
    • ιϑτ΢ΣΞΤϯδχΞϦϯά
    • γεςϜΤϯδχΞϦϯά
    • τΠϧ࡟ݮ
    • Φʔόʔϔου
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5

    View full-size slide

  6. MSP ! SRE
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 6

    View full-size slide

  7. SRE ͷओͳৼΔ෣͍͔ΒݟͨϋʔτϏʔπ
    • ιϑτ΢ΣΞΤϯδχΞϦϯά
    • ΞϓϦ։ൃ͸୲౰Ͱ͸ͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͸͋Δ
    • ࣾ಺ͷγεςϜվળ
    • γεςϜΤϯδχΞϦϯά
    • ओʹ΍ͬͯΔ
    • τΠϧ࡟ݮ
    • ӡ༻ͷݱ৔͸τΠϧͷࢁ
    • Φʔόʔϔου
    • ଞͷاۀಉ༷ʹ͋Δ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7

    View full-size slide

  8. Site Reliability Engineering
    • ιϑτ΢ΣΞΤϯδχΞϦϯά
    • γεςϜΤϯδχΞϦϯά
    • τΠϧ࡟ݮ
    • ͦͷଞɺΦʔόʔϔου
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8

    View full-size slide

  9. τΠϧ
    αʔϏεʹؔ܎ͷ͋Δ࡞ۀͷ಺ɺҎԼͷ͍ͣΕ͔ʹ౰ͯ͸·Δ΋ͷ
    • ख࡞ۀͰ͋Δ
    • ܁Γฦ͞ΕΔ
    • ࣗಈԽͰ͖Δ
    • ઓज़తͰ͋Δ
    • ௕ظతՁ஋Λ࣋ͨͳ͍
    • αʔϏε੒௕ʹରͯ͠O(n)Ͱ͋Δ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9

    View full-size slide

  10. τΠϧͷྫ
    • ଟ਺͋Δ
    • ఆྫλεΫ
    • ؂ࢹઃఆ
    • υΩϡϝϯτ࡞੒
    • …
    • Nagios ͔Β௨஌͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ
    ӡ༻͔Β͢ΔͱτΠϧʁ
    • Ξϥʔτ࡟ݮʹτΠϧ࡟ݮͱ΋ݴ͑Δʁ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10

    View full-size slide

  11. τΠϧͰ͸ͳ͍
    • ߃ٱతͳมߋ
    • ࠜຊղܾͷͨΊͷνϡʔχϯά
    • RDBMS ͷΠϯσοΫε௥Ճ
    • …
    • ਓؒͷ൑அΛඞཁͱ͢Δ࡞ۀ
    • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭ൑அͷ಺༰΋ϓϩάϥϜͰॲཧͰ͖Δ৔߹͸
    τΠϧͱݴͬͯྑͦ͞͏
    • ʢ͜͏͍͏৔߹͸γεςϜ΍Ξϥʔτͷઃܭ͔Βݟ௚͢ඞཁ͕͋Δʣ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11

    View full-size slide

  12. τΠϧΛूΊΔ
    • ೔ʑͷΞϥʔτ֬ೝ
    • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
    • ۀ຿ؔ࿈ͷ৔߹͸ࢥཱ͍ͬͨਓ͕΍ͬͯΔ
    • ֤࡞ۀͷ࣌ؒɾස౓ͷચ͍ग़͠
    • ӡ༻ۀ຿ͷଟ͍ӡ༻άϧʔϓ͸࡞ۀͷ৘ใΛચ͍ग़ͨ͠
    • ༏ઌ౓΍Ͳͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12

    View full-size slide

  13. ೔ʑͷΞϥʔτ֬ೝ
    • ো֐ൃੜ
    • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ
    • ϝʔϧͰରԠใࠂૹ৴
    • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ
    • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13

    View full-size slide

  14. ೔ʑͷΞϥʔτ֬ೝ
    • ো֐ൃੜ
    • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ
    • ϝʔϧͰରԠใࠂૹ৴
    • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ
    • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14

    View full-size slide

  15. Slack ͷ Thread Ͱ૬ஊ
    • #alert-extermination ͱ͍͏෦԰ʹසൃΞϥʔτΛूΊΔ
    • ӡ༻άϧʔϓ΋͘͠͸ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
    • ӡ༻ͱΤϯδχΞͰײ૝΍஌ࣝΛدͤूΊͯରԠΛܾΊΔ
    • Thread ͷ಺༰͸ bot ͔Β Redmine ʹొ࿥͞ΕΔ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15

    View full-size slide

  16. ೔ʑͷΞϥʔτ֬ೝ
    • ো֐ൃੜ
    • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ
    • ϝʔϧͰରԠใࠂૹ৴
    • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ
    • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16

    View full-size slide

  17. ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ
    • ରԠใࠂ
    • ݱঢ়໨ࢹͰ֬ೝ
    • ਺͸೔ʹΑΔ
    • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔࿩͢
    • SRE ຊͰݴ͑͹ 12. Troubleshooting ͷτϦΞʔδʹ౰ͯ͸·Γͦ͏
    • ఆ࣌֬ೝ
    • ໰͍߹Θͤ΍ӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17

    View full-size slide

  18. HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18

    View full-size slide

  19. HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19

    View full-size slide

  20. ۀ຿ͷτΠϧ࡟ݮࣄྫ
    • ઌํґཔ
    • ઌํʹΑΔ༏ઌ౓͸ͳ͍ͷͰجຊతʹ͸͢΂ͯଈ࣌಺༰͸֬ೝʢࢲͷνʔϜ͸ʣ
    • ׂΓࠐΈ
    • ಺༰ʹΑͬͯௐ੔͸͋Δ
    • ස౓͕ߴ͍৔߹͸ࣗಈԽ͢Δ
    • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash ΍ Python Ͱ
    εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟ਺ʢ࢖͍·Θͤͳ͍͜ͱ͕͋Δͷ͸ͭΒ͍ʣ
    • ස౓ͷߴ͍ࣾ಺ͷ࡞ۀ
    • ιϑτ΢ΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20

    View full-size slide

  21. ؂ࢹઃఆͷࣗಈԽ
    • ݱࡏ 150 Ҋ݅ఔ౓ɺߏ੒ͷҟͳΔαʔό 2000 ୆͕૬ख
    • Nagios ʴ Cacti ΛखಈͰߋ৽
    • ෳ੡Ͱαʔό͕૿͑ΔΑΓॳΊͯ৮Δ؀ڥ΁ͷ؂ࢹઃఆ͕ଟ͍
    • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ
    • ೺Ѳ΍؀ڥ΁ͷରԠʹ͕͔͔࣌ؒΔ
    • Subversion Ͱ؅ཧ͞Ε͍ͯΔ
    • ৑௕ԽͷͨΊʹ༻ҙ͞Εͨෳ਺ͷ Nagios Λߋ৽
    • ࣾ಺Ͱ؂ࢹઃఆπʔϧΛ࡞੒͢Δ͜ͱͰղܾ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21

    View full-size slide

  22. happo1
    • Golang Ͱॻ͔Ε͍ͯΔ
    • ैདྷͷ؂ࢹ
    • Nagios 2 ୆ʢʴςετ༻ 1 ୆ʣʴ Cacti
    • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯ؅ཧɺ
    ຊ൪αʔό΁ίϛοτɺҰ୆ͣͭϩάΠϯͯ͠൓ө
    • ݱࡏͷ؂ࢹ
    • Nagios 2 ୆ʢʴςετ༻ 1 ୆ʣʴ Grafana ʢ஋ͷऔಘ͸ mackerel-agent ʣ
    • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔ౓ੜ੒
    • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪؀ڥ΁൓ө
    1 https://github.com/heartbeatsjp/happo-agent
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22

    View full-size slide

  23. ·ͱΊ
    • SRE ຊͷ಺༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧ࡟ݮࣄྫΛ঺հ
    • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ
    • దࡐదॴʹ͍Ζ͍Ζͳํ๏Λ࢖ͬͯ஍ಓʹτΠϧΛ࡟ݮ͍ͯ͠Δ
    • ײ૝
    • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε͹
    SRE ຊͷ಺༰ͱ೔ʑͷۀ຿Λݟൺ΂ͯΈΔͱSRE Β͍͠ৼΔ෣͍͕݁ߏ
    ݟ͔ͭΔ
    HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23

    View full-size slide