Tweet
Tweet

Slide 1

Slide 1 text

MSP ʹ͓͚ΔτΠϧ࡟ݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1

Slide 2

Slide 2 text

ࣗݾ঺հ • ৽ଔ2೥໨ೖࣾ6೥໨ • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 • ͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2

Slide 3

Slide 3 text

ϋʔτϏʔπ • 24࣌ؒ365೔ͷ༗ਓ؂ࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻Λ୅ߦʢϝσΟΞɺ EC αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷ໾ׂ • ӡ༻άϧʔϓ 24࣌ؒ365೔ͷো֐ରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲౰ऀʣ • ٕज़։ൃࣨ ࣾ಺αʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾ෍ڭɺࣾ಺શମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺ͸ͳ͍ • SRE ຊΛʢ࿨༁ͯ͠ʣྠಡ͢Δ෦׆͸͋Δ • ΍͍ͬͯΔ͜ͱ͸ඇৗʹ͍ۙͷͰؔ৺͸ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3

Slide 4

Slide 4 text

Managed Service Provider • αʔϏεʹ෇ਵͯ͠ൃੜ͢Δۀ຿Λ·ͱΊͯ੥͚ෛ͏ҰछͷΞ΢τιʔγϯά • αʔόӡ༻ • αʔόͷख഑ • OS΍ϛυϧ΢ΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷ؂ࢹ • ো֐ରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365೔ͷి࿩΍ϝʔϧʹΑΔαϙʔτ • ։ൃऀͰ͸ͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ΋͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4

Slide 5

Slide 5 text

Site Reliability Engineering • ιϑτ΢ΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧ࡟ݮ • Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5

Slide 6

Slide 6 text

MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 6

Slide 7

Slide 7 text

SRE ͷओͳৼΔ෣͍͔ΒݟͨϋʔτϏʔπ • ιϑτ΢ΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ͸୲౰Ͱ͸ͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͸͋Δ • ࣾ಺ͷγεςϜվળ • γεςϜΤϯδχΞϦϯά • ओʹ΍ͬͯΔ • τΠϧ࡟ݮ • ӡ༻ͷݱ৔͸τΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7

Slide 8

Slide 8 text

Site Reliability Engineering • ιϑτ΢ΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧ࡟ݮ • ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8

Slide 9

Slide 9 text

τΠϧ αʔϏεʹؔ܎ͷ͋Δ࡞ۀͷ಺ɺҎԼͷ͍ͣΕ͔ʹ౰ͯ͸·Δ΋ͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ • ௕ظతՁ஋Λ࣋ͨͳ͍ • αʔϏε੒௕ʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9

Slide 10

Slide 10 text

τΠϧͷྫ • ଟ਺͋Δ • ఆྫλεΫ • ؂ࢹઃఆ • υΩϡϝϯτ࡞੒ • … • Nagios ͔Β௨஌͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • Ξϥʔτ࡟ݮʹτΠϧ࡟ݮͱ΋ݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10

Slide 11

Slide 11 text

τΠϧͰ͸ͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫε௥Ճ • … • ਓؒͷ൑அΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭ൑அͷ಺༰΋ϓϩάϥϜͰॲཧͰ͖Δ৔߹͸ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏৔߹͸γεςϜ΍Ξϥʔτͷઃܭ͔Βݟ௚͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11

Slide 12

Slide 12 text

τΠϧΛूΊΔ • ೔ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ • ۀ຿ؔ࿈ͷ৔߹͸ࢥཱ͍ͬͨਓ͕΍ͬͯΔ • ֤࡞ۀͷ࣌ؒɾස౓ͷચ͍ग़͠ • ӡ༻ۀ຿ͷଟ͍ӡ༻άϧʔϓ͸࡞ۀͷ৘ใΛચ͍ग़ͨ͠ • ༏ઌ౓΍Ͳͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12

Slide 13

Slide 13 text

೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13

Slide 14

Slide 14 text

೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14

Slide 15

Slide 15 text

Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦԰ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ΋͘͠͸ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST • ӡ༻ͱΤϯδχΞͰײ૝΍஌ࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ಺༰͸ bot ͔Β Redmine ʹొ࿥͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15

Slide 16

Slide 16 text

೔ʑͷΞϥʔτ֬ೝ • ো֐ൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ͸ Slack ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16

Slide 17

Slide 17 text

ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়໨ࢹͰ֬ೝ • ਺͸೔ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔࿩͢ • SRE ຊͰݴ͑͹ 12. Troubleshooting ͷτϦΞʔδʹ౰ͯ͸·Γͦ͏ • ఆ࣌֬ೝ • ໰͍߹Θͤ΍ӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17

Slide 18

Slide 18 text

HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18

Slide 19

Slide 19 text

HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19

Slide 20

Slide 20 text

ۀ຿ͷτΠϧ࡟ݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌ౓͸ͳ͍ͷͰجຊతʹ͸͢΂ͯଈ࣌಺༰͸֬ೝʢࢲͷνʔϜ͸ʣ • ׂΓࠐΈ • ಺༰ʹΑͬͯௐ੔͸͋Δ • ස౓͕ߴ͍৔߹͸ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash ΍ Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟ਺ʢ࢖͍·Θͤͳ͍͜ͱ͕͋Δͷ͸ͭΒ͍ʣ • ස౓ͷߴ͍ࣾ಺ͷ࡞ۀ • ιϑτ΢ΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20

Slide 21

Slide 21 text

؂ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔ౓ɺߏ੒ͷҟͳΔαʔό 2000 ୆͕૬ख • Nagios ʴ Cacti ΛखಈͰߋ৽ • ෳ੡Ͱαʔό͕૿͑ΔΑΓॳΊͯ৮Δ؀ڥ΁ͷ؂ࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ೺Ѳ΍؀ڥ΁ͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱ؅ཧ͞Ε͍ͯΔ • ৑௕ԽͷͨΊʹ༻ҙ͞Εͨෳ਺ͷ Nagios Λߋ৽ • ࣾ಺Ͱ؂ࢹઃఆπʔϧΛ࡞੒͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21

Slide 22

Slide 22 text

happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷ؂ࢹ • Nagios 2 ୆ʢʴςετ༻ 1 ୆ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯ؅ཧɺ ຊ൪αʔό΁ίϛοτɺҰ୆ͣͭϩάΠϯͯ͠൓ө • ݱࡏͷ؂ࢹ • Nagios 2 ୆ʢʴςετ༻ 1 ୆ʣʴ Grafana ʢ஋ͷऔಘ͸ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔ౓ੜ੒ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪؀ڥ΁൓ө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22

Slide 23

Slide 23 text

·ͱΊ • SRE ຊͷ಺༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧ࡟ݮࣄྫΛ঺հ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏Λ࢖ͬͯ஍ಓʹτΠϧΛ࡟ݮ͍ͯ͠Δ • ײ૝ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε͹ SRE ຊͷ಺༰ͱ೔ʑͷۀ຿Λݟൺ΂ͯΈΔͱSRE Β͍͠ৼΔ෣͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23