Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE Lounge 20180117
Search
abnoumaru
January 17, 2018
0
6.6k
SRE Lounge 20180117
クローズな勉強会で話した資料を社外向けにしたものです
abnoumaru
January 17, 2018
Tweet
Share
More Decks by abnoumaru
See All by abnoumaru
IVRyエンジニア忘年LT大会2024 クリティカルユーザージャーニーの整理
abnoumaru
0
210
ゆるSRE勉強会 #8 組織的にSREが始まる中で意識したこと
abnoumaru
2
1.5k
3-shake SRE Tech Talk #10 LLMのO11yに触れる
abnoumaru
2
11k
マイクロサービスの現場からプラットフォームエンジニアリングの可能性を探る!
abnoumaru
2
9.8k
SLOいつ決めましょう?
abnoumaru
5
2.3k
あなたらしくSRE(公開用)
abnoumaru
5
7.7k
IDCFクラウドを使ってどこまでチューニングできるか試してみた
abnoumaru
0
220
AWS認定ソリューションアーキテクトを受けた話
abnoumaru
1
1.9k
Featured
See All Featured
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.4k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.2k
Docker and Python
trallard
43
3.2k
The Cult of Friendly URLs
andyhume
78
6.1k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
45
2.3k
Unsuck your backbone
ammeep
669
57k
Speed Design
sergeychernyshev
25
740
Fireside Chat
paigeccino
34
3.1k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
26
1.9k
Building an army of robots
kneath
302
45k
Faster Mobile Websites
deanohume
305
30k
Transcript
MSP ʹ͓͚ΔτΠϧݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB
Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
ࣗݾհ • ৽ଔ2ೖࣾ6 • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •
͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
ϋʔτϏʔπ • 24࣌ؒ365ͷ༗ਓࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻ΛߦʢϝσΟΞɺ EC
αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷׂ • ӡ༻άϧʔϓ 24࣌ؒ365ͷোରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲ऀʣ • ٕज़։ൃࣨ ࣾαʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾڭɺࣾશମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺͳ͍ • SRE ຊΛʢ༁ͯ͠ʣྠಡ͢Δ෦׆͋Δ • ͍ͬͯΔ͜ͱඇৗʹ͍ۙͷͰؔ৺ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
Managed Service Provider • αʔϏεʹਵͯ͠ൃੜ͢ΔۀΛ·ͱΊ͚ͯෛ͏ҰछͷΞτιʔγϯά • αʔόӡ༻ • αʔόͷख •
OSϛυϧΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷࢹ • োରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365ͷిϝʔϧʹΑΔαϙʔτ • ։ൃऀͰͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (
@abnoumaru ) 6
SRE ͷओͳৼΔ͍͔ΒݟͨϋʔτϏʔπ • ιϑτΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ୲Ͱͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͋Δ • ࣾͷγεςϜվળ • γεςϜΤϯδχΞϦϯά
• ओʹͬͯΔ • τΠϧݮ • ӡ༻ͷݱτΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
τΠϧ αʔϏεʹؔͷ͋Δ࡞ۀͷɺҎԼͷ͍ͣΕ͔ʹͯ·Δͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ
• ظతՁΛ࣋ͨͳ͍ • αʔϏεʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
τΠϧͷྫ • ଟ͋Δ • ఆྫλεΫ • ࢹઃఆ • υΩϡϝϯτ࡞ •
… • Nagios ͔Β௨͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • ΞϥʔτݮʹτΠϧݮͱݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
τΠϧͰͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫεՃ • …
• ਓؒͷஅΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭஅͷ༰ϓϩάϥϜͰॲཧͰ͖Δ߹ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏߹γεςϜΞϥʔτͷઃܭ͔Βݟ͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
τΠϧΛूΊΔ • ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
• ۀؔ࿈ͷ߹ࢥཱ͍ͬͨਓ͕ͬͯΔ • ֤࡞ۀͷ࣌ؒɾසͷચ͍ग़͠ • ӡ༻ۀͷଟ͍ӡ༻άϧʔϓ࡞ۀͷใΛચ͍ग़ͨ͠ • ༏ઌͲͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ͘͠ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
• ӡ༻ͱΤϯδχΞͰײࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ༰ bot ͔Β Redmine ʹొ͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়ࢹͰ֬ೝ • ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔͢ •
SRE ຊͰݴ͑ 12. Troubleshooting ͷτϦΞʔδʹͯ·Γͦ͏ • ఆ࣌֬ೝ • ͍߹Θͤӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19
ۀͷτΠϧݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌͳ͍ͷͰجຊతʹͯ͢ଈ࣌༰֬ೝʢࢲͷνʔϜʣ • ׂΓࠐΈ • ༰ʹΑͬͯௐ͋Δ •
ස͕ߴ͍߹ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟʢ͍·Θͤͳ͍͜ͱ͕͋ΔͷͭΒ͍ʣ • සͷߴ͍ࣾͷ࡞ۀ • ιϑτΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔɺߏͷҟͳΔαʔό 2000 ͕૬ख • Nagios ʴ
Cacti ΛखಈͰߋ৽ • ෳͰαʔό͕૿͑ΔΑΓॳΊͯ৮Δڥͷࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ѲڥͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱཧ͞Ε͍ͯΔ • ԽͷͨΊʹ༻ҙ͞Εͨෳͷ Nagios Λߋ৽ • ࣾͰࢹઃఆπʔϧΛ࡞͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷࢹ • Nagios 2 ʢʴςετ༻
1 ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯཧɺ ຊ൪αʔόίϛοτɺҰͣͭϩάΠϯͯ͠ө • ݱࡏͷࢹ • Nagios 2 ʢʴςετ༻ 1 ʣʴ Grafana ʢͷऔಘ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔੜ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪ڥө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
·ͱΊ • SRE ຊͷ༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧݮࣄྫΛհ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏ΛͬͯಓʹτΠϧΛݮ͍ͯ͠Δ
• ײ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε SRE ຊͷ༰ͱʑͷۀΛݟൺͯΈΔͱSRE Β͍͠ৼΔ͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23