Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
Ken Hamada
June 23, 2018
Technology
0
4.9k
Mackerelチームにおけるインフラオーナーシップ / Hatena Pepabo tech conf 4 DevOps Kyoto
Mackerelチームにおけるインフラオーナーシップ
Ken Hamada
June 23, 2018
Tweet
Share
More Decks by Ken Hamada
See All by Ken Hamada
サーバレスアーキテクチャによる時系列データベースの構築と監視 / Serverlessconf Tokyo 2017
itchyny
18
26k
Other Decks in Technology
See All in Technology
Istio入門
nutslove
15
5k
ソフトウェアライセンス 2022 / Software License 2022
cybozuinsideout
PRO
1
1.1k
OPENLOGI Company Profile
hr01
0
490
ROS再入門-はじめてのSLAM-
miura55
0
400
QiitaConference2022
fuwasegu
0
180
SI企業が「アジャイル推し」になったら 幸せになれますか?/Can SI company be happy if it becomes “Agile stan” ?
chinmo
1
1.2k
ラブグラフ紹介資料 〜プロダクト解体新書〜 / Lovegraph Product Deck
lovegraph
0
250
FoodTechにおける商流・金流・物流の進化/Evolution of Commercial, Financial, and Logistics in FoodTech
dskst
0
400
MRTK3 - DataBinding and Theming 入門
futo23
0
180
How to start with DDD when you have a Monolith
javujavichi
0
300
モブに早く慣れたい人のためのガイド / A Guide to Getting Started Quickly with Mob Programming
cybozuinsideout
PRO
2
1.8k
miisan's career talk
mii3king
0
220
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1M
Fontdeck: Realign not Redesign
paulrobertlloyd
73
4.1k
Atom: Resistance is Futile
akmur
255
20k
Code Reviewing Like a Champion
maltzj
506
37k
Design by the Numbers
sachag
271
17k
For a Future-Friendly Web
brad_frost
166
7.4k
Designing the Hi-DPI Web
ddemaree
272
32k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
100
5.9k
Scaling GitHub
holman
451
140k
How to Ace a Technical Interview
jacobian
265
21k
A designer walks into a library…
pauljervisheath
196
16k
The Illustrated Children's Guide to Kubernetes
chrisshort
15
36k
Transcript
MackerelνʔϜʹ͓͚Δ ΠϯϑϥΦʔφʔγοϓ גࣜձࣾͯͳ id:itchyny 2018/6/23 ͯͳɾϖύϘٕज़େձ#4
ࣗݾհ גࣜձࣾͯͳ ΞϓϦέʔγϣϯΤϯδχΞ MackerelνʔϜ ࢛ ͖: Go, Rust, Scala, Haskell,
Vim ϓϩάϥϜϞάϞά itchyny.hatenablog.com ⅓ా ݈ id:itchyny
ϓϩάϥϜϞάϞά ද࡞: lightline.vim
ʮΠϯϑϥΦʔφʔγοϓʯ
ΠϯϑϥΦʔφʔγοϓ • ։ൃνʔϜ͕ΠϯϑϥͷΦʔφʔγοϓΛ࣋ͭ • ϞχλϦϯάɾϗετཧɾΩϟύγςΟϓϥϯχϯά ։ൃڥCIڥͷߏஙɾΠϯϑϥߏͷઃܭ • ΠϯϑϥνʔϜωοτϫʔΫͳͲͷڞ௨ج൫ʹઐ೦ • DevOpsͷ͋ΓํΛݟͨ݁͠Ռ
৴པੑͱඅ༻ΛαʔϏε։ൃνʔϜ͕Λ࣋ͭ
͜Ε·Ͱ • ΠϯϑϥνʔϜ (ԣ۲৫) ͕ݸʑͷαʔϏεΛ୲ • αʔόʔͷཧϞχλϦϯάΠϯϑϥϝϯόʔ͕୲ • αʔϏεʹΑͬͯٻΊΒΕΔΠϯϑϥ͕ࣝΒΒ •
Apache + mod_perl + MySQL ΦϯϓϨ ͷαʔϏε͕ଟ → Scala + PostgreSQL / Go on EC2, ECS, Lambda… • ٻΊΒΕΔ৴པੑ (Մ༻ੑ): ಛʹtoBtoBtoC
• Πϯϑϥϝϯόʔͷ୲αʔϏεͷࣄ͕ଟ͘ɺ ج൫γεςϜͷཧ։ൃ͕͓Ζ͔ͦʹ… • Πϯϑϥϝϯόʔͷ୲αʔϏεସ͕͑ࠔ • োରԠଞͷΠϯϑϥϝϯόʔʹ͍͠ • ։ൃνʔϜ͕Մ༻ੑΠϯϑϥඅ༻Λग़࣌͢ʹ
ΠϯϑϥνʔϜΛר͖ࠐ·ͳ͍ͱ͍͚ͳ͍
͜Ε͔Β • ։ൃνʔϜ͕αʔόʔ͔ΒϛυϧΣΞͷཧΛߦ͏ • αʔόʔߏங/ཧɾΩϟύγςΟϓϥϯχϯά ։ൃڥɾCIڥߏஙɾϞχλϦϯάɾোରԠ • αʔϏεͷ৴པੑͱඅ༻Λ֤։ൃνʔϜ͕ཧ • ΠϯϑϥνʔϜԾԽɾωοτϫʔΫج൫ɾOSΛ୲
։ൃνʔϜʹͱͬͯ • ΠϯϑϥࣝΛ͚͍͔ͭͯ͘͠ͳ͍ • αʔόʔΛཱͯΒΕΔΑ͏ʹͳΔ • దͳࢹΛઃఆͰ͖ΔΑ͏ʹͳΔ • োରԠͰ͖ΔΑ͏ʹͳΔ •
Մ༻ੑΛτϥοΩϯάͰ͖ΔΑ͏ʹͳΔ
։ൃνʔϜͲ͏͢Ε ΠϯϑϥͷΦʔφʔγοϓΛ ࣗ৴ΛͬͯऔΕΔΑ͏ʹͳΔͷ͔
Mackerel mackerel.io
Mackerelͱ • SaaSͷαʔόʔཧɾࢹπʔϧ • ͯͳࣾͷαʔόʔཧγεςϜΛαʔϏεԽͯ͠ެ։ • Mackerelࡾ • Scala +
Go / PostgreSQL + Redis
ݱࡏͷνʔϜͷମ੍ • ΞϓϦέʔγϣϯΤϯδχΞ໊ + σβΠφʔ + Director + Producer +
Sales + CRE • Πϯϑϥϝϯόʔ1ਓ͕νʔϜʹೖ͍ͬͯΔ • ேձνʔϜͷձٞʹࢀՃ • σεΫྡ: ίϛϡχέʔγϣϯେࣄ
ೖࣾͨ࣌͠ͷࢲ • ΣϒΞϓϦέʔγϣϯͷΈΛΒͳ͍ • SQLΛΒͳ͍ • LinuxΛΒͳ͍ (ͳ͔ͥMacΛ͍ͬͯΔ) • AWSΛΒͳ͍
• ίʔυͪΐͬͱॻ͚Δ (ͳ͔ͥHaskellͱVim͕͖)
ඞཁͳࣝ • ͦͦαʔόʔͱɺࢹͱ… • σβΠφʔʹجຊతͳ͜ͱཧղͯ͠Β͏ • LinuxɾWindowsͷγεςϜϝτϦοΫ • ֤छϛυϧΣΞͷಛࢹɾϝτϦοΫɾάϥϑͷݟํ •
ϚωʔδυαʔϏε • ֤छνϟοτπʔϧͱͷ࿈ܞɹChefɾAnsible • rpm/debϦϙδτϦɾWindows msi
υοάϑʔσΟϯά • MackerelνʔϜ࠷ߴͷυοάϑʔσΟϯά͕Ͱ͖Δ • ࡞͍ͬͯΔαʔϏεΛΔ → ࣗͷαʔϏεΛࢹ͢Δ • ࢹ͍ͯ͠ͳ͍ͱ͜Ζ͕͔Δ →
࡞Δ → ࢹΛ࢝ΊΔ • ։ൃΤϯδχΞසൟʹը໘ΛݟΔ • ۀʹΈࠐΉ·Ͱ͕ػೳ։ൃ ྫ: σϓϩΠͰάϥϑΞϊςʔγϣϯ • ଞνʔϜͷ։ൃΤϯδχΞMackerelΛΑ͘͏
σϓϩΠ • σϓϩΠ୲ͷΤϯδχΞΛॱ൪ʹΞαΠϯ • ୲ΤϯδχΞμογϡϘʔυΛॱ൪ʹݟ͍ͯ͘ • ීஈͷϝτϦοΫͷ༷ࢠΛΔ܇࿅ ࢹ͢Δ = ҟৗঢ়ଶΛఆΊΔ
= ීஈΛΔ • σϓϩΠΠϯϑϥΛݟΔྑ͍ػձ • σϓϩΠͱɺͲ͜ʹԿΛͲ͏͢Δ͜ͱͳͷ͔ʁ • σϓϩΠίϚϯυͷཪଆͰԿ͕ى͖͍ͯΔͷ͔ʁ
MackerelνʔϜͷΤϯδχΞ ීஈͷ։ൃΛ͠ͳ͕Β ΠϯϑϥΛֶΜͰ͍Δ ֶ͟ΔΛಘͳ͍αʔϏεΛ࡞͍ͬͯΔ
͍߹Θͤͷྫ • MySQL/DockerϓϥάΠϯͰϝτϦοΫ͕औಘͰ͖ͳ͍ • ϒϥβʔͰݟΕΔͷʹ֎ܗࢹͰΤϥʔ͕ग़Δ • CPU steal͕100%ʹுΓ͍͍ͯΔ • WindowsͰprocess
queue length͕औΕ͍ͯͳ͍ • loadavg͕7࣌ؒ͝ͱʹ্ঢ͢Δ ֤छϛυϧΣΞOSɺ֎ܗࢹͰDNSͷΈͳͲͷ͕ࣝΘΕΔ
https://mackerel.io/ja/blog/entry/tech/high-loadavg-every-7-hours
αϙʔτ୲ • αϙʔτ͔ΒͷௐࠪґཔΛड͚ΔΤϯδχΞΛॱ൪ʹ୲ • ຖि୲Λճͯ͠શһ͕͋ͨΔ • ͔Βͳ͚Εଞͷਓͱڞʹௐࠪɾݟͷڞ༗ • ෆ۩߹ͷൃݟ मਖ਼ผλεΫͱͯ͠issueΛཱͯΔ
؆୯ͳͷ͍ͭͰʹ͢͜ͱ͋Δ
αϙʔτ୲Λճ͢͜ͱͰ Βͳ͍ػೳϛυϧΣΞΛֶͿ ௐࠪํ๏ɾΠϯϑϥࣝͷԣల։ loadavgʹ͍ͭͯҟৗʹৄ͘͠ͳͬͨΓ͢Δ
PWG • Performance Working Group • ύϑΥʔϚϯεͷ֬ೝɾى͖ͨোͱରࡦਐḿͷ֬ೝ • ࢀՃऀΠϯϑϥΤϯδχΞͱ։ൃΤϯδχΞશһ •
࢘ձ୲Λॱ൪ʹճ͢: ϝϯόʔͷཧղΛਂΊΔ ฉ͚ͩ͘ΑΓɺ࢘ձͱͯ͠આ໌͢Δ͜ͱͰཧղ • άϥϑΛݟΔίʔφʔ
োରԠৼΓฦΓ • োͷৼΓฦΓͱผʹɺରԠͨ͠खॱͳͲΛ֬ೝ • ଐਓੑ: ܦݧଟ͍ΤϯδχΞ͕ҰॠͰͯ͠͠·͏ ԿΛࢥͬͯԿΛͲ͏ݟͯͲͷΑ͏ʹஅ͔ͨ͠ • ͲͷάϥϑΛݟ͔ͨɺͲͷαʔόʔͷϩάΛݟ͔ͨ •
ରԠ͠ͳ͔ͬͨਓ͕stagingͰ࣮ࡍʹखॱΛ࠶ݱ • ো࣌ͷௐࠪͷͨΊͷΈͮ͘Γ ϩάཧը໘ͷվળɾμογϡϘʔυ࡞ͳͲ োى͖ͯཉ͘͠ͳ͍͕ɺى͖ͨͱ͖ଟ͘ͷֶͼΛಘΔ
Մ༻ੑɾίεττϥοΩϯά • Մ༻ੑ: ো͝ͱʹه͠ɺPWGͰ֬ೝ • ։ൃνʔϜ͕ظ͝ͱʹඪΛ࣋ͭ • SLAެ։Λࢦ͢ • ίετ:
ΠϯϑϥϝϯόʔͱσΟϨΫλʔ͕֬ೝձ • ։ൃϝϯόʔAWS cost explorerͰ֬ೝͰ͖Δ • Node.js 6.10→8.10 Ͱ Lambdaͷίετ1ׂݮ
None
ϚωʔδυαʔϏε • ࣌ܥྻσʔλϕʔε: LambdaɾDynamoDBɾS3 • Πϯϑϥ৮Γ͘͢ͳ͖͍ͬͯͯΔ • ΩϟύγςΟʔมߋ୭ͰͰ͖Δ࣌ • ʮ࣌ܥྻσʔλϕʔεݟڞ༗ձʯ
• ։ൃΤϯδχΞࣗવͱίετΛݟΔΑ͏ʹͳΔ • Cost ExplorerͰreportΛඋ
ίϯςφԽ • deploy / rollback͕ѹతʹָʹͳΔ • ΦʔτεέʔϧΛߟྀͨ͠ίϯςφج൫ • ࣾͷChefڞ௨ج൫ͷґଘΛࣙΊΒΕΔ •
ϞχλϦϯάؚΊͯࠓͷ՝ • MackerelͷίϯςφରԠ։ൃதͰ͢ • Dockerfileͱ͍͏ڞ௨ݴޠɾϛυϧΣΞͱͷଓΛએݴతʹߦ͏
࠷ۙͷMackerelνʔϜͷձ • DevʮݴͬͯͨϓϥάΠϯͷόάͨ͠ͷͰΞοϓσʔτ͓͍ͯͨ͠ʯ OpsʮͲͲʯ • Devʮ͜͜ϝτϦοΫݟ͓͚ͯΑͦ͞͏ͳͷͰࢹ࡞͓͍ͬͯͨΑʯ Opsʮྃղʯ • DevʮલݴΘΕͯͨ৽ػೳ࣮ͨ͠Αʙʯ Opsʮͬͨʔศརʙʯ
• Opsʮstagingͷ΄͛΄͛ΛίϯςφԽ͠·ͨ͠ʯ Devʮ͗͢͢͝Δʯ
·ͱΊ • ΠϯϑϥΛֶͿͷָ͍͠ • ٕज़ͷਐาͰΠϯϑϥཧָʹͳ͖͍ͬͯͯΔ • ௐୡɾεέʔϧΞτɾϩʔϧόοΫAPIͰͰ͖Δ࣌ • ιϑτΣΞʹΑͬͯࣗಈԽ͞ΕΔ࣌ •
ΠϯϑϥΤϯδχΞωοτϫʔΫ/OSͷϨΠϠʔʹઐ೦ • ΦϖϨʔγϣϯπʔϧΛ͘͢࡞ΕΔਓڧ͍
αʔϏεզ͕ࢠ
None