Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Mackerelにおける Cloud Nativeへの取り組みと チームへ与えた変化 / CloudNative Days Tokyo 2020
Hayato Imai
September 08, 2020
Technology
2
920
Mackerelにおける Cloud Nativeへの取り組みと チームへ与えた変化 / CloudNative Days Tokyo 2020
Hayato Imai
September 08, 2020
Tweet
Share
More Decks by Hayato Imai
See All by Hayato Imai
hayajo
19
7.1k
hayajo
1
1.3k
hayajo
0
2.4k
hayajo
6
1.1k
hayajo
1
3.9k
hayajo
16
9.7k
hayajo
0
250
hayajo
1
2.9k
hayajo
3
1.7k
Other Decks in Technology
See All in Technology
hanasuke
0
220
sogaoh
2
420
line_developers
PRO
2
370
shigeruoda
2
220
oztick139
0
650
okuken
0
320
comucal
PRO
0
280
yuhta28
0
130
ishiayaya
PRO
0
130
ayatokura
0
240
teruterubohz_ibm
0
160
surumegohan
1
160
Featured
See All Featured
keavy
107
14k
smashingmag
232
18k
smashingmag
283
47k
orderedlist
PRO
330
36k
bermonpainter
343
26k
kastner
54
2k
philnash
10
660
jmmastey
10
690
samlambert
237
10k
maggiecrowley
10
540
vanstee
117
4.9k
zakiwarfel
88
3.4k
Transcript
Mackerelʹ͓͚Δ Cloud NativeͷऔΓΈͱ νʔϜ༩͑ͨมԽ CloudNative Days Tokyo 2020
ࣗݾհ • ࠓҪ൏ਓ (id:hayajo_77 / @hayajo) • גࣜձࣾͯͳ • MackerelνʔϜ
• SRE https://developer.hatenastaff.com/entry/2019/06/10/120000
Mackerel?
None
None
None
Cloud Nativeͱ
“εέʔϥϒϧͳΞϓϦέʔγϣϯΛߏஙɾ ࣮ߦ͢ΔͨΊͷೳྗΛ৫ʹͨΒ͢” https://github.com/cncf/toc/blob/master/DEFINITION.md
Cloud NativeΛ࣮ݱ͢Δʹ
2ͭͷΞϓϩʔν͕ඞཁ • εέʔϥϒϧͳΞϓϦέʔγϣϯͷߏங ◦ ճ෮ੑɺཧྗɺՄ؍ଌੑͷ͋ΔγεςϜͷߏங ◦ Cloud Native Trail Map,
Cloud Native Landscape • εέʔϥϒϧͳΞϓϦέʔγϣϯͷӡ༻ ◦ ΞϓϦέʔγϣϯͷ։ൃɺϦϦʔεɺӡ༻ϑϩʔͷ࣮ફ ◦ DevOpsϓϥΫςΟε
ࠓճͷ͓
Cloud NativeͷऔΓΈͱνʔϜʹ༩͑ͨӨڹ 1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ 2. ϞχλϦϯάαΠυΧʔͷ։ൃ 3. OpenTelemetryϥΠϒϥϦͷ։ൃ 4. CI/CDͷߏங
5. Ҡߦ࣌ͷΧφϦΞϦϦʔε 6. SLOΛఆٛ 7. ϚωʔδυαʔϏεͷҠߦ 8. Terraformͷ࠾༻ 9. োରԠԋश 10. ίϯςφνΣοΫϦετͷ࡞ ͦΕͧΕͷऔΓΈؔ࿈͍͋ͬͯ͠ΔͨΊɺհॱ࣮ࡍʹऔΓΜͩॱংΛө͍ͯ͠ΔͷͰ͋Γ·ͤΜɻ
1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ
1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ ~ͬͨ͜ͱ~ • ίϯςφԽͨ͜͠ͱͰΞϓϦέʔγϣϯઃܭ͕มԽ • ίϯςφσβΠϯύλʔϯʹΑΔઃܭΛߦ͏Α͏ʹͳͬͨ ◦ αΠυΧʔɺΞϯόαμɺΞμϓλͳͲ •
ίϯςφ୯ҐͰؔ৺Λ͠ɺσβΠϯύλʔϯʹԊͬͨܗͰػೳ Ճػೳͷڞ௨Խɺૄ݁߹ԽΛਤͬͨ ◦ ࢹػೳΛՃ͢ΔΤʔδΣϯτϝοηʔδΩϡʔͱΓͱ Γ͢ΔϓϩΩγͳͲ
1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ΞϓϦέʔγϣϯͷґଘػೳΛίϯςφʹΓग़͢͜ͱͰίʔ υΛγϯϓϧʹͰ͖ͨ ◦ ґଘػೳΛ࠶ར༻Մೳͳίϯςφͱͯ͠ϞδϡʔϧԽͰ͖ͨ
• ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ػೳ͝ͱʹϦιʔεΛͰ͖ɺΞϓϦέʔγϣϯಈ࡞ͷӨ ڹΛগͳ͘͢Δ͜ͱ͕Մೳͱͳͬͨ
2. ϞχλϦϯάαΠυΧʔͷ։ൃ
2. ϞχλϦϯάαΠυΧʔͷ։ൃ ~ͬͨ͜ͱ~ • ίϯςφͰಈ࡞͢ΔΞϓϦέʔγϣϯΛࢹ͢Δඞཁ͕͋Δ ◦ Ϧιʔεͷར༻ঢ়گ ◦ ΞϓϦέʔγϣϯґଘ͢ΔαʔϏεͷϝτϦοΫ •
ઐ༻ͷϞχλϦϯάΤʔδΣϯτΛ։ൃ • αΠυΧʔίϯςφͱͯ͠σϓϩΠ
2. ϞχλϦϯάαΠυΧʔͷ։ൃ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ αΠυΧʔʹ͢Δ͜ͱͰΞϓϦέʔγϣϯʹखΛೖΕΔ͜ͱͳ͘ࢹ ػೳΛՃ ◦ ӡ༻ΤϯδχΞʹΑͬͯσϓϩΠ͞Εͦͷઃఆʹด͍ͯͨ͡ࢹػೳ
͕ɺ։ൃΤϯδχΞͰ؆୯ʹՃՄೳʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ࢹઃఆϑΝΠϧΛશʹɺઃఆ͢Δඞཁ͕ͳ͘ͳͬͨ ◦ σϓϩΠϑϩʔ͕ίϯςφͷσϓϩΠʹ౷Ұ͞Εͨ
3. OpenTelemetryϥΠϒϥϦͷ։ൃ
3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~ͬͨ͜ͱ~ • ϞχλϦϯάαΠυΧʔΞϓϦέʔγϣϯ֎෦ ͔Βͷܭଌ • ΑΓৄ͍͠ܭଌΞϓϦέʔγϣϯ෦Ͱߦ͏ඞཁ͕͋Δ ◦ ΞϓϦέʔγϣϯΛվम͢Δ͜ͱʹͳΔ
3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~ͬͨ͜ͱ~ • ʰmackerelexporter-goʱ ◦ https://github.com/mackerelio-labs/mackerelexporter-go • ΞϓϦέʔγϣϯ෦Ͱܭଌ͠ɺͦͷσʔλΛMackerelΤΫε ϙʔτ͢ΔGoϥΠϒϥϦ
3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ಛఆͷؔͷ࣮ߦ࣌ؒͳͲɺΞϓϦέʔγϣϯಈ࡞ʹؔΘΔ (APMͷΑ͏ͳ)ϝτϦοΫͷऔಘ͕༰қʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ
◦ ΞϓϦέʔγϣϯ෦ͷڍಈʹৄ͘͠ͳ͍ӡ༻ΤϯδχΞͰ ΞϓϦέʔγϣϯͷৄࡉͳϞχλϦϯά͕࣮͘͢͠ͳͬͨ
4. CI/CDͷߏங
4. CI/CDͷߏங ~ͬͨ͜ͱ~ • ίϯςφӡ༻ʹ߹ΘͤͨCI/CDΛ࠶ߏங ◦ ैདྷͷCI/CDπʔϧͷӡ༻ɺڞ௨ج൫ͷσϓϩΠπʔϧ͔Βͷ ٫తͱͯ͋ͬͨ͠ • ίϯςφϦϙδτϦΛڥքʹBuildͱReleaseΛ͚ͯߟ͑Δ
◦ ૄ݁߹ʹ͢Δ͜ͱͰॊೈੑ্͕͕Γπʔϧͷબࢶ͕͕Δ
4. CI/CDͷߏங ~ͬͨ͜ͱ~ • Build Stage ◦ Ռ (ΞʔςΟϑΝΫτ) Λ࡞͢Δεςʔδ
◦ ςετɺίϯςφΠϝʔδͷϏϧυɺϦϙδτϦͷPUSH ◦ GitHub Actions (or Jenkins) • Release Stage ◦ ՌΛϦϦʔε͢Δεςʔδ ◦ PUSH͞ΕͨίϯςφΠϝʔδͷσϓϩΠ ◦ CodePipeline + CodeDeploy
GitHub GHA or Jenkins Code Pipeline Code Deploy ECS ECR
S3 Build Stage Release Stage 4. CI/CDͷߏங ~ͬͨ͜ͱ~ Test/Build Push Image Put Metadata Get Metadata Start Deployment Deploy Pull Image
4. CI/CDͷߏங ~νʔϜʹ༩͑ͨӨڹ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ CI/CDπʔϧͷӡ༻ෛՙݮ ◦ Blue/GreenσϓϩΠͷ࠾༻ ◦
؆୯ͳϩʔϧόοΫ ◦ γϯϓϧͳϦϦʔεखॱͷ࣮ݱ
5. ҠߦظؒͷΧφϦΞϦϦʔε
5. ҠߦظؒͷΧφϦΞϦϦʔε ~ͬͨ͜ͱ~ • طଘγεςϜ͔ΒECSஈ֊తʹτϥϑΟοΫΛྲྀ༷ͯ͠ࢠΛݟ ͭͭҠߦ͍ͨ͠ • ALBͷWeightedRoutingΛར༻࣮ͯ͠ݱ ◦ ҠߦରͷALBʹECS༻ͷλʔήοτάϧʔϓΛ࡞͠ɺECS
λεΫWeightedRouting͢Δ • σϓϩΠ࣌ʹCodePipelineͷLambdaͰઃఆ
5. ҠߦظؒͷΧφϦΞϦϦʔε ~ͬͨ͜ͱ~ Code Pipeline Code Deploy ECS Lambda ALB
TG For EC2 TG For ECS Start Deployment Deploy Run Collect Task IPs Modify Listener Rule Register Task IPs Weighted Routing
5. ҠߦظؒͷΧφϦΞϦϦʔε ~νʔϜʹ༩͑ͨӨڹ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ECSҠߦظؒʹ͓͚ΔϦϦʔεͷϦεΫ͕ݮͬͨ ▪ ෆ۩߹͕͋ͬͨ߹Ͱૣ͍ஈ֊Ͱؾͮ͘͜ͱ͕Ͱ͖ͨ ▪
ΞϓϦέʔγϣϯઃఆΛΓସ͑Δඞཁͳ͘ɺ WeightedRoutingͷൺΛઃఆͯ͠σϓϩΠ͚ͩ͢͠Ͱ؆ ୯ʹݩͷঢ়گʹ͢͜ͱ͕Ͱ͖ͨ
6. SLOΛఆٛ
6. SLOΛఆٛ ~ͬͨ͜ͱ~ • SLO(αʔϏεϨϕϧඪ)Λఆٛ ◦ MackerelνʔϜͰSREΛ࣮ફ͓ͯ͠ΓɺνʔϜͰSLOΛܾΊ ͍ͯΔ • νʔϜͰ߹ҙͷ্ͰɺఆྔతͳஅʹΑͬͯ։ൃɾӡ༻ΛਐΊΔ͜ͱ
͕Ͱ͖Δ ◦ Cloud NativeͷҠߦγεςϜͷมߋ͕͏ ◦ ࣮ӡ༻͍ͯ͠ΔαʔϏεΛ҆৺ͯ͠Cloud NativeʹҠߦ͍ͯ͘͠
6. SLOΛఆٛ ~νʔϜʹ༩͑ͨมԽ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ Ҡߦ࣌ͷޭϨΠςϯγʹΑΔΓ͠அ ▪ ΧφϦΞϦϦʔεͷൺΓ͠ͳͲ ◦
ϞχλϦϯά͍ͨ͠ϝτϦοΫ(SLI)͕໌֬ʹͳͬͨ ▪ ϞχλϦϯάαΠυΧʔOpenTelemetryͰՄࢹԽ ◦ ࠓ·Ͱհͨ͠औΓΈ༰ͱSREͷݪଇ͕૬ޓʹ࡞༻ͯ͠ӡ༻ ΛճͤΔΑ͏ʹͳͬͨ
7. ϚωʔδυαʔϏεͷҠߦ
7. ϚωʔδυαʔϏεͷҠߦ ~ͬͨ͜ͱ~ • γεςϜશମΛεέʔϥϒϧʹ͢ΔͨΊʹΞϓϦέʔγϣϯ͕ґ ଘ͢Δίϯϙʔωϯτεέʔϥϒϧʹ͢Δඞཁ͕͋Δ • ࣗલͰߏஙɺӡ༻ཧ͍ͯͨ͠ίϯϙʔωϯτΛϚωʔδυαʔϏ εҠߦͨ͠
7. ϚωʔδυαʔϏεͷҠߦ ~ͬͨ͜ͱ~ • ʮAWS Purpose-Built Databases WeekʯͰElastiCacheͷҠߦࣄ ྫΛհ •
ʰMackerelͷ࣌ܥྻσʔλϕʔεʹ͓͚ΔRedis Clusterͷར༻ ͱAmazon ElastiCacheͷҠߦʹ͍ͭͯʱ • https://speakerdeck.com/astj/aws-purpose-built-databases-week
7. ϚωʔδυαʔϏεͷҠߦ ʙ νʔϜʹ༩͑ͨมԽ ʙ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ෳࡶͳϓϩϏδϣχϯάπʔϧɺσϓϩΠπʔϧ͔Βͷ٫ ◦
CloudFormation, TerraformͳͲͷϚωʔδυαʔϏεͱੑͷߴ͍ πʔϧʹΑͬͯߏஙָ͕ʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ εέʔϥϒϧͰߴՄ༻ͳγεςϜΛ࣮ݱ ◦ ཧʹඞཁͳखॱೳྗ͕গͳ͘ͳͬͨ ◦ ΩϟύγςΟϓϥϯχϯά͕Γ͘͢ͳͬͨ
8. Terraformͷ࠾༻
8. Terraformͷ࠾༻ ~ͬͨ͜ͱ~ • ͜Ε·ͰෳͷπʔϧΛͬͯΠϯϑϥߏங͍ͯͨ͠ ◦ CloudFormation, Chef, CapistranoͳͲ •
ϚωʔδυαʔϏεҠߦ͢Δ͜ͱͰIaC͍͕͢͠Ͱ͖ͨ • ϚϧνΫϥυରԠΛߟྀͯ͠TerraformΛ࠾༻ͨ͠
8. Terraformͷ࠾༻ ~ͬͨ͜ͱ~ • MackerelνʔϜͰͷTerraform׆༻ͷ༷ࢠʮHashiTalks: Japan 2020ʯͰհ ◦ ʰTerraformʹΑΔϓϥΫςΟεͷͱվળʱ ◦
https://speakerdeck.com/heleeen/our-best-practice
8. Terraformͷ࠾༻ ~νʔϜʹ༩͑ͨมԽ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ σϓϩΠ·ΘΓϚωʔδυαʔϏεʹҠߦͨ͜͠ͱͰ։ൃΤϯδχ ΞʹΘ͔Γ͍͢ίʔυԽ͕ਐΜͩ ◦ ෳͷڥͰಉ͡Α͏ʹ࡞͢ΔϦιʔεఆٛΛϞδϡʔϧͱͯ͠ڞ
௨Խ͢Δ͜ͱͰߏஙͷෛ୲͕ݮͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ γεςϜߏΛϦϙδτϦཧͯ͠։ൃϑϩʔʹࡌͤΔ͜ͱͰɺ։ൃ ΤϯδχΞͰมߋ͘͢͠ͳͬͨ
9. োରԠԋश
9. োରԠԋश ~ͬͨ͜ͱ~ • ίϯςφӡ༻ϚωʔδυαʔϏεͷҠߦʹΑΓɺγεςϜͷ ཧɾӡ༻༷มΘΓ͖ͯͨ͠ • ։ൃΤϯδχΞΛަ͑ͯোԋशΛ࣮ࢪ͠ɺΞʔΩςΫνϟͱӡ༻ खॱɺোରԠखॱΛֶͿ͜ͱͰͦͷཧղΛଅਐͨ͠
9. োରԠԋश ~ͬͨ͜ͱ~ • ͜Ε·ͰͷোରԠԋशͷऔΓΈͪ͜Βͷࢿྉࢀর ◦ ʮϓϩμΫτΛ10ӡ༻͢Δ νʔϜΛͭ͘Δʯ ◦ https://speakerdeck.com/daiksy/devsumi2020?slide=39
9. োରԠԋश ~νʔϜʹ༩͑ͨมԽ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ίϯςφԽϚωʔδυαʔϏεͷར༻ͰɺϦιʔεͷ࠶ߏங खॱಛఆͷϛυϧΣΞͷਂ͍ࣝͳͲɺҎલʹൺΔͱӡ ༻ʹඞཁͳखॱೳྗ͕গͳ͘ͳͬͨͱ࣮ײͰ͖ͨ ◦
ଐਓੑ͕ഉআ͞ΕͯνʔϜʹ৽͘͠ՃΘͬͨϝϯόʔͷΦϯϘʔ σΟϯά͕εϜʔζʹͳͬͨ
10. ίϯςφνΣοΫϦετͷ࡞
10. ίϯςφνΣοΫϦετͷ࡞ ~ͬͨ͜ͱ~ • ଞνʔϜͰίϯςφӡ༻ͷҠߦ͕૿͖͑ͯͨ • ͯ͢ͷνʔϜʹίϯςφӡ༻ʹৄ͍͠ΤϯδχΞ͕͍ΔΘ͚Ͱ ͳ͍ • ίϯςφӡ༻ͷUnknown
unknownsΛͳͨ͘͢ΊɺଞνʔϜͷΤϯ δχΞͱڠྗͯ͠ίϯςφӡ༻ͷݟΛνΣοΫϦετͱͯ͠·ͱ Ίͨ
10. ίϯςφνΣοΫϦετͷ࡞ ~ͬͨ͜ͱ~
10. ίϯςφνΣοΫϦετͷ࡞ ~͜ͷऔΓΈͰظ͢ΔมԽ~ • ৽نίϯςφӡ༻ʹ͚ͨॳಈָ͕ʹͳΔ • طଘͷίϯςφӡ༻ʹ͓͚Δߟྀ࿙Ε࣮ݱϨϕϧΛ֬ೝͰ͖Δ • ৫શମͰCloud NativeʹऔΓΉʹɺϊϋࣄྫΛࣾͰڞ
༗͢Δ͜ͱ͕ॏཁ
νʔϜ༩͑ͨมԽͷ·ͱΊ
ଐਓੑͷഉআ • औΓΈΛਐΊΔ͜ͱͰ։ൃΤϯδχΞɺӡ༻ΤϯδχΞͦΕͧΕ ͷྖҬ͕ࣗવʹ͍ۙͮͯΦʔόʔϥοϓ͢Δ෦͕૿͖͑ͯͨ ◦ ίϯςφσβΠϯύλʔϯʹΑΔઃܭՄ؍ଌੑͷͨΊͷϥΠϒ ϥϦͷ։ൃɺIaCͳͲɺίϯςφΛத৺ͱͨ͠Cloud Nativeπʔ ϧͰ͜ΕΛ࣮ݱ͘͢͠ͳ͍ͬͯΔ •
։ൃ͕εέʔϥϒϧʹͳ͖͍ͬͯͯΔ
ߏஙɾӡ༻ෛՙͷݮ • ϚωδϝϯταʔϏεͷҠߦCloud Nativeπʔϧͷ׆༻Ͱεέʔ ϥϒϧͳγεςϜͷߏங͕؆୯ʹͳͬͨ • ίϯςφΛϕʔεͱͨ͠ճ෮ੑͷ͋ΔߴՄ༻ͳγεςϜϦϦʔε࡞ ۀͷվળʹΑΓɺӡ༻ෛՙ͕ݮͨ͠
Cloud NativeͱSREͷ૬ޓ࡞༻ • SLOΛఆٛ͢Δ͜ͱͰఆྔతͳஅͰ҆৺ͯ͠Cloud NativeΛਐΊΔ ͜ͱ͕Ͱ͖Δ • Cloud NativeΛਐΊΔ͜ͱͰՄ؍ଌੑ͕ߴ·ΓɺSLOͷఆٛΛ࢝Ί ͱͨ͠SREΛ͞Βʹਪਐ͠ɺνʔϜΛεέʔϥϒϧʹͰ͖Δ
• ͦΕͧΕ͕࡞༻͠߹͍ɺੵۃతʹมԽ͍ͯͨ͘͠Ίͷ͕Ͱ͖ͭ ͭ͋Δ
·ͱΊ
·ͱΊ • Cloud Nativeͷ࣮ݱʹ2ͭͷΞϓϩʔν͕ඞཁ ◦ εέʔϥϒϧͳΞϓϦέʔγϣϯͷߏங ◦ εέʔϥϒϧͳΞϓϦέʔγϣϯͷӡ༻ • ྆ํΛ܁Γฦ͠ܧଓͯ͠औΓΉ͜ͱͰνʔϜʹมԽ͕ੜ·ΕΔ
• มԽ͕ੵΈॏͳΔ͜ͱͰCloud NativeͳνʔϜʹͳΔ
curl -sIL mackerel.io | grep career