Mackerelにおける Cloud Nativeへの取り組みと チームへ与えた変化 / CloudNative Days Tokyo 2020

Transcript

1. Mackerelʹ͓͚Δ Cloud Native΁ͷऔΓ૊Έͱ νʔϜ΁༩͑ͨมԽ CloudNative Days Tokyo 2020
2. ࣗݾ঺հ • ࠓҪ൏ਓ (id:hayajo_77 / @hayajo) • גࣜձࣾ͸ͯͳ • MackerelνʔϜ • SRE https://developer.hatenastaff.com/entry/2019/06/10/120000
3. Mackerel?
7. Cloud Nativeͱ͸
8. “εέʔϥϒϧͳΞϓϦέʔγϣϯΛߏஙɾ ࣮ߦ͢ΔͨΊͷೳྗΛ૊৫ʹ΋ͨΒ͢” https://github.com/cncf/toc/blob/master/DEFINITION.md
9. Cloud NativeΛ࣮ݱ͢Δʹ͸
10. 2ͭͷΞϓϩʔν͕ඞཁ • εέʔϥϒϧͳΞϓϦέʔγϣϯͷߏங ◦ ճ෮ੑɺ؅ཧྗɺՄ؍ଌੑͷ͋ΔγεςϜͷߏங ◦ Cloud Native Trail Map, Cloud Native Landscape • εέʔϥϒϧͳΞϓϦέʔγϣϯͷӡ༻ ◦ ΞϓϦέʔγϣϯͷ։ൃɺϦϦʔεɺӡ༻ϑϩʔͷ࣮ફ ◦ DevOpsϓϥΫςΟε
11. ࠓճͷ͓࿩
12. Cloud Native΁ͷऔΓ૊ΈͱνʔϜʹ༩͑ͨӨڹ 1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ 2. ϞχλϦϯάαΠυΧʔͷ։ൃ 3. OpenTelemetryϥΠϒϥϦͷ։ൃ 4. CI/CDͷߏங 5. Ҡߦ࣌ͷΧφϦΞϦϦʔε 6. SLOΛఆٛ 7. ϚωʔδυαʔϏε΁ͷҠߦ 8. Terraformͷ࠾༻ 9. ো֐ରԠԋश 10. ίϯςφνΣοΫϦετͷ࡞੒ ͦΕͧΕͷऔΓ૊Έ͸ؔ࿈͍͋ͬͯ͠ΔͨΊɺ঺հॱ͸࣮ࡍʹऔΓ૊ΜͩॱংΛ൓ө͍ͯ͠Δ΋ͷͰ͸͋Γ·ͤΜɻ
13. 1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ
14. 1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ ~΍ͬͨ͜ͱ~ • ίϯςφԽͨ͜͠ͱͰΞϓϦέʔγϣϯઃܭ͕มԽ • ίϯςφσβΠϯύλʔϯʹΑΔઃܭΛߦ͏Α͏ʹͳͬͨ ◦ αΠυΧʔɺΞϯόαμɺΞμϓλͳͲ • ίϯςφ୯ҐͰؔ৺Λ෼཭͠ɺσβΠϯύλʔϯʹԊͬͨܗͰػೳ ௥Ճ΍ػೳͷڞ௨Խɺૄ݁߹ԽΛਤͬͨ ◦ ؂ࢹػೳΛ௥Ճ͢ΔΤʔδΣϯτ΍ϝοηʔδΩϡʔͱ΍Γͱ Γ͢ΔϓϩΩγͳͲ
15. 1. ίϯςφσβΠϯύλʔϯʹΑΔઃܭ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ΞϓϦέʔγϣϯͷґଘػೳΛίϯςφʹ੾Γग़͢͜ͱͰίʔ υΛγϯϓϧʹͰ͖ͨ ◦ ґଘػೳΛ࠶ར༻Մೳͳίϯςφͱͯ͠ϞδϡʔϧԽͰ͖ͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ػೳ͝ͱʹϦιʔεΛ෼཭Ͱ͖ɺΞϓϦέʔγϣϯಈ࡞΁ͷӨ ڹΛগͳ͘͢Δ͜ͱ͕Մೳͱͳͬͨ
16. 2. ϞχλϦϯάαΠυΧʔͷ։ൃ
17. 2. ϞχλϦϯάαΠυΧʔͷ։ൃ ~΍ͬͨ͜ͱ~ • ίϯςφͰಈ࡞͢ΔΞϓϦέʔγϣϯΛ؂ࢹ͢Δඞཁ͕͋Δ ◦ Ϧιʔεͷར༻ঢ়گ ◦ ΞϓϦέʔγϣϯ΍ґଘ͢ΔαʔϏεͷϝτϦοΫ • ઐ༻ͷϞχλϦϯάΤʔδΣϯτΛ։ൃ • αΠυΧʔίϯςφͱͯ͠σϓϩΠ
18. 2. ϞχλϦϯάαΠυΧʔͷ։ൃ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ αΠυΧʔʹ͢Δ͜ͱͰΞϓϦέʔγϣϯʹखΛೖΕΔ͜ͱͳ͘؂ࢹ ػೳΛ௥Ճ ◦ ӡ༻ΤϯδχΞʹΑͬͯσϓϩΠ͞Εͦͷઃఆʹด͍ͯͨ͡؂ࢹػೳ ͕ɺ։ൃΤϯδχΞͰ΋؆୯ʹ௥ՃՄೳʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ؂ࢹઃఆϑΝΠϧΛશ୆ʹ഑෍ɺઃఆ͢Δඞཁ͕ͳ͘ͳͬͨ ◦ σϓϩΠϑϩʔ͕ίϯςφͷσϓϩΠʹ౷Ұ͞Εͨ
19. 3. OpenTelemetryϥΠϒϥϦͷ։ൃ
20. 3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~΍ͬͨ͜ͱ~ • ϞχλϦϯάαΠυΧʔ͸ΞϓϦέʔγϣϯ֎෦ ͔Βͷܭଌ • ΑΓৄ͍͠ܭଌ͸ΞϓϦέʔγϣϯ಺෦Ͱߦ͏ඞཁ͕͋Δ ◦ ΞϓϦέʔγϣϯΛվम͢Δ͜ͱʹͳΔ
21. 3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~΍ͬͨ͜ͱ~ • ʰmackerelexporter-goʱ ◦ https://github.com/mackerelio-labs/mackerelexporter-go • ΞϓϦέʔγϣϯ಺෦Ͱܭଌ͠ɺͦͷσʔλΛMackerel΁ΤΫε ϙʔτ͢ΔGoϥΠϒϥϦ
22. 3. OpenTelemetryϥΠϒϥϦͷ։ൃ ~νʔϜʹ༩͑ͨӨڹ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ಛఆͷؔ਺ͷ࣮ߦ࣌ؒͳͲɺΞϓϦέʔγϣϯಈ࡞ʹؔΘΔ (APMͷΑ͏ͳ)ϝτϦοΫͷऔಘ͕༰қʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ΞϓϦέʔγϣϯ಺෦ͷڍಈʹৄ͘͠ͳ͍ӡ༻ΤϯδχΞͰ΋ ΞϓϦέʔγϣϯͷৄࡉͳϞχλϦϯά͕࣮૷͠΍͘͢ͳͬͨ
23. 4. CI/CDͷߏங
24. 4. CI/CDͷߏங ~΍ͬͨ͜ͱ~ • ίϯςφӡ༻ʹ߹ΘͤͨCI/CDΛ࠶ߏங ◦ ैདྷͷCI/CDπʔϧͷӡ༻ɺڞ௨ج൫ͷσϓϩΠπʔϧ͔Βͷ୤ ٫΋໨తͱͯ͋ͬͨ͠ • ίϯςφϦϙδτϦΛڥքʹBuildͱReleaseΛ෼͚ͯߟ͑Δ ◦ ૄ݁߹ʹ͢Δ͜ͱͰॊೈੑ্͕͕Γπʔϧͷબ୒ࢶ͕޿͕Δ
25. 4. CI/CDͷߏங ~΍ͬͨ͜ͱ~ • Build Stage ◦ ੒Ռ෺ (ΞʔςΟϑΝΫτ) Λ࡞੒͢Δεςʔδ ◦ ςετɺίϯςφΠϝʔδͷϏϧυɺϦϙδτϦ΁ͷPUSH ◦ GitHub Actions (or Jenkins) • Release Stage ◦ ੒Ռ෺ΛϦϦʔε͢Δεςʔδ ◦ PUSH͞ΕͨίϯςφΠϝʔδͷσϓϩΠ ◦ CodePipeline + CodeDeploy
26. GitHub GHA or Jenkins Code Pipeline Code Deploy ECS ECR S3 Build Stage Release Stage 4. CI/CDͷߏங ~΍ͬͨ͜ͱ~ Test/Build Push Image Put Metadata Get Metadata Start Deployment Deploy Pull Image
27. 4. CI/CDͷߏங ~νʔϜʹ༩͑ͨӨڹ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ CI/CDπʔϧͷӡ༻ෛՙ௿ݮ ◦ Blue/GreenσϓϩΠͷ࠾༻ ◦ ؆୯ͳϩʔϧόοΫ ◦ γϯϓϧͳϦϦʔεखॱͷ࣮ݱ
28. 5. ҠߦظؒͷΧφϦΞϦϦʔε
29. 5. ҠߦظؒͷΧφϦΞϦϦʔε ~΍ͬͨ͜ͱ~ • طଘγεςϜ͔ΒECS΁͸ஈ֊తʹτϥϑΟοΫΛྲྀ༷ͯ͠ࢠΛݟ ͭͭҠߦ͍ͨ͠ • ALBͷWeightedRoutingΛར༻࣮ͯ͠ݱ ◦ Ҡߦର৅ͷALBʹECS༻ͷλʔήοτάϧʔϓΛ࡞੒͠ɺECS λεΫ΁WeightedRouting͢Δ • σϓϩΠ࣌ʹCodePipelineͷLambdaͰઃఆ
30. 5. ҠߦظؒͷΧφϦΞϦϦʔε ~΍ͬͨ͜ͱ~ Code Pipeline Code Deploy ECS Lambda ALB TG For EC2 TG For ECS Start Deployment Deploy Run Collect Task IPs Modify Listener Rule Register Task IPs Weighted Routing
31. 5. ҠߦظؒͷΧφϦΞϦϦʔε ~νʔϜʹ༩͑ͨӨڹ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ECSҠߦظؒʹ͓͚ΔϦϦʔεͷϦεΫ͕ݮͬͨ ▪ ෆ۩߹͕͋ͬͨ৔߹Ͱ΋ૣ͍ஈ֊Ͱؾͮ͘͜ͱ͕Ͱ͖ͨ ▪ ΞϓϦέʔγϣϯઃఆΛ੾Γସ͑Δඞཁͳ͘ɺ WeightedRoutingͷൺ཰Λઃఆͯ͠σϓϩΠ͠௚͚ͩ͢Ͱ؆ ୯ʹݩͷঢ়گʹ໭͢͜ͱ͕Ͱ͖ͨ
32. 6. SLOΛఆٛ
33. 6. SLOΛఆٛ ~΍ͬͨ͜ͱ~ • SLO(αʔϏεϨϕϧ໨ඪ)Λఆٛ ◦ MackerelνʔϜͰ͸SREΛ࣮ફ͓ͯ͠ΓɺνʔϜ಺ͰSLOΛܾΊ ͍ͯΔ • νʔϜͰ߹ҙͷ্Ͱɺఆྔతͳ൑அʹΑͬͯ։ൃɾӡ༻ΛਐΊΔ͜ͱ ͕Ͱ͖Δ ◦ Cloud Native΁ͷҠߦ͸γεςϜͷมߋ͕൐͏ ◦ ࣮ӡ༻͍ͯ͠ΔαʔϏεΛ҆৺ͯ͠Cloud NativeʹҠߦ͍ͯ͘͠
34. 6. SLOΛఆٛ ~νʔϜʹ༩͑ͨมԽ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ Ҡߦ࣌ͷ੒ޭ཰΍ϨΠςϯγʹΑΔ੾Γ໭͠൑அ ▪ ΧφϦΞϦϦʔεͷൺ཰΍੾Γ໭͠ͳͲ ◦ ϞχλϦϯά͍ͨ͠ϝτϦοΫ(SLI)͕໌֬ʹͳͬͨ ▪ ϞχλϦϯάαΠυΧʔ΍OpenTelemetryͰՄࢹԽ ◦ ࠓ·Ͱ঺հͨ͠औΓ૊Έ಺༰ͱSREͷݪଇ͕૬ޓʹ࡞༻ͯ͠ӡ༻ ΛճͤΔΑ͏ʹͳͬͨ
35. 7. ϚωʔδυαʔϏε΁ͷҠߦ
36. 7. ϚωʔδυαʔϏε΁ͷҠߦ ~΍ͬͨ͜ͱ~ • γεςϜશମΛεέʔϥϒϧʹ͢ΔͨΊʹ͸ΞϓϦέʔγϣϯ͕ґ ଘ͢Δίϯϙʔωϯτ΋εέʔϥϒϧʹ͢Δඞཁ͕͋Δ • ࣗલͰߏஙɺӡ༻؅ཧ͍ͯͨ͠ίϯϙʔωϯτΛϚωʔδυαʔϏ ε΁Ҡߦͨ͠
37. 7. ϚωʔδυαʔϏε΁ͷҠߦ ~΍ͬͨ͜ͱ~ • ʮAWS Purpose-Built Databases WeekʯͰElastiCache΁ͷҠߦࣄ ྫΛ঺հ • ʰMackerelͷ࣌ܥྻσʔλϕʔεʹ͓͚ΔRedis Clusterͷར༻ ͱAmazon ElastiCache΁ͷҠߦʹ͍ͭͯʱ • https://speakerdeck.com/astj/aws-purpose-built-databases-week
38. 7. ϚωʔδυαʔϏε΁ͷҠߦ ʙ νʔϜʹ༩͑ͨมԽ ʙ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ ෳࡶͳϓϩϏδϣχϯάπʔϧɺσϓϩΠπʔϧ͔Βͷ୤٫ ◦ CloudFormation, TerraformͳͲͷϚωʔδυαʔϏεͱ਌࿨ੑͷߴ͍ πʔϧʹΑͬͯߏஙָ͕ʹͳͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ εέʔϥϒϧͰߴՄ༻ͳγεςϜΛ࣮ݱ ◦ ؅ཧʹඞཁͳखॱ΍ೳྗ͕গͳ͘ͳͬͨ ◦ ΩϟύγςΟϓϥϯχϯά͕΍Γ΍͘͢ͳͬͨ
39. 8. Terraformͷ࠾༻
40. 8. Terraformͷ࠾༻ ~΍ͬͨ͜ͱ~ • ͜Ε·Ͱ͸ෳ਺ͷπʔϧΛ࢖ͬͯΠϯϑϥߏங͍ͯͨ͠ ◦ CloudFormation, Chef, CapistranoͳͲ • ϚωʔδυαʔϏε΁Ҡߦ͢Δ͜ͱͰIaC͠΍͍͢౔୆͕Ͱ͖ͨ • ϚϧνΫϥ΢υରԠΛߟྀͯ͠TerraformΛ࠾༻ͨ͠
41. 8. Terraformͷ࠾༻ ~΍ͬͨ͜ͱ~ • MackerelνʔϜͰͷTerraform׆༻ͷ༷ࢠ͸ʮHashiTalks: Japan 2020ʯͰ঺հ ◦ ʰTerraformʹΑΔϓϥΫςΟεͷ഑෍ͱվળʱ ◦ https://speakerdeck.com/heleeen/our-best-practice
42. 8. Terraformͷ࠾༻ ~νʔϜʹ༩͑ͨมԽ~ • ʮߏஙʯʹ༩͑ͨӨڹ ◦ σϓϩΠ·ΘΓ΋ϚωʔδυαʔϏεʹҠߦͨ͜͠ͱͰ։ൃΤϯδχ Ξʹ΋Θ͔Γ΍͍͢ίʔυԽ͕ਐΜͩ ◦ ෳ਺ͷ؀ڥͰಉ͡Α͏ʹ࡞੒͢ΔϦιʔεఆٛΛϞδϡʔϧͱͯ͠ڞ ௨Խ͢Δ͜ͱͰߏஙͷෛ୲͕ݮͬͨ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ γεςϜߏ੒ΛϦϙδτϦ؅ཧͯ͠։ൃϑϩʔʹࡌͤΔ͜ͱͰɺ։ൃ ΤϯδχΞͰ΋มߋ͠΍͘͢ͳͬͨ
43. 9. ো֐ରԠԋश
44. 9. ো֐ରԠԋश ~΍ͬͨ͜ͱ~ • ίϯςφӡ༻΍ϚωʔδυαʔϏε΁ͷҠߦʹΑΓɺγεςϜͷ؅ ཧɾӡ༻͸༷มΘΓ͖ͯͨ͠ • ։ൃΤϯδχΞΛަ͑ͯো֐ԋशΛ࣮ࢪ͠ɺΞʔΩςΫνϟͱӡ༻ खॱɺো֐ରԠखॱΛֶͿ͜ͱͰͦͷཧղΛଅਐͨ͠
45. 9. ো֐ରԠԋश ~΍ͬͨ͜ͱ~ • ͜Ε·Ͱͷো֐ରԠԋश΁ͷऔΓ૊Έ͸ͪ͜Βͷࢿྉ΋ࢀর ◦ ʮϓϩμΫτΛ10೥ӡ༻͢Δ νʔϜΛͭ͘Δʯ ◦ https://speakerdeck.com/daiksy/devsumi2020?slide=39
46. 9. ো֐ରԠԋश ~νʔϜʹ༩͑ͨมԽ~ • ʮӡ༻ʯʹ༩͑ͨӨڹ ◦ ίϯςφԽ΍ϚωʔδυαʔϏεͷར༻ͰɺϦιʔεͷ࠶ߏங खॱ΍ಛఆͷϛυϧ΢ΣΞͷਂ͍஌ࣝͳͲɺҎલʹൺ΂Δͱӡ ༻ʹඞཁͳखॱ΍ೳྗ͕গͳ͘ͳͬͨͱ࣮ײͰ͖ͨ ◦ ଐਓੑ͕ഉআ͞ΕͯνʔϜʹ৽͘͠ՃΘͬͨϝϯόʔͷΦϯϘʔ σΟϯά͕εϜʔζʹͳͬͨ
47. 10. ίϯςφνΣοΫϦετͷ࡞੒
48. 10. ίϯςφνΣοΫϦετͷ࡞੒ ~΍ͬͨ͜ͱ~ • ଞνʔϜͰ΋ίϯςφӡ༻΁ͷҠߦ͕૿͖͑ͯͨ • ͢΂ͯͷνʔϜʹίϯςφӡ༻ʹৄ͍͠ΤϯδχΞ͕͍ΔΘ͚Ͱ͸ ͳ͍ • ίϯςφӡ༻ͷUnknown unknownsΛͳͨ͘͢ΊɺଞνʔϜͷΤϯ δχΞͱڠྗͯ͠ίϯςφӡ༻ͷ஌ݟΛνΣοΫϦετͱͯ͠·ͱ Ίͨ
49. 10. ίϯςφνΣοΫϦετͷ࡞੒ ~΍ͬͨ͜ͱ~
50. 10. ίϯςφνΣοΫϦετͷ࡞੒ ~͜ͷऔΓ૊ΈͰظ଴͢ΔมԽ~ • ৽نίϯςφӡ༻ʹ޲͚ͨॳಈָ͕ʹͳΔ • طଘͷίϯςφӡ༻ʹ͓͚Δߟྀ࿙Ε΍࣮ݱϨϕϧΛ֬ೝͰ͖Δ • ૊৫શମͰCloud NativeʹऔΓ૊Ήʹ͸ɺϊ΢ϋ΢΍ࣄྫΛࣾ಺Ͱڞ ༗͢Δ͜ͱ͕ॏཁ
51. νʔϜ΁༩͑ͨมԽͷ·ͱΊ
52. ଐਓੑͷഉআ • औΓ૊ΈΛਐΊΔ͜ͱͰ։ൃΤϯδχΞɺӡ༻ΤϯδχΞͦΕͧΕ ͷྖҬ͕ࣗવʹ͍ۙͮͯΦʔόʔϥοϓ͢Δ෦෼͕૿͖͑ͯͨ ◦ ίϯςφσβΠϯύλʔϯʹΑΔઃܭ΍Մ؍ଌੑͷͨΊͷϥΠϒ ϥϦͷ։ൃɺIaCͳͲɺίϯςφΛத৺ͱͨ͠Cloud Nativeπʔ ϧͰ͜ΕΛ࣮ݱ͠΍͘͢ͳ͍ͬͯΔ • ։ൃ͕εέʔϥϒϧʹͳ͖͍ͬͯͯΔ
53. ߏஙɾӡ༻ෛՙͷ௿ݮ • ϚωδϝϯταʔϏε΁ͷҠߦ΍Cloud Nativeπʔϧͷ׆༻Ͱεέʔ ϥϒϧͳγεςϜͷߏங͕؆୯ʹͳͬͨ • ίϯςφΛϕʔεͱͨ͠ճ෮ੑͷ͋ΔߴՄ༻ͳγεςϜ΍ϦϦʔε࡞ ۀͷվળʹΑΓɺӡ༻ෛՙ͕௿ݮͨ͠
54. Cloud NativeͱSREͷ૬ޓ࡞༻ • SLOΛఆٛ͢Δ͜ͱͰఆྔతͳ൑அͰ҆৺ͯ͠Cloud NativeΛਐΊΔ ͜ͱ͕Ͱ͖Δ • Cloud NativeΛਐΊΔ͜ͱͰՄ؍ଌੑ͕ߴ·ΓɺSLOͷఆٛΛ࢝Ί ͱͨ͠SREΛ͞Βʹਪਐ͠ɺνʔϜΛεέʔϥϒϧʹͰ͖Δ • ͦΕͧΕ͕࡞༻͠߹͍ɺੵۃతʹมԽ͍ͯͨ͘͠Ίͷ౔୆͕Ͱ͖ͭ ͭ͋Δ
55. ·ͱΊ
56. ·ͱΊ • Cloud Nativeͷ࣮ݱʹ͸2ͭͷΞϓϩʔν͕ඞཁ ◦ εέʔϥϒϧͳΞϓϦέʔγϣϯͷߏங ◦ εέʔϥϒϧͳΞϓϦέʔγϣϯͷӡ༻ • ྆ํΛ܁Γฦ͠ܧଓͯ͠औΓ૊Ή͜ͱͰνʔϜʹมԽ͕ੜ·ΕΔ • มԽ͕ੵΈॏͳΔ͜ͱͰCloud NativeͳνʔϜʹͳΔ
57. curl -sIL mackerel.io | grep career