Dev D Dev E αʔϏεC αʔϏ εD Dev F αʔϏεE Dev G Dev H αʔϏ εG αʔϏ εH αʔϏ εF SRE w ਓ w ͯ͢ͷϓϩμΫταʔϏεͷΠϯϑϥΛࢧ͑ΔԣஅతͳνʔϜ w αʔϏεͷՁΛϢʔβʔʹಧ͚ΔͨΊʹɺ҆ఆͨ͠ΠϯϑϥΛ ఏڙ͠ଓ͚Δͷ͕ϛογϣϯ GSFFFͷ43&νʔϜ GSFFFʹ͍ͭͯ
w -#Ճ w "VUP4DBMJOH(SPVQ Ճ w σϓϩΠڥඋ w %#Ճ w 3PVUFొ w ηΩϡϦςΟ֬อ w *".ϩʔϧՃ SG ALB SG Kubernetes SG RDS AutoScalingGroup Developers w ΞϓϦέʔγϣϯ։ൃ w ΞϓϦέʔγϣϯσϓ ϩΠ
ALB SG SG RDS C SG ALB SG SG RDS D SG ALB SG SG RDS E SG ALB SG SG RDS ECR ίϯςφΛ,VCFSOFUFTͰಈ͔͢ namespace namespace namespace namespace namespace pod pod pod pod pod Manifests Manifests ΞϓϦέʔγϣϯͷߏ͕ίʔυԽ͞ΕΔ
ALB SG SG RDS C SG ALB SG SG RDS D SG ALB SG SG RDS E SG ALB SG SG RDS એݴతʹ"84ϦιʔεΛ֬อ namespace namespace namespace namespace namespace pod pod pod pod pod SRE w ωοτϫʔΫඋ w -#Ճ w "VUP4DBMJOH(SPVQ Ճ w σϓϩΠڥඋ w %#Ճ w 3PVUFొ w ηΩϡϦςΟ֬อ w *".ϩʔϧՃ
ALB SG SG RDS C SG ALB SG SG RDS D SG ALB SG SG RDS E SG ALB SG SG RDS એݴతʹ"84ϦιʔεΛ֬อ namespace namespace namespace namespace namespace pod pod pod pod pod Manifests TF Files "84Ϧιʔε͕ίʔυԽ͞ΕΔ
Service A-3 Product B Service B-1 ServiceB-2 Service B-3 Product C Service C-1 Service C-2 Service C-3 K8s cluster Product A Service A-1 Service A-2 Service A-3 K8s cluster Product B Service B-1 Service B-2 Service B-3 K8s cluster Product C Service C-1 Service C-2 Service C-3 ϓϩμΫτ ͍ͨ͠ݖ ݶ ୯ҐͰׂͨ͠γϯά ϧςφϯτ γϯάϧςφϯτͰݖݶΛͯ͠Ϋϥελͷӡ༻Λ͓·͔ͤ͢Δ ͯ͢ͷϓϩμΫτ͕ಈ͍͍ͯΔ Ϛϧνςφϯτ
Product B Service B-1 ServiceB-2 Service B-3 Product C Service C-1 Service C-2 Service C-3 K8sͷόά Φϖϛε શαʔϏεμϯͷةݥ w #MBTUSBEJVT োͷӨڹൣғ ͕େ͖͍ w ӡ༻ͷқ͕ߴ͍ w νϟϨϯδͮ͠Β͍ۭؾ ϚϧνςφϯτͷϦεΫ 56 γϯάϧςφϯτͰݖݶΛͯ͠Ϋϥελͷӡ༻Λ͓·͔ͤ͢Δ
Service A-3 K8s cluster Product B Service B-1 Service B-2 Service B-3 K8s cluster Product C Service C-1 Service C-2 Service C-3 K8sͷόά Φϖϛε Ұ෦ͷΈαʔϏεμϯ w #MBTUSBEJVT োͷӨڹൣғ ͕খ͍͞ w ӡ༻ͷқԼ͕Δ w νϟϨϯδ͍ۭ͢͠ؾ w ৺ཧత҆શੑ͕ߴ͍ γϯάϧςφϯτͰݖݶΛͯ͠Ϋϥελͷӡ༻Λ͓·͔ͤ͢Δ
node Service A-1 Service B-2 Service B-3 Kubernetes node Kubernetes node Service B-1 Service A-2 Service A-3 SG SG 4FDVSJUZ(SPVQʹΑΔׂෆՄ *".ͱ,JBNͰ"84Ϧιʔεͷ੍ޚՄೳ 3#"$Ͱ/BNFTQBDFؒͷΞΫηε੍ޚՄೳ ͨͩ͠ϓϩμΫτؒͰ7.ڞ௨ ˣ ϓϩμΫτ୯ҐͰ/PEF(SPVQΛׂ͢Εର ԠՄೳ͕ͩɺͦͷͨΊͷΈͮ͘Γ͕ඞཁ γϯάϧςφϯτͰݖݶΛͯ͠Ϋϥελͷӡ༻Λ͓·͔ͤ͢Δ
Service A-1 Service B-2 Service B-3 Kubernetes node Kubernetes node Service B-1 Service A-2 Service A-3 SG SG ςφϯτ͍ͨ͠ݖݶ୯Ґʹͳ͍ͬͯΔ 4FDVSJUZ(SPVQ͕ར༻Մೳ 3#"$Λซ༻ 7.ϨϕϧͰׂ͞Ε͍ͯΔ ˣ ࠓ·Ͱӡ༻͖ͯͨ͠ ރΕͨ ߏͱ ҰॹͳͷͰѻ͍͕؆୯ γϯάϧςφϯτͳΒڥքͷ໌֬Խ༰қ 61 γϯάϧςφϯτͰݖݶΛͯ͠Ϋϥελͷӡ༻Λ͓·͔ͤ͢Δ
Service A-3 Product B Service B-1 ServiceB-2 Service B-3 Product C Service C-1 Service C-2 Service C-3 Developers A Developers B Developers C ӡ ༻ ӡ ༻ ӡ ༻ ϚϧνςφϯτΫϥελશମʹؔ͢Δ Ξοϓσʔτ࡞ۀ͕ͮ͠Β͍ ڞ௨෦ Product A SRE ΫϥελͷΞοϓ άϨʔυͳͲ w αʔϏεΛͯ͢ఀࢭͤ͞Δඞཁ͕͋Δ w ΞοϓάϨʔυʹࣦഊ͢ΔՄೳੑ͕͋Δ w ࣦഊͨ͠ͱ͖ͷϩʔϧόοΫͷίετ͕ߴ͍
ӡ ༻ ӡ ༻ γϯάϧςφϯτΫϥελશମʹؔ͢Δ Ξοϓσʔτ࡞ۀ͕͍͢͠ SRE ΫϥελͷΞοϓ άϨʔυͳͲ w αʔϏεͷఀࢭ࠷খݶ w ΞοϓάϨʔυʹࣦഊͯ͠࠷খݶ w ࣦഊͨ͠ͱ͖ͷϩʔϧόοΫͷίετ࠷খݶ K8s cluster Product A Service A-1 Service A-2 Service A-3 K8s cluster Product B Service B-1 Service B-2 Service B-3 K8s cluster Product C Service C-1 Service C-2 Service C-3
B SG SG SG Kubernetes node applications ,VCFSOFUFTʹͲ͜·ͰͤΔʁ w "QQMJDBUJPO w %BUBCBTF w -PBE#BMBODFS w 4FDVSJUZ w "VUI &,4ΛϚωʔδυαʔϏεͱΈ߹ΘͤͯΫϥελͷӡ༻ίετΛ͑Δ ALB RDS ALB RDS
Kubernetes node Kubernetes node Service B-1 Service B-2 Service B-3 Team B helmfile sync PR Commands SRE Review/Approve environments: production: values: - production.yaml releases: - name: kube-state-metrics namespace: kube-system chart: stable/kube-state-metrics version: 0.13.0 - name: metricbeat namespace: kube-system chart: stable/metricbeat version: 1.2.1 values: - values.yaml.gotmpl w )FMN )FMN$IBSU w 5IF,VCFSOFUFT1BDLBHF .BOBHFS w ϚχϑΣετΛύοέʔδԽ w Α͋͘ΔπʔϧͷςϯϓϨ w )FMNGJMF w )FMN$IBSUͷґଘؔΛϑΝ ΠϧͰϑΝΠϧͰఆٛ w IFMNGJMFTZOD w IFMNGJMFEJGG w IFMNGJMFEFMFUF Ϛϧνςφϯτ͔Βγϯάϧςφϯτͳ&,4ʹҠߦ࣮ͨ͠ྫ
SG SG SG Kubernetes node Kubernetes node Service B-1 Service B-2 Service B-3 SG Kubernetes node Kubernetes node Service B-1 Service B-2 Service B-3 Kube-aws্ͷϓϩμΫτ (࣮ࡍϚϧνςφϯτ) EKS্ͷϓϩμΫτ Weighted Routing 80% 20% w ಉ͡ߏͷΫϥελΛ༻ҙ w "84Ϧιʔεڞ༗Ͱ͖Δͷڞ ༗͢Δ %#ඞਢ w 3PVUFͷ8FJHIUFE3PVUJOHΛར ༻ͯ͠ঃʑʹϦΫΤετΛྲྀ͠ࠐΉ w αʔϏεʹΑͬͯ3PVUFͰͳ ͘-#Λڞ௨Խͯ͠ɺ,VCFSOFUFT OPEFΛࠩ͠ସ͑Δํ๏Λ༻ w ϊʔϝϯςͰΓସ͑