Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Terraform管理下のマネージドリソースとk8sリソースを一元的にGitOpsするまでの試行錯誤
Search
uya116
February 28, 2023
Technology
1
600
Terraform管理下のマネージドリソースとk8sリソースを一元的にGitOpsするまでの試行錯誤
2023/2/27 CI/CD Conference 2023 前夜祭で登壇した資料です。
https://cloudnativedays.connpass.com/event/274402/
uya116
February 28, 2023
Tweet
Share
More Decks by uya116
See All by uya116
競プロのすすめ
uya116
0
950
Other Decks in Technology
See All in Technology
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
3
600
2024春 注目のWeb系 OSS & SaaS 3選
makies
0
170
Handling focus in 2024
tahia910
0
120
Android Target SDK 35 (Android 15) 対応の概要
akkie76
0
130
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
140
Cracking the KubeCon CfP
inductor
2
260
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
3
560
One engineer company with Ruby on Rails
rstankov
2
390
FrontDoorとWebAppsを組み合わせた際のリダイレクト処理の注意点
kenichirokimura
1
690
Building Dashboards as a Hobby
egmc
0
330
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
250
Featured
See All Featured
The Language of Interfaces
destraynor
151
23k
Building a Modern Day E-commerce SEO Strategy
aleyda
20
6.4k
Design by the Numbers
sachag
274
18k
Rails Girls Zürich Keynote
gr2m
91
13k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Rebuilding a faster, lazier Slack
samanthasiow
74
8.2k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
19
6.9k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
21
1.9k
Principles of Awesome APIs and How to Build Them.
keavy
121
16k
How GitHub (no longer) Works
holman
305
140k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
26
2.3k
Transcript
Terraform ཧԼͷϚωʔδυϦιʔεͱ k8s ϦιʔεΛҰݩతʹ GitOps ͢Δ·Ͱͷ ࢼߦࡨޡ 2023/02/27 ιϑτόϯΫגࣜձࣾ ΫϥυΤϯδχΞϦϯάຊ෦
Cloud Native Days CI/CD Conference 2023 લࡇ ࡾվ ༟
ࡾվ ༟ (Mizorogi Yuya) 2 ࣗݾհ ॴଐ ιϑτόϯΫגࣜձࣾ ๏ਓࣄۀ౷ׅ
ΫϥυΤϯδχΞϦϯάຊ෦ ɾ2021ʹத్ೖࣾ ɾ๏ਓ͚ͷΞϓϦέʔγϣϯج൫ͷ։ൃΛ୲ ɾझຯย͚ͱڝϓϩ @_uya116
• ΞϓϦέʔγϣϯΤϯδχΞ͕ Kubernetes Λҙࣝ͠ͳͯ͘ྑ͍Έ 3 ιϑτόϯΫͰ͜ΜͳαʔϏεΛ࡞͍ͬͯ·͢ ϖʔδͪ͜Β
• Ϛωʔδυ Kubernetes ڥʹࣄલఆٛࡁͷ Helm ύοέʔδΛద༻ 4 ιϑτόϯΫͰ͜ΜͳαʔϏεΛ࡞͍ͬͯ·͢ DEPLOYMENT STORAGE
RDB VAULT MONITORING IAM ࣄલఆٛࡁύοέʔδ MESSAGING ͳͲ
• ϚωʔδυϦιʔεͱ Kubernetes Ϧιʔεͷํͷཧ͕ඞཁ ◦ ύϒϦοΫΫϥυͷϚωʔδυ Kubernetes Λ͏ͱͳ͓͞Β 5 എܠ
• IaC ܗࣜͱͦΕʹ͏ CD αʔϏε͕ҟͳΓύΠϓϥΠϯ͕͔Εͯ͠·͏ ◦ ͜ΕʹΑͬͯཧ͕ࡶʹͳΔ 6 ೋछྨͷϦιʔεΛཧ͢Δ͜ͱʹΑΔGitOpsͷ՝ Github
Actions Jenkins AWS Code γϦʔζ Ϧιʔε CD αʔϏε Git Terraform CloudFormation Kubernetes ϚχϑΣετ ͳͲ ͳͲ ͳͲ IaC ܗࣜ ϚωʔδυϦιʔε Kubernetes Ϧιʔε
• IaC ܗࣜ or CD αʔϏεΛἧ͑Δ͜ͱʹΑΓҰݩԽͰ͖ͳ͍͔ݕ౼ 7 ݕ౼ Github Actions
Jenkins AWS CodePipeline ͳͲ ͳͲ ͳͲ Ϧιʔε CD αʔϏε Git IaC ܗࣜ ϚωʔδυϦιʔε Kubernetes Ϧιʔε Terraform CloudFormation Kubernetes ϚχϑΣετ
• IaC ܗࣜΛἧ͑Δ߹ ◦ Terraform k8s provider ͔֤ϕϯμʔͷఏڙ͢Δ k8s controller
͕ީิ ◦ ύΠϓϥΠϯ͕؆ܿʹͳΔ͕ରԠϦιʔε͕ݶఆ͞Ε͍ͯΔ 8 ݕ౼ ʙIaC ܗࣜΛἧ͑Δʙ Github Actions Jenkins AWS CodePipeline ͳͲ ͳͲ ͳͲ Ϛωʔδυ Ϧιʔε ʹدͤΔ ᶃ Terraform k8s provider Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK controller ϚωʔδυϦιʔε Kubernetes Ϧιʔε IaC ܗࣜ Terraform CloudFormation Kubernetes ϚχϑΣετ
• CD αʔϏεΛἧ͑ͨ߹ ◦ ύΠϓϥΠϯద༻ڥͰίϚϯυΛ࣮ߦ͠ڧҾʹϦιʔεΛσϓϩΠ͢Δ ◦ ίϚϯυ࣮ߦͷͨΊࣗ༝ߴ͍͕ύΠϓϥΠϯ͕ෳࡶʹͳΔ 9 ݕ౼ ʙCD
αʔϏεΛἧ͑Δʙ Github Actions Jenkins AWS CodePipeline ͳͲ ͳͲ Ϛωʔδυ Ϧιʔε ʹدͤΔ ᶅ k8s ίϯςΩετΛऔಘͯ͠ apply Kubernetes Ϧιʔε ʹدͤΔ ᶆ k8s Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ apply ϚωʔδυϦιʔε Kubernetes Ϧιʔε CD αʔϏε
10 ݕ౼ ʙ֤ํ๏ͷൺֱʙ IaC ܗࣜΛἧ͑Δ CD αʔϏεΛἧ͑Δ Ϛωʔδυ Ϧιʔε ʹدͤΔ
ᶃ Terraform k8s provider ᶅ k8s ίϯςΩετΛऔಘͯ͠ apply Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK ᶆ k8s Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ controller apply ෳࡶʹͳΔ͕ ࣗ༝͕ߴ͍ Ϧιʔε੍ݶ͕ ͋Δ͕؆ܿ ͲͪΒ͕ ϝΠϯ͔
• ฐαʔϏε Kubernetes Ϧιʔεத৺ͷͨΊᶄᶆΛ࠾༻ ◦ ֤ϕϯμͷ k8s controller ʹରԠ͍ͯ͠ΔϚωʔδυϦιʔε →
ᶄͰ࡞ ◦ ະରԠ͘͠ k8s controller Ͱͷಈ࡞͕ෆ҆ఆͳϦιʔε → ᶆͰ࡞ 11 ݕ౼݁Ռ IaC ܗࣜΛἧ͑Δ CD αʔϏεΛἧ͑Δ Kubernetes Ϧιʔε ʹدͤΔ ᶄ ConfigConnector / ASO / ACK ᶆ k8s Ͱ Ϛωʔδυ༻ͷ IaC Λద༻ controller
• ݕ౼݁Ռʹैͬͯ GitOps ύΠϓϥΠϯΛߏங • Job Ͱ terraform ίϚϯυΛ࣮ߦ͢Δ͕࣮ͷෛՙ͕ߴ͍ ◦
Helm আ࣌ʹ Job Λ࣮ߦ͢Δ͕ terraform destroy ʹࣦഊ͢Δͱ Job ͕ࣦഊ͠ Helm ͕ফͤͳ͘ͳΔ → ঢ়ଶʹԠͨ͡ίϚϯυͷ੍ޚ͕ඞཁ ◦ drift ൃੜ࣌ͷ੍ޚ ࢪࡦ ʙฐαʔϏεͷ GitOps ύΠϓϥΠϯʙ vender controller 12 ϕϯμͷ k8s controller ࣗ࡞ Job Ͱ terraform ࣮ߦ ※ ฐαʔϏεͷ্ཱ͚ͯ Helm Ͱύοέʔδϯά͍ͯ͠Δ
• Terraform ϦιʔεΛཧ͢Δ Kubernetes Controller ◦ Flux ͱ࿈ܞͯ͠ Terraform ͷ
GitOps Λ࣮ݱ͢Δ ◦ TF state ϑΝΠϧͷΫϥυετϨʔδཧɺOIDC ࿈ܞͳͲඞཁͳػೳ͕ἧ͍ͬͯΔ 13 Weave GitOps Terraform Controller https://weaveworks.github.io/tf-controller/ Terraform Controller kind: Terraform ᶃݕ ᶅ࡞ ᶄ TF ϑΝΠϧऔಘ
Weave GitOps Terraform Controller “ࣗͷϖʔε”Ͱ GitOps ͱ͍͏ίϯηϓτΛܝ͓͛ͯΓ ϚχϑΣετʹ߹Θͤͨ Terraform
apply / destroy ͷࣗಈద༻͚ͩͰͳ͘ drift ͷݕग़ͷΈߦ͏͜ͱՄೳ TF ϑΝΠϧͷ֨ೲݩɻflux ͷ GitRepository / OCIRepository Λࢦఆ TF ϑΝΠϧʹΘͨ͢ڥมͷઃఆ terraform ίϚϯυΛ࣮ߦ͢Δ ServiceAccount Terraform CR Λআͨ͠ͱ͖ʹΫϥυ্ͷϦιʔεফ͔͢ backend ͷઃఆ
σϞ
• Weave GitOps Terraform Controller Λ༻͍Δ͜ͱͰҰݩ GitOps Λ࣮ݱ 16 ࢪࡦ
ʙฐαʔϏεͷ GitOps ύΠϓϥΠϯʢAfterʣʙ vender controller ϕϯμͷ k8s controller ͰϦιʔε࡞ Terraform Controller Ͱ terraform ࣮ߦ terraform controller
• Flux v0.32.0 ͰରԠͨ͠ OCI ϦϙδτϦ ʹ TF ϑΝΠϧΛ֨ೲ͍ͯ͠Δ ◦
͜ΕʹΑΓ Docker image, Helm Chart, TF ϑΝΠϧΛಉҰαʔϏεͰཧ͢Δ͜ͱ͕Մೳ ◦ ೝূํ๏ڞ௨ԽͰ͖Δ Weave GitOps Terraform Controller Ͱͷ 17 AKS ฐαʔϏεͷ୲ൣғ OCI ϦϙδτϦ (Artifact Registry) GKE EKS
• Artifact Registry ʹ֤ k8s ͔ΒΞΫηε͢Δඞཁ͕͋Δ ◦ ظݶ͕͍ΫϨσϯγϟϧใ࣋ͪͨ͘ͳ͍ͨΊ OIDC ࿈ܞ͍ͨ͠
◦ ͔͠͠ݱঢ় OCI ϦϙδτϦʹର͢Δ Flux source-controller Ͱ GC ͱͷ OIDC ࿈ܞ͕ະରԠ 18 Weave GitOps Terraform Controller Ͱͷ ✕ AWSͰݖݶҕ͞ΕͨτʔΫϯΛ༻͍ͯ GC ͱ࿈ܞ͍͕ͨ͠ɾɾɾ Flux
• OIDC ࿈ܞͨ͠ CronJob Ͱ imagePullSecret Λ࡞͢Δ ◦ imagePullSecret ͷߋ৽Λ
OIDC ࿈ܞͨ͠ CronJob Ͱఆظతʹ࣮ࢪ͢Δ͜ͱͰ՝Λճආ ◦ ֤Ϛωʔδυ k8s ͔Β Artifact Registry ͷΞΫηε͕Մೳͱͳͬͨ 19 Weave GitOps Terraform Controller Ͱͷ ˕ Flux ✕ Flux Before After
• GitOps Λڞ௨Խ͢Δͱ࣍ςετڞ௨Խͨ͘͠ͳΔɾɾɾ 20 ςετʹ͍ͭͯ ςετର vender controller terraform controller
• νʔϜϝϯόʔ͕ CI/CD ΧϯϑΝϨϯεຊฤͰ͠·͢ʂ 21 ςετʹ͍ͭͯ
22 ·ͱΊ 1. Weave GitOps Terraform Controller Λ༻͍Δ͜ͱͰ GitOps ͷҰݩԽΛ࣮ݱ
2. TF ϑΝΠϧ OCI ϦϙδτϦͰཧ͠ imagePullSecret ൃߦʹΑΓΞΫηε 3. ςετͷڞ௨Խʹ͍ͭͯຊฤΛָ͓͠Έʹ vender controller terraform controller OCI ϦϙδτϦ