Terraform, GitHub Actions, Cloud Buildでデータ基盤をProvisioningする / Data Platform provisioning for Google Cloud and Terraform

会社名は入ってますが100%個人の開発と趣味の話です.

Transcript

1. Terraform, GitHub Actions, Cloud BuildͰ σʔλج൫ͷProvisioningΛ΍ͬͯΈͨ࿩. αʔόϨεΞʔΩςΫνϟͰMicro ServiceԽͨ͠ੈքઢͷDevOpsతͳߟ࡯. Shinichi Nakagawa 2024/03/15 Jagu'e'r Cloud Native #13 ϋΠϒϦου Meetup
2. ໔੹ࣄ߲ • ͜ͷࣄྫ͸ࢲʢத઒৳Ұʣݸਓͷझຯ։ൃ͔Βੜ·Εͨߟ࡯Ͱ͢. • ॴଐ૊৫ɾاۀɾஂମΛ୅ද͢ΔൃݴɾҙݟͰ͸͍͟͝·ͤΜ. • ॴଐ૊৫ɾاۀ͓ΑͼͦͷεςʔΫϗϧμʔʹର͢Δ࣭໰ɾҙݟɾ໰ ͍߹Θ͓ͤΑͼۀ຿಺༰ʹ͸Ұ੾͓౴͑͢Δ͜ͱ͸Ͱ͖·ͤΜ. • εϥΠυ಺ͷը૾͸ݪଇʮ͍Β͢ͱ΍ʯΛར༻͍ͯ͠·͢. ※ʮ͍Β͢ͱ΍ʯҎ֎ͷը૾͸ग़యɾҾ༻ݩΛهࡌ͍ͯ͠·͢.
3. Who am I ? ʢ͓લ୭Α?ʣ • Shinichi Nakagawaʢத઒৳Ұʣ / @shinyorkeʢSNSʣ • ΞΫηϯνϡΞגࣜձࣾϚωδϟʔ • Ϋϥ΢υΤϯδχΞϦϯά/SREʢຊ৬ʣ • ʮGoogle Cloud Partner Top Engineer 2024ʯड৆ • AI, ML, Dataplatform, Baseballʢաڈͷ࢓ࣄ+झຯಛٕʣ • ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ ʮٕज़ʯʮ໺ٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢. https://shinyorke.hatenablog.com/
4. ຊ೔ͷ͓͠ͳ͕͖ʢ࿩͢͜ͱʣ • σʔλج൫ΛTerraformͰؙͬͱIaCʢInfrastructure as Codeʣ. • GitHub ActionsͰGoogle CloudΛDevOpsͯ͠Έͨ. Cloud RunͰ࡞ͬͨMicro ServiceΛ޻෉ͯ͠IaC&DevOpsͨ͠࿩Ͱ͢. ※͜ͷࣄྫ͸ۀ຿Ͱ͸ͳ͘झຯͰ͢ʢ͕࣮຿Ͱ΋࢖͑Δ͸ͣʣ.
5. ໺ٿσʔλ෼ੳج൫ ※ݸਓ։ൃͷ࿩Ͱ͢ • झຯͷ໺ٿΛຊ֨తʹσʔλͰݟΔͨΊͷج൫ΛຊؾͰ ࡞͍ͬͯ·͢ʢٕज़తͳݕূΛ݉Ͷͯʣ. • ಛʹϝδϟʔϦʔά͸ʮBaseball Savantʯͱ͍͏ެࣜͷ σʔλαΠτ͕͋Γ, ϑΝϯͷզʑͰ΋CSVܗࣜͰσʔλ ͕खʹೖΓ·͢&σʔλͷத਎͕݁ߏ΍΍͍͜͠Ͱ͢. • ͦͷʮ΍΍͍͜͠CSVσʔλʯΛऔΓࠐΈ, ෼ੳɾղੳΛ ߦ͏ϝδϟʔϦʔά෼ੳΞϓϦΛ2022೥͔Β։ൃɾӡ༻. ຖ೥ϦΞʔΩςΫνϟʢ࡞Γ௚͠ʣ͍ͯ͠·͢, ࠓ೥͸ Cloud Runͷ৽ػೳʢCloud Run JobsʣͰ࡞Γ௚͠. ෼ੳج൫ͷΞϓϦέʔγϣϯʢ2022೥൛ʣ ©Shinichi Nakagawa ©Shinichi Nakagawa
6. ໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ ©Shinichi Nakagawa
7. σʔλج൫ͷ֤ػೳͰ࢖ͬͯΔϞϊͱղઆ .JDSP
   4FSWJDF
   PS
   ໾ׂ (PPHMF
   $MPVE
   4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE
   4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ
   ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE
   3VO
   +PCT &YQPSUFS
   
   *NQPSUFSΛ࣮ߦ
   ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷
   -VJHJ౳
   ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE
   3VO #BTFCBMM
   4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %BUBMBLF $MPVE
   4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ
   ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE
   3VO $47σʔλΛνΣοΫ͠
   #JH2VFSZʹ&YQPSU (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀ
   PS
   ΞϓϦ͔Β࢖͏
   %BUB
   8BSF
   )PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻
8. .JDSP
   4FSWJDF
   PS
   ໾ׂ (PPHMF
   $MPVE
   4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE
   4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ
   ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE
   3VO
   +PCT &YQPSUFS
   
   *NQPSUFSΛ࣮ߦ
   ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷
   -VJHJ౳
   ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE
   3VO #BTFCBMM
   4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %BUBMBLF $MPVE
   4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ
   ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE
   3VO $47σʔλΛνΣοΫ͠
   #JH2VFSZʹ&YQPSU (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀ
   PS
   ΞϓϦ͔Β࢖͏
   %BUB
   8BSF
   )PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻ σʔλج൫ͷ֤ػೳͰ࢖ͬͯΔϞϊͱղઆ શ෦खಈͰઃఆ͢Δͱഁ୼͢Δະདྷ͕଴͍ͬͯ·͢ʂ
9. Micro ServiceʹTerraform͸ඞཁʢॾઆ༗Γʣ. • ෳ਺ͷΞϓϦέʔγϣϯΛ਺चͭͳ͗ʢϐλΰϥεΠονʣ͢Δ৔߹, TerraformͰͷIaC͸΍͓ͬͯ͜͏, খ͍͞γεςϜͰ΋. • ࠓճͷߏ੒Ͱ͸·͋·͋ͷ਺ͷGoogle CloudͷServiceΛར༻. • Cloud RunΞϓϦ͕3ݸ + Volume, LoggingͳͲͷݻ༗ઃఆ • Pub/Sub TopicͱSchema͕ͦΕͧΕ2ݸ • Cloud Schedulerͷઃఆ • Cloud StorageͷBucketͱΞΫηε੍ޚ • ͜ΕΒͱ৭ʑඥͮ͘ઃఆ, ಛʹService Account͓ΑͼIAM←͜Ε͕࠷΋ۂऀʢηΩϡϦςΟతʹ΋ʣ • Ҏલ͸Cloud Console or GUIͰ࡞͍͕ͬͯͨഁ୼͢Δʢྫ: ਎ʹ͕֮͑ແ͍Service Account, InstanceͳͲʣͷͰࠓ೥͔ΒIaCԽ. ઃܭͱ࣮૷ͷॳظίετʢओʹ࣌ؒͱΩϟονΞοϓʣ͸͋Δ͕, ౤ࢿͨ͠΄͏͕ઈରʹྑ͍ʢͱ࣮ײ͍ͯ͠·͢ʣ.
10. Google CloudͰTerraformΛ࢝ΊΔ࣌͸ • Quick StartͳͲ, ެࣜυΩϡϝϯτʹsnippet͕͋ΔͷͰ࢖͓͏ʂ • gcloud command΍GUIઃఆͱͷฒͼʹTerraformͷsnippet͕ଘࡏ. • ࢲ͸ެࣜυΩϡϝϯτͷTerraformͷsnippetΛmain.tfʹషΓ෇͚, ಈ࡞ ֬ೝ͠ͳ͕Βຊ൪༻ͷ࣮૷Λߦͬͨʢ͘͢͝ḿͬͨʣ. શͯͷυΩϡϝϯτʹ͋Δ༁Ͱ͸ແͦ͞͏͕ͩੵۃతʹ࢖͓͏.
11. ʁʁʁʮ͍͍ײ͡ʹࣗಈԽͱDevOps͍ͨ͠ΜͰ͕͢Ͳ͏͢Ε͹🤔ʯ Enterprise޲͚Ͱ΋࠷ۙ͋Δͱࢥ͏͜ͷཁ๬ʹͲ͏Ԡ͑Δ͔!?
12. ʲ౴ʳGitHub Actions͔ΒTerraformΛ࢖ͬͨ • ʮ໎ͬͨΒGitHub Actionsʯͱࢥͬͯ΍ͬͨΒҙ֎ͱ্ख͍͖͘·ͨ͠. • ͬ͘͟Γॻ͘ͱҎԼͷखॱͰ࣮ݱ. 1. Workload Identityͷ४උ. 2. CDʢܧଓతσϓϩΠʣͱ࣮ͯ͠ߦ͢Δର৅ͷܾఆ. 3. λεΫͷॱ൪ΛܾΊΔ. • Cloud RunΞϓϦέʔγϣϯͷ৔߹, ͜ͷ۠෼͚Ͱ͏·͘ߦͬͨ. • Workload Identity͓ΑͼΞϓϦͷService AccountʢSAʣൃߦ͸GitHub ActionsͰ͸ͳ͘TerraformͰखಈ࣮ߦ. • SAͱ֤छϦιʔεͷBinding, ͦͷଞͷϦιʔεͷCD͸GitHub ActionsͰ࣮ߦ. • ArtifactʢDocker imageʣੜ੒ -> DeployΛઈରతͳॱংͱͨ͠GitHub Actions Work fl owͷઃܭͱ࣮૷. ※͕͜͜ॏཁ
13. ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
14. ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
15. ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
16. ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
17. ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa
18. GitHub Actions + Terraform, Cloud Buildͷ޻෉ • CI/CDεςʔδͷ໾ׂΛ໌֬Խ. • ʢ౰ͨΓલͰ͕͢ʣςετ͸ຖճࣗಈ࣮ߦ. • Pull Request࣌ʹImage Buildͱterraform planͷ֬ೝ. • Deployʢterraform applyʣ͸main΁ͷMergeͷΈ. • GitHub ActionsͰCI/CDͷϑϩʔΛ੍ޚ. • ͢΂ͯͷΞϓϦέʔγϣϯͷArtifactΛCloud BuildͰ࡞੒ޙʹterraform planΛݩʹϨϏϡʔ. • GitHub ActionsͷneedsઃఆͱPull RequestͷonΠϕϯτͰ͍͍ײ͡ʹ੍ޚ. • ϨϏϡʔ༻ʹterraform planͷ݁ՌΛPull Requestͷίϝϯτʹ৐ͤΔ.
19. ݁ͼ • ෳࡶʹͳΓ͕ͪͳMicro Service͸IaCԽ͠·͠ΐ͏, ๨ΕΔͱେม. • GitHub ActionsͱCloud BuildͰTerraform͸ͻͱ޻෉͕ඞཁ. ॳखͰIaCʢTerraformʣΛೖΕΔۤ࿑͸͋Γ·͕͢େมʹॏཁ. ͍͖ͳΓ࢓ࣄͰ΍ΔͱେมͳͷͰԿ͔͠Β࿅शΛ͠·͠ΐ͏.
20. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠. Shinichi Nakagawa ©Shinichi Nakagawa