Terraform, GitHub Actions, Cloud Buildでデータ基盤をProvisioningする / Data Platform provisioning for Google Cloud and Terraform

Slide 1

Slide 1 text

Terraform, GitHub Actions, Cloud BuildͰ σʔλج൫ͷProvisioningΛ΍ͬͯΈͨ࿩. αʔόϨεΞʔΩςΫνϟͰMicro ServiceԽͨ͠ੈքઢͷDevOpsతͳߟ࡯. Shinichi Nakagawa 2024/03/15 Jagu'e'r Cloud Native #13 ϋΠϒϦου Meetup

Slide 2

Slide 2 text

໔੹ࣄ߲ • ͜ͷࣄྫ͸ࢲʢத઒৳Ұʣݸਓͷझຯ։ൃ͔Βੜ·Εͨߟ࡯Ͱ͢. • ॴଐ૊৫ɾاۀɾஂମΛ୅ද͢ΔൃݴɾҙݟͰ͸͍͟͝·ͤΜ. • ॴଐ૊৫ɾاۀ͓ΑͼͦͷεςʔΫϗϧμʔʹର͢Δ࣭໰ɾҙݟɾ໰ ͍߹Θ͓ͤΑͼۀ຿಺༰ʹ͸Ұ੾͓౴͑͢Δ͜ͱ͸Ͱ͖·ͤΜ. • εϥΠυ಺ͷը૾͸ݪଇʮ͍Β͢ͱ΍ʯΛར༻͍ͯ͠·͢. ※ʮ͍Β͢ͱ΍ʯҎ֎ͷը૾͸ग़యɾҾ༻ݩΛهࡌ͍ͯ͠·͢.

Slide 3

Slide 3 text

Who am I ? ʢ͓લ୭Α?ʣ • Shinichi Nakagawaʢத઒৳Ұʣ / @shinyorkeʢSNSʣ • ΞΫηϯνϡΞגࣜձࣾϚωδϟʔ • Ϋϥ΢υΤϯδχΞϦϯά/SREʢຊ৬ʣ • ʮGoogle Cloud Partner Top Engineer 2024ʯड৆ • AI, ML, Dataplatform, Baseballʢաڈͷ࢓ࣄ+झຯಛٕʣ • ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ ʮٕज़ʯʮ໺ٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢. https://shinyorke.hatenablog.com/

Slide 4

Slide 4 text

ຊ೔ͷ͓͠ͳ͕͖ʢ࿩͢͜ͱʣ • σʔλج൫ΛTerraformͰؙͬͱIaCʢInfrastructure as Codeʣ. • GitHub ActionsͰGoogle CloudΛDevOpsͯ͠Έͨ. Cloud RunͰ࡞ͬͨMicro ServiceΛ޻෉ͯ͠IaC&DevOpsͨ͠࿩Ͱ͢. ※͜ͷࣄྫ͸ۀ຿Ͱ͸ͳ͘झຯͰ͢ʢ͕࣮຿Ͱ΋࢖͑Δ͸ͣʣ.

Slide 5

Slide 5 text

໺ٿσʔλ෼ੳج൫ ※ݸਓ։ൃͷ࿩Ͱ͢ • झຯͷ໺ٿΛຊ֨తʹσʔλͰݟΔͨΊͷج൫ΛຊؾͰ ࡞͍ͬͯ·͢ʢٕज़తͳݕূΛ݉Ͷͯʣ. • ಛʹϝδϟʔϦʔά͸ʮBaseball Savantʯͱ͍͏ެࣜͷ σʔλαΠτ͕͋Γ, ϑΝϯͷզʑͰ΋CSVܗࣜͰσʔλ ͕खʹೖΓ·͢&σʔλͷத਎͕݁ߏ΍΍͍͜͠Ͱ͢. • ͦͷʮ΍΍͍͜͠CSVσʔλʯΛऔΓࠐΈ, ෼ੳɾղੳΛ ߦ͏ϝδϟʔϦʔά෼ੳΞϓϦΛ2022೥͔Β։ൃɾӡ༻. ຖ೥ϦΞʔΩςΫνϟʢ࡞Γ௚͠ʣ͍ͯ͠·͢, ࠓ೥͸ Cloud Runͷ৽ػೳʢCloud Run JobsʣͰ࡞Γ௚͠. ෼ੳج൫ͷΞϓϦέʔγϣϯʢ2022೥൛ʣ ©Shinichi Nakagawa ©Shinichi Nakagawa

Slide 6

Slide 6 text

໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ ©Shinichi Nakagawa

Slide 7

Slide 7 text

σʔλج൫ͷ֤ػೳͰ࢖ͬͯΔϞϊͱղઆ .JDSP4FSWJDFPS໾ׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷ -VJHJ౳ ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀPSΞϓϦ͔Β࢖͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻

Slide 8

Slide 8 text

.JDSP4FSWJDFPS໾ׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷ -VJHJ౳ ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀPSΞϓϦ͔Β࢖͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻ σʔλج൫ͷ֤ػೳͰ࢖ͬͯΔϞϊͱղઆ શ෦खಈͰઃఆ͢Δͱഁ୼͢Δະདྷ͕଴͍ͬͯ·͢ʂ

Slide 9

Slide 9 text

Micro ServiceʹTerraform͸ඞཁʢॾઆ༗Γʣ. • ෳ਺ͷΞϓϦέʔγϣϯΛ਺चͭͳ͗ʢϐλΰϥεΠονʣ͢Δ৔߹, TerraformͰͷIaC͸΍͓ͬͯ͜͏, খ͍͞γεςϜͰ΋. • ࠓճͷߏ੒Ͱ͸·͋·͋ͷ਺ͷGoogle CloudͷServiceΛར༻. • Cloud RunΞϓϦ͕3ݸ + Volume, LoggingͳͲͷݻ༗ઃఆ • Pub/Sub TopicͱSchema͕ͦΕͧΕ2ݸ • Cloud Schedulerͷઃఆ • Cloud StorageͷBucketͱΞΫηε੍ޚ • ͜ΕΒͱ৭ʑඥͮ͘ઃఆ, ಛʹService Account͓ΑͼIAM←͜Ε͕࠷΋ۂऀʢηΩϡϦςΟతʹ΋ʣ • Ҏલ͸Cloud Console or GUIͰ࡞͍͕ͬͯͨഁ୼͢Δʢྫ: ਎ʹ͕֮͑ແ͍Service Account, InstanceͳͲʣͷͰࠓ೥͔ΒIaCԽ. ઃܭͱ࣮૷ͷॳظίετʢओʹ࣌ؒͱΩϟονΞοϓʣ͸͋Δ͕, ౤ࢿͨ͠΄͏͕ઈରʹྑ͍ʢͱ࣮ײ͍ͯ͠·͢ʣ.

Slide 10

Slide 10 text

Google CloudͰTerraformΛ࢝ΊΔ࣌͸ • Quick StartͳͲ, ެࣜυΩϡϝϯτʹsnippet͕͋ΔͷͰ࢖͓͏ʂ • gcloud command΍GUIઃఆͱͷฒͼʹTerraformͷsnippet͕ଘࡏ. • ࢲ͸ެࣜυΩϡϝϯτͷTerraformͷsnippetΛmain.tfʹషΓ෇͚, ಈ࡞ ֬ೝ͠ͳ͕Βຊ൪༻ͷ࣮૷Λߦͬͨʢ͘͢͝ḿͬͨʣ. શͯͷυΩϡϝϯτʹ͋Δ༁Ͱ͸ແͦ͞͏͕ͩੵۃతʹ࢖͓͏.

Slide 11

Slide 11 text

ʁʁʁʮ͍͍ײ͡ʹࣗಈԽͱDevOps͍ͨ͠ΜͰ͕͢Ͳ͏͢Ε͹🤔ʯ Enterprise޲͚Ͱ΋࠷ۙ͋Δͱࢥ͏͜ͷཁ๬ʹͲ͏Ԡ͑Δ͔!?

Slide 12

Slide 12 text

ʲ౴ʳGitHub Actions͔ΒTerraformΛ࢖ͬͨ • ʮ໎ͬͨΒGitHub Actionsʯͱࢥͬͯ΍ͬͨΒҙ֎ͱ্ख͍͖͘·ͨ͠. • ͬ͘͟Γॻ͘ͱҎԼͷखॱͰ࣮ݱ. 1. Workload Identityͷ४උ. 2. CDʢܧଓతσϓϩΠʣͱ࣮ͯ͠ߦ͢Δର৅ͷܾఆ. 3. λεΫͷॱ൪ΛܾΊΔ. • Cloud RunΞϓϦέʔγϣϯͷ৔߹, ͜ͷ۠෼͚Ͱ͏·͘ߦͬͨ. • Workload Identity͓ΑͼΞϓϦͷService AccountʢSAʣൃߦ͸GitHub ActionsͰ͸ͳ͘TerraformͰखಈ࣮ߦ. • SAͱ֤छϦιʔεͷBinding, ͦͷଞͷϦιʔεͷCD͸GitHub ActionsͰ࣮ߦ. • ArtifactʢDocker imageʣੜ੒ -> DeployΛઈରతͳॱংͱͨ͠GitHub Actions Work fl owͷઃܭͱ࣮૷. ※͕͜͜ॏཁ

Slide 13

Slide 13 text

ʲਤʳࠓճ࠾༻ͨ͠GitHub ActionsͷCI/CD Flow. Cloud BuildʹΑΔImage BuildͱPull Request͕੒ཱ͔ͯ͠ΒTerraform ApplyͷྲྀΕʹ. ©Shinichi Nakagawa

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

GitHub Actions + Terraform, Cloud Buildͷ޻෉ • CI/CDεςʔδͷ໾ׂΛ໌֬Խ. • ʢ౰ͨΓલͰ͕͢ʣςετ͸ຖճࣗಈ࣮ߦ. • Pull Request࣌ʹImage Buildͱterraform planͷ֬ೝ. • Deployʢterraform applyʣ͸main΁ͷMergeͷΈ. • GitHub ActionsͰCI/CDͷϑϩʔΛ੍ޚ. • ͢΂ͯͷΞϓϦέʔγϣϯͷArtifactΛCloud BuildͰ࡞੒ޙʹterraform planΛݩʹϨϏϡʔ. • GitHub ActionsͷneedsઃఆͱPull RequestͷonΠϕϯτͰ͍͍ײ͡ʹ੍ޚ. • ϨϏϡʔ༻ʹterraform planͷ݁ՌΛPull Requestͷίϝϯτʹ৐ͤΔ.

Slide 19

Slide 19 text

݁ͼ • ෳࡶʹͳΓ͕ͪͳMicro Service͸IaCԽ͠·͠ΐ͏, ๨ΕΔͱେม. • GitHub ActionsͱCloud BuildͰTerraform͸ͻͱ޻෉͕ඞཁ. ॳखͰIaCʢTerraformʣΛೖΕΔۤ࿑͸͋Γ·͕͢େมʹॏཁ. ͍͖ͳΓ࢓ࣄͰ΍ΔͱେมͳͷͰԿ͔͠Β࿅शΛ͠·͠ΐ͏.