Cloud RunとCloud PubSubでサーバレスなデータ基盤2024 with Terraform / Cloud Run and PubSub with Terraform

Slide 1

Slide 1 text

Cloud RunͱCloud Pub/SubͰ αʔόϨεͳσʔλج൫Λ ࡞ͬͨ࿩2024 Ϋϥ΢υωΠςΟϒͳ࣌୅ͷMicro ServiceͱIaCͷࡏΓํͷߟ࡯ͱ࣮ફ Shinichi Nakagawa(@shinyorke) 2024/3/8 ୈ22ճ ञͱήʔϜͱΠϯϑϥͱGCP

Slide 2

Slide 2 text

TL;DRʢ͜ͷ࿩ͷ֓ཁʣ • Cloud RunͱCloud Pub/SubͰαʔόϨεͳσʔλج൫Λ࡞ͬͨ. • Micro ServiceΛTerraformͰؙͬͱIaCʢInfrastructure as Codeʣ. SoIʢSystem Of InsightsʣͷγεςϜ࣮ݱͷҰྫͱͯ͠ฉ͍ͯԼ͍͞. ※͜ͷࣄྫ͸ۀ຿Ͱ͸ͳ͘झຯͰ͢ʢ͕࣮຿Ͱ΋࢖͑Δ͸ͣʣ.

Slide 3

Slide 3 text

ࣗݾ঺հ • Shinichi Nakagawa(@shinyorke) • େख֎ࢿܥITίϯαϧاۀϚωʔδϟʔ/SRE • Google Cloud Partner Top Engineer 2024 • ޷͖ͳGCP: Cloud Run, BigQuery, Cloud Pub/Sub, etc… • ޷͖ͳήʔϜ: ৴௕ͷ໺๬, ΢Πχϯάϙετ, ΍͖Ύͭ͘ • झຯϓϩάϥϛϯάΛ͓ञยखʹ΍Γ·͢ • 🍶: ෩ͷ৿, ࡞ʢ७ถܥ޷͖ʣ • 🍺: ଖͷेʢࢤլߴݪϏʔϧͷIPAʣ • ීஈ͸ϗοϐʔͱ๐੝ιʔμׂΓ͕޷Έ

Slide 4

Slide 4 text

໺ٿσʔλ෼ੳج൫ ※ݸਓ։ൃͰ͢ • झຯͷ໺ٿΛຊ֨తʹσʔλͰݟΔͨΊͷج൫ΛຊؾͰ ࡞͍ͬͯ·͢ʢٕज़తͳݕূΛ݉Ͷͯʣ. • ಛʹϝδϟʔϦʔά͸ʮBaseball Savantʯͱ͍͏ެࣜͷ σʔλαΠτ͕͋Γ, ϑΝϯͷզʑͰ΋CSVܗࣜͰσʔλ ͕खʹೖΓ·͢&σʔλͷத਎͕݁ߏ΍΍͍͜͠Ͱ͢. • ͦͷʮ΍΍͍͜͠CSVσʔλʯΛऔΓࠐΈ, ෼ੳɾղੳΛ ߦ͏ϝδϟʔϦʔά෼ੳΞϓϦΛ2022೥͔Β։ൃɾӡ༻. ຖ೥ϦΞʔΩςΫνϟʢ࡞Γ௚͠ʣ͍ͯ͠·͢, ࠓ೥͸ Cloud Runͷ৽ػೳʢCloud Run JobsʣͰ࡞Γ௚͠. ෼ੳج൫ͷΞϓϦέʔγϣϯʢ2022೥൛ʣ

Slide 5

Slide 5 text

໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ

Slide 6

Slide 6 text

໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ

Slide 7

Slide 7 text

ຊ೔ͷτʔΫͷൣғ͸ͪ͜ΒͷData Work fl owʢσʔλج൫ʣʹ͍ͭͯ.

Slide 8

Slide 8 text

Cloud RunͱCloud Pub/SubͰWork fl ow • Cloud Run JobsΛSchedulerܦ༝Ͱ࣮ߦ, ֤Micro ServiceʢCloud RunʣΛݺͼग़͢ߏ੒. • ֤Micro Service͸Pub/SubͷϝοηʔδͰಈ͘Cloud RunΞϓϦʹ. • Cloud Run Jobs্ͷWork fl ow͕Micro ServiceͷPub/SubʹPublish. • ͜ΕΒ͸͢΂ͯGoogle CloudͷυΩϡϝϯτʹਅࣅͰ͖Δώϯτ༗Γ. • Pub/SubͰCloud Run https://cloud.google.com/run/docs/tutorials/pubsub?hl=ja • SchedulerΛ࢖ͬͨఆظ࣮ߦ https://cloud.google.com/run/docs/triggering/using-scheduler?hl=ja খ͍͞Work fl owͳΒAir fl ow(Cloud Composer)Λ࢖Θͳͯ͘΋(K8sΫϥελ͕ແͯ͘΋)Cloud Run͚ͩͰ࣮ݱՄೳ.

Slide 9

Slide 9 text

ʲࢀߟʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ .JDSP4FSWJDFPS໾ׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷ -VJHJ౳ ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀPSΞϓϦ͔Β࢖͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻

Slide 10

Slide 10 text

αʔόϨεͳͷͰ͓ࡒ෍ʹ΋༏͍͠, ֹ݄Ͱίʔώʔ3.34ഋఔ౓ʢ֓ࢉʣ Cloud ComposerͷΫϥελΛMinimumͰӡ༻ͨ͠ͱͯ$100Ҏ্͸͔͔Δ͸ͣ?೔࣍Ͱಈ͔͢Cloud RunͳΒ଴ػίετ͸΄΅θϩʹ.

Slide 11

Slide 11 text

TerraformͰIaCʢInfrastructure as Codeʣେ੾΍ͰʢझຯͰ΋ʣ

Slide 12

Slide 12 text

ʲ࠶ܝʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ .JDSP4FSWJDFPS໾ׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷ -VJHJ౳ ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀPSΞϓϦ͔Β࢖͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻

Slide 13

Slide 13 text

.JDSP4FSWJDFPS໾ׂ (PPHMF$MPVE4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE3VO+PCT &YQPSUFS*NQPSUFSΛ࣮ߦ ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷ -VJHJ౳ ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE3VO #BTFCBMM4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %BUBMBLF $MPVE4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE3VO $47σʔλΛνΣοΫ͠ #JH2VFSZʹ&YQPSU (P੡ͷ8FC"1* 1VC4VC5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀPSΞϓϦ͔Β࢖͏ %BUB8BSF)PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻ ʲ࠶ܝʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ શ෦खಈͰઃఆ͢Δͱഁ୼͢Δະདྷ͕଴͍ͬͯ·͢ʂ

Slide 14

Slide 14 text

Micro ServiceʹTerraform͸෇͖෺Ͱ͢Ͷઆ. • ෳ਺ͷΞϓϦέʔγϣϯΛ਺चͭͳ͗ʢϐλΰϥεΠονʣ͢Δ৔߹, TerraformΛೖΕ͓ͯ͜͏, খن໛Ͱ͋ͬͨͱͯ͠΋. • ࠓճͷߏ੒Ͱ͸·͋·͋ͷ਺ͷGoogle CloudͷServiceΛར༻. • Cloud RunΞϓϦ͕3ݸ + Volume, LoggingͳͲͷݻ༗ઃఆ • Pub/Sub TopicͱSchema͕ͦΕͧΕ2ݸ • Cloud Schedulerͷઃఆ • Cloud StorageͷBucketͱΞΫηε੍ޚ • ͜ΕΒͱ৭ʑඥͮ͘ઃఆ, ಛʹService Account͓ΑͼIAM←͜Ε͕࠷΋ۂऀʢηΩϡϦςΟతʹ΋ʣ • Ҏલ͸Cloud Console or GUIͰ࡞͍͕ͬͯͨഁ୼͢Δʢྫ: ਎ʹ͕֮͑ແ͍Service Account, InstanceͳͲʣͷͰࠓ೥͔ΒIaCԽ. ઃܭͱ࣮૷ͷॳظίετʢओʹ࣌ؒͱΩϟονΞοϓʣ͸൒୺ͳ͍Ͱ͕͢, ౤ࢿͨ͠΄͏͕ઈରʹྑ͍.

Slide 15

Slide 15 text

Google CloudͰTerraformΛ࢝ΊΔ࣌͸ • Quick StartͳͲ, ެࣜυΩϡϝϯτʹsnippet͕͋ΔͷͰ࢖͓͏ʂ • gcloud command΍GUIઃఆͱͷฒͼʹTerraformͷsnippet͕ଘࡏ. • ࢲ͸ެࣜυΩϡϝϯτͷTerraformͷsnippetΛmain.tfʹషΓ෇͚, ಈ࡞ ֬ೝ͠ͳ͕Βຊ൪༻ͷ࣮૷Λߦͬͨʢ͘͢͝ḿͬͨʣ. શͯͷυΩϡϝϯτʹ͋Δ༁Ͱ͸ແͦ͞͏͕ͩੵۃతʹ࢖͓͏.

Slide 16

Slide 16 text

݁ͼ • Cloud RunͱCloud Pub/Sub͚ͩͰ͓҆͘σʔλج൫͸࡞ΕΔ. • ෳࡶʹͳΓ͕ͪͳMicro Service͸IaCԽ͠·͠ΐ͏, झຯͰ΋. • ⚾γʔζϯ࢝·ͬͨΒຊ֨ӡ༻͠·͢&ͦͷ࿩΋Ͳ͔͜Ͱ·ͨ. ͱΓ͋͑ͣσʔλج൫తͳϞϊ͕ཉ͘͠ͳͬͨΒਅࣅ͸ΞϦ͔ͳͱ. গͳ͘ͱ΋Cloud Run͸૬౰࢖͑·͢ʢϑΝϯͳͷͰେ੠Ͱݴ͏ʣ.