Cloud RunとCloud PubSubでサーバレスなデータ基盤2024 with Terraform / Cloud Run and PubSub with Terraform

Google Cloudのサーバレスなサービスでデータ基盤を作った話.

Transcript

1. Cloud RunͱCloud Pub/SubͰ αʔόϨεͳσʔλج൫Λ ࡞ͬͨ࿩2024 Ϋϥ΢υωΠςΟϒͳ࣌୅ͷMicro ServiceͱIaCͷࡏΓํͷߟ࡯ͱ࣮ફ Shinichi Nakagawa(@shinyorke) 2024/3/8 ୈ22ճ ञͱήʔϜͱΠϯϑϥͱGCP
2. TL;DRʢ͜ͷ࿩ͷ֓ཁʣ • Cloud RunͱCloud Pub/SubͰαʔόϨεͳσʔλج൫Λ࡞ͬͨ. • Micro ServiceΛTerraformͰؙͬͱIaCʢInfrastructure as Codeʣ. SoIʢSystem Of InsightsʣͷγεςϜ࣮ݱͷҰྫͱͯ͠ฉ͍ͯԼ͍͞. ※͜ͷࣄྫ͸ۀ຿Ͱ͸ͳ͘झຯͰ͢ʢ͕࣮຿Ͱ΋࢖͑Δ͸ͣʣ.
3. ࣗݾ঺հ • Shinichi Nakagawa(@shinyorke) • େख֎ࢿܥITίϯαϧاۀϚωʔδϟʔ/SRE • Google Cloud Partner Top Engineer 2024 • ޷͖ͳGCP: Cloud Run, BigQuery, Cloud Pub/Sub, etc… • ޷͖ͳήʔϜ: ৴௕ͷ໺๬, ΢Πχϯάϙετ, ΍͖Ύͭ͘ • झຯϓϩάϥϛϯάΛ͓ञยखʹ΍Γ·͢ • 🍶: ෩ͷ৿, ࡞ʢ७ถܥ޷͖ʣ • 🍺: ଖͷेʢࢤլߴݪϏʔϧͷIPAʣ • ීஈ͸ϗοϐʔͱ๐੝ιʔμׂΓ͕޷Έ
4. ໺ٿσʔλ෼ੳج൫ ※ݸਓ։ൃͰ͢ • झຯͷ໺ٿΛຊ֨తʹσʔλͰݟΔͨΊͷج൫ΛຊؾͰ ࡞͍ͬͯ·͢ʢٕज़తͳݕূΛ݉Ͷͯʣ. • ಛʹϝδϟʔϦʔά͸ʮBaseball Savantʯͱ͍͏ެࣜͷ σʔλαΠτ͕͋Γ, ϑΝϯͷզʑͰ΋CSVܗࣜͰσʔλ ͕खʹೖΓ·͢&σʔλͷத਎͕݁ߏ΍΍͍͜͠Ͱ͢. • ͦͷʮ΍΍͍͜͠CSVσʔλʯΛऔΓࠐΈ, ෼ੳɾղੳΛ ߦ͏ϝδϟʔϦʔά෼ੳΞϓϦΛ2022೥͔Β։ൃɾӡ༻. ຖ೥ϦΞʔΩςΫνϟʢ࡞Γ௚͠ʣ͍ͯ͠·͢, ࠓ೥͸ Cloud Runͷ৽ػೳʢCloud Run JobsʣͰ࡞Γ௚͠. ෼ੳج൫ͷΞϓϦέʔγϣϯʢ2022೥൛ʣ
5. ໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ
6. ໺ٿσʔλ෼ੳج൫શମ૾ʢ2024೥౓൛ʣ
7. ຊ೔ͷτʔΫͷൣғ͸ͪ͜ΒͷData Work fl owʢσʔλج൫ʣʹ͍ͭͯ.
8. Cloud RunͱCloud Pub/SubͰWork fl ow • Cloud Run JobsΛSchedulerܦ༝Ͱ࣮ߦ, ֤Micro ServiceʢCloud RunʣΛݺͼग़͢ߏ੒. • ֤Micro Service͸Pub/SubͷϝοηʔδͰಈ͘Cloud RunΞϓϦʹ. • Cloud Run Jobs্ͷWork fl ow͕Micro ServiceͷPub/SubʹPublish. • ͜ΕΒ͸͢΂ͯGoogle CloudͷυΩϡϝϯτʹਅࣅͰ͖Δώϯτ༗Γ. • Pub/SubͰCloud Run https://cloud.google.com/run/docs/tutorials/pubsub?hl=ja • SchedulerΛ࢖ͬͨఆظ࣮ߦ https://cloud.google.com/run/docs/triggering/using-scheduler?hl=ja খ͍͞Work fl owͳΒAir fl ow(Cloud Composer)Λ࢖Θͳͯ͘΋(K8sΫϥελ͕ແͯ͘΋)Cloud Run͚ͩͰ࣮ݱՄೳ.
9. ʲࢀߟʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ .JDSP
   4FSWJDF
   PS
   ໾ׂ (PPHMF
   $MPVE
   4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE
   4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ
   ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE
   3VO
   +PCT &YQPSUFS
   
   *NQPSUFSΛ࣮ߦ
   ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷
   -VJHJ౳
   ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE
   3VO #BTFCBMM
   4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %BUBMBLF $MPVE
   4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ
   ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE
   3VO $47σʔλΛνΣοΫ͠
   #JH2VFSZʹ&YQPSU (P੡ͷ8FC
   "1*
   1VC4VC
   5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀ
   PS
   ΞϓϦ͔Β࢖͏
   %BUB
   8BSF
   )PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻
10. αʔόϨεͳͷͰ͓ࡒ෍ʹ΋༏͍͠, ֹ݄Ͱίʔώʔ3.34ഋఔ౓ʢ֓ࢉʣ Cloud ComposerͷΫϥελΛMinimumͰӡ༻ͨ͠ͱͯ$100Ҏ্͸͔͔Δ͸ͣ?೔࣍Ͱಈ͔͢Cloud RunͳΒ଴ػίετ͸΄΅θϩʹ.
11. TerraformͰIaCʢInfrastructure as Codeʣେ੾΍ͰʢझຯͰ΋ʣ
12. ʲ࠶ܝʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ .JDSP
    4FSWJDF
    PS
    ໾ׂ (PPHMF
    $MPVE
    4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE
    4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ
    ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE
    3VO
    +PCT &YQPSUFS
    
    *NQPSUFSΛ࣮ߦ
    ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷
    -VJHJ౳
    ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE
    3VO #BTFCBMM
    4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC
    "1*
    1VC4VC
    5PQJD͔Βىಈ %BUBMBLF $MPVE
    4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ
    ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE
    3VO $47σʔλΛνΣοΫ͠
    #JH2VFSZʹ&YQPSU (P੡ͷ8FC
    "1*
    1VC4VC
    5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀ
    PS
    ΞϓϦ͔Β࢖͏
    %BUB
    8BSF
    )PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻
13. .JDSP
    4FSWJDF
    PS
    ໾ׂ (PPHMF
    $MPVE
    4FSWJDF ֓ཁ ࣮૷ݴޠɾ'8 5JNFS $MPVE
    4DIFEVMFS 5SJHHFSΛఆظ࣮ߦ
    ˞γʔζϯ։࢝ޙຖ೔ DSPOࣜͰͷ5JNFS 5SJHHFS 8PSL fl PX $MPVE
    3VO
    +PCT &YQPSUFS
    
    *NQPSUFSΛ࣮ߦ
    ͢Δ5SJHHFSͱͳΔόον 1SFGFDU 1ZUIPO Ͱ࣮૷
    -VJHJ౳
    ଞͷ࣮૷Ͱ΋0, &YQPSUFS $MPVE
    3VO #BTFCBMM
    4BWBOUΫΤϦΛ࣮ ߦ͠$47Λμ΢ϯϩʔυ (P੡ͷ8FC
    "1*
    1VC4VC
    5PQJD͔Βىಈ %BUBMBLF $MPVE
    4UPSBHF &YQPSUFS͕ग़ྗͨ͠$47Λ
    ແՃ޻ͷੜσʔλͱͯ͠อ؅ ͳ͠ *NQPSUFS $MPVE
    3VO $47σʔλΛνΣοΫ͠
    #JH2VFSZʹ&YQPSU (P੡ͷ8FC
    "1*
    1VC4VC
    5PQJD͔Βىಈ %8) #JH2VFSZ ෼ੳऀ
    PS
    ΞϓϦ͔Β࢖͏
    %BUB
    8BSF
    )PVTF 42-Ͱ7JFXΛ࡞ͬͯӡ༻ ʲ࠶ܝʳ֤Micro ServiceͰ࢖ͬͯΔϞϊͱղઆ શ෦खಈͰઃఆ͢Δͱഁ୼͢Δະདྷ͕଴͍ͬͯ·͢ʂ
14. Micro ServiceʹTerraform͸෇͖෺Ͱ͢Ͷઆ. • ෳ਺ͷΞϓϦέʔγϣϯΛ਺चͭͳ͗ʢϐλΰϥεΠονʣ͢Δ৔߹, TerraformΛೖΕ͓ͯ͜͏, খن໛Ͱ͋ͬͨͱͯ͠΋. • ࠓճͷߏ੒Ͱ͸·͋·͋ͷ਺ͷGoogle CloudͷServiceΛར༻. • Cloud RunΞϓϦ͕3ݸ + Volume, LoggingͳͲͷݻ༗ઃఆ • Pub/Sub TopicͱSchema͕ͦΕͧΕ2ݸ • Cloud Schedulerͷઃఆ • Cloud StorageͷBucketͱΞΫηε੍ޚ • ͜ΕΒͱ৭ʑඥͮ͘ઃఆ, ಛʹService Account͓ΑͼIAM←͜Ε͕࠷΋ۂऀʢηΩϡϦςΟతʹ΋ʣ • Ҏલ͸Cloud Console or GUIͰ࡞͍͕ͬͯͨഁ୼͢Δʢྫ: ਎ʹ͕֮͑ແ͍Service Account, InstanceͳͲʣͷͰࠓ೥͔ΒIaCԽ. ઃܭͱ࣮૷ͷॳظίετʢओʹ࣌ؒͱΩϟονΞοϓʣ͸൒୺ͳ͍Ͱ͕͢, ౤ࢿͨ͠΄͏͕ઈରʹྑ͍.
15. Google CloudͰTerraformΛ࢝ΊΔ࣌͸ • Quick StartͳͲ, ެࣜυΩϡϝϯτʹsnippet͕͋ΔͷͰ࢖͓͏ʂ • gcloud command΍GUIઃఆͱͷฒͼʹTerraformͷsnippet͕ଘࡏ. • ࢲ͸ެࣜυΩϡϝϯτͷTerraformͷsnippetΛmain.tfʹషΓ෇͚, ಈ࡞ ֬ೝ͠ͳ͕Βຊ൪༻ͷ࣮૷Λߦͬͨʢ͘͢͝ḿͬͨʣ. શͯͷυΩϡϝϯτʹ͋Δ༁Ͱ͸ແͦ͞͏͕ͩੵۃతʹ࢖͓͏.
16. ݁ͼ • Cloud RunͱCloud Pub/Sub͚ͩͰ͓҆͘σʔλج൫͸࡞ΕΔ. • ෳࡶʹͳΓ͕ͪͳMicro Service͸IaCԽ͠·͠ΐ͏, झຯͰ΋. • ⚾γʔζϯ࢝·ͬͨΒຊ֨ӡ༻͠·͢&ͦͷ࿩΋Ͳ͔͜Ͱ·ͨ. ͱΓ͋͑ͣσʔλج൫తͳϞϊ͕ཉ͘͠ͳͬͨΒਅࣅ͸ΞϦ͔ͳͱ. গͳ͘ͱ΋Cloud Run͸૬౰࢖͑·͢ʢϑΝϯͳͷͰେ੠Ͱݴ͏ʣ.
17. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠. Shinichi Nakagawa ©Shinichi Nakagawa