Share
Embed
一人でも小さく始められるGoogle Cloudで実現するほぼサーバレスなデータ基盤 / Serverless Dataplatform for Google Cloud
Jagu'e'r データ利活用分科会LTスライド
Shinichi Nakagawa PRO
December 02, 2022
Transcript
25 slides
-
ҰਓͰখ࢝͘͞ΊΒΕΔ Google CloudͰ࣮ݱ͢Δ ΄΅αʔόϨεͳσʔλج൫ ㅟ ㅟ ㅟ ㅟ ㅟ ㅟ
ㅟ ⚾؍ઓΛศརʹ͢ΔͨΊͷʮݸਓతͳDXʯઓུͱͦͷ࣮ Shinichi Nakagawa 2022/12/02 Jagu’e’rσʔλར׆༻Պձ #8 -
Who am I ? • Shinichi Nakagawaʢத৳Ұʣ • ΞΫηϯνϡΞגࣜձࣾ
ςΫϊϩδʔίϯαϧςΟϯάຊ෦Ϛωδϟʔ • Ҏલͷࣄ: ελʔτΞοϓ, ϝΨϕϯνϟʔͷΤϯδχΞ • ΞΫηϯνϡΞͰGoogle Cloudؔ࿈ͷσϦόϦʔ • ݸਓͱͯ͠ҎԼͷతͰϓϩμΫτ։ൃʢ㲈झຯʣ • ٿσʔλੳɾղੳ • ࣗࣗͷϔϧεέΞ • ্هΛςʔϚʹٕͨ͠ज़ݕূ • ਪ͠ͷGoogle Cloud: BigQuery, Cloud Run • ਪ͠ͷBaseball Human: ৽ঙ߶ࢤ, ສதਖ਼ -
ϝδϟʔϦʔάͷϏοάσʔλ • ϝδϟʔϦʔάʮStatcastʯͱ͍͏γεςϜͰ৭ΜͳσʔλΛه͍ͯ͠·͢. ※ݪଇΧϝϥɾϨʔμʔͱ͍ͬͨܭଌػثͰهʢҰ෦ਓྗͰͷهɾਪଌΛؚΉʣ • ྫ͑, ࣮گɾղઆͷݩωλͯ͢͜ͷʮStatcastʯͱ͍͏Ϗοάσʔλ͕ݩωλʹͳ͍ͬͯ·͢. • ΦΦλχαϯʂ˓߸ຊྥଧʂଧٿ180km/h,
ඈڑ130m • ΦΦλχαϯʂ162km/hͷਅ͙ͬͰݟಀ͠ࡾৼʂʂʂ • ٿͷҰڍखҰ, ͯ͢ͷٿɾଧٿσʔλ͕ه͞ΕΔ. • ϨΪϡϥʔγʔζϯʢ30νʔϜɾ162ࢼ߹ʣͰ͓͓Αͦ70ʙ80ສٿલޙ. ϙετγʔζϯɾय़Ωϟϯϓσʔλ͋Δ. • σʔλ91ݸͷ߲ʢ!?ʣͰߏ͞ΕΔ, ϨΪϡϥʔγʔζϯͰ͓͓Αͦ400MBʙ600MB͙Β͍ͷσʔλ. • baseballsavant.mlb.com ͱ͍͏αΠτͰ୭ͰӾཡɾμϯϩʔυʢCSV ϑΥʔϚοτʣͰ͖·͢. -
2022ͷΦΦλχαϯ, εϥΠμʔͱ2γʔϜ, ΧοτϘʔϧܑ͞ΜʹͳΔ • ࠓͷΦΦλχαϯ, ΊͬͪΌ εϥΠμʔ͍͛ͯΔ • ͓ؾ͖ͮͩΖ͏͔?ޙઓ
2γʔϜʢσʔλ্Sinkerʣ͕ ૿͍͑ͯΔ͜ͱʹ!? • εϥΠμʔ, 2γʔϜ, ΧοτϘʔϧͰ บ͕ڧ͍ۂ͕Γٿ͛ΔϚϯʹΩϟϥม ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022 ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022 -
This presentation makes reference to marks owned by third parties.
Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied. -
ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ • ຖσʔλ֬ೝɾຖσʔλߋ৽Λ͍͍ײ͡ʹ࣮ݱ͢ΔͨΊ, ʮϑϧϚωʔδυͳαʔόϨεܥΫϥυαʔϏεʯΛશ໘తʹ׆༻ͯ͠ߏஙɾӡ༻. • αʔϏεબఆͷجຊํ • ʮDWHBigQueryʯΛىʹ, ETLͱΞϓϦέʔγϣϯຊମΛઃܭʢBQΛ͍͍͔ͨΒʣ
• ֤ίϯϙʔωϯτϚΠΫϩαʔϏεͱͯ͠ಠཱͤ͞Δ, ࡞Γ͍͢ɾςετ͍͢͠ͷͰ. • ҰͭҰͭͷཁૉখ͍͞ΞϓϦͳͷͰ, Cloud Funcions or Cloud RunͰߏஙɾӡ༻ • GitHub ActionsͷCI/CDͷύΠϓϥΠϯʹΈࠐΜͰσϓϩΠɾεέʔϧͰ͖ͨΓ جຊతʹʮ͚ͬͨͩ՝ۚʯʹͳΔͷͰ͓ࡒʹ༏͍͠ʢ݄͋ͨΓ$5લޙʣ👛 -
• ΞϓϦຊମCloud RunͰϗεςΟϯά, ΞϓϦຊମDashͱ͍͏PythonͷFrameworkͰ࣮ • API GatewayΛ௨ͯ͠BackendʢCloud FunctionsʣʹΞΫηε. BackendFunctions FrameworkͰ࡞ͬͨRESTful
API • DatabaseFirestore, ޙʹհ͢ΔETLͰBigQuery͔ΒETLͯ͠ߏங μογϡϘʔυΞϓϦ This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied. -
σʔλऩू&BigQueryอଘ • σʔλݩαΠτʢBaseball Savantʣ͔Βఆظతʹσʔλऩू͢ΔΫϩʔϥʔʢCloud Functionsʣ࣮ߦ • ࣮ߦ݁ՌGoogle Cloud StorageʢGCSʣʹCSVͱͯ͠อଘ. ͜Ε͕ݯઘͷσʔλʢDatalakeʣ
• GCS্ͷCSVΛαϚϦʔ͍͍ͯ͠ײ͡ʹͯ͠BigQueryʹอଘ͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied. -
FirestoreೖʢDatabaseʹσʔλҠૹʣ • BigQueryσʔλΛμογϡϘʔυ༻σʔλͷܗࣜʢJSONʣʹม͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ • ࣮ߦ݁ՌʢGCS্ʹJSONܗࣜͰอଘʣΛFirestoreʹೖΕΔͨΊͷPythonεΫϦϓτΛ࣮ߦ • DataprocͱFirestoreͷॲཧखݩͷεΫϦϓτΛΛखಈ࣮ߦʢશࣗಈԽΛ્͢Δ੍͕͋ͬͨͨΊʣ This presentation
makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied. -
ӡ༻ͯ͠ͷৼΓฦΓ • ΞϓϦଆCloud Run & Cloud FunctionsͰ͍͍ײ͡ʹӡ༻Ͱ͖ͯΔ👏 અͷͨΊϦιʔεΛίʔϧυελϯόΠͷঢ়ଶͰӡ༻͍ͯ͠Δ͕
ݸਓར༻ͳͷͰࢧোͳ͍ʢ͔ͭinstanceCIճͯ͠૿ݮͰ͖Δߏʣ • σʔλଆධՁ͕͔ΕΔ • Cloud FunctionsͱSchedulerͰͷϐλΰϥεΠονͳσʔλॲཧ˕ • BigQuery͍͍ײ͡, ूܭॲཧͳͲετϨεແ͘ߦ͚͍ͯΔ̋ • Dataproc serverlessΛࠓճͷنͰ͏ͷ͔ͩͬͨ͠Εͳ͍ʁ -
This presentation makes reference to marks owned by third parties.
Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied. -
ࠓޙΖ͏ͱࢥ͍ͬͯΔ͜ͱ • ETLCloud Functions & Cloud SchedulerͷϐλΰϥεΠονʹ౷Ұ • Dataproc࣮ࡍͬͨ݁Ռ, ࣗͷϢʔεέʔεʹtoo
much • γϯϓϧͳCSV͔ͭσʔλྔগͳ͍ͷͰCloud FunctionsͰॲཧՄೳ • σʔλऩूɾॲཧϑϩʔͷશࣗಈԽ, IaCʹΑΔΠϯϑϥཧ • BigQueryΛ֤ͬͨछ౷ܭσʔλͷॆ࣮Խ • ࠓ΄΅ੜσʔλΛΫΤϦͯ͠Δ͚ͩʢඞཁʹԠͯ͡viewΛ࡞ΔͳͲʣ • Spark͑Δ͠͏ͪΐͬͱؾͷར͍ͨDatamartΛॆ࣮ͤ͞Δ -
ߨԋͷ·ͱΊ • ϝδϟʔϦʔάʹΦʔϓϯσʔλ͕͋Γɺ େ୩ᠳฏબखͳͲͷύϑΥʔϚϯε͕ ֬ೝɾධՁͰ͖·͢ɻ • ΦʔϓϯσʔλͷੳɾՄࢹԽΛৗతʹ
׆༻͢ΔͨΊɺGoogle CloudͰ σʔλੳج൫Λ࡞Γ·ͨ͠ɻ • αʔόϨεɾΞʔΩςΫνϟͷΈͰ σʔλج൫ߏஙɾӡ༻࣮ݱՄೳɺ ڧ͘Φεεϝ͍͖͍ͯͨ͠Ͱ͢ɻ ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022