Slide 1

Slide 1 text

ҰਓͰ΋খ࢝͘͞ΊΒΕΔ Google CloudͰ࣮ݱ͢Δ ΄΅αʔόϨεͳσʔλج൫ ㅟ ㅟ ㅟ ㅟ ㅟ ㅟ ㅟ ⚾؍ઓΛศརʹ͢ΔͨΊͷʮݸਓతͳDXʯઓུͱͦͷ࣮૷ Shinichi Nakagawa 2022/12/02 Jagu’e’rσʔλར׆༻෼Պձ #8

Slide 2

Slide 2 text

Who am I ? • Shinichi Nakagawaʢத઒৳Ұʣ • ΞΫηϯνϡΞגࣜձࣾ 
 ςΫϊϩδʔίϯαϧςΟϯάຊ෦Ϛωδϟʔ • Ҏલͷ࢓ࣄ: ελʔτΞοϓ, ϝΨϕϯνϟʔͷΤϯδχΞ • ΞΫηϯνϡΞͰ͸Google Cloudؔ࿈ͷσϦόϦʔ • ݸਓͱͯ͠͸ҎԼͷ໨తͰϓϩμΫτ։ൃʢ㲈झຯʣ • ໺ٿσʔλ෼ੳɾղੳ • ࣗ෼ࣗ਎ͷϔϧεέΞ • ্هΛςʔϚʹٕͨ͠ज़ݕূ • ਪ͠ͷGoogle Cloud: BigQuery, Cloud Run • ਪ͠ͷBaseball Human: ৽ঙ߶ࢤ, ສ೾தਖ਼

Slide 3

Slide 3 text

ຊ೔ͷ͓࿩ • ϝδϟʔϦʔάͷϏοάσʔλͷ͝঺հ • PythonͱGoogle CloudͰ࡞ΔαʔόϨεͰ͍͍ײ͡ͳσʔλج൫ ʮGoogle CloudͰσʔλج൫ΛαʔόϨεʹখ࢝͘͞ΊΔʯ࿩Λ 
 ओʹٕज़બఆͱ؍఺ͷଆ໘Ͱ͓࿩͠·͢, ໺ٿΛྫʹ⽁

Slide 4

Slide 4 text

ϝδϟʔϦʔάͷϏοάσʔλ • ϝδϟʔϦʔά͸ʮStatcastʯͱ͍͏γεςϜͰ৭ΜͳσʔλΛه࿥͍ͯ͠·͢. 
 ※ݪଇΧϝϥɾϨʔμʔͱ͍ͬͨܭଌػثͰه࿥ʢҰ෦ਓྗͰͷه࿥ɾਪଌ஋ΛؚΉʣ • ྫ͑͹, ࣮گɾղઆͷݩωλ͸͢΂ͯ͜ͷʮStatcastʯͱ͍͏Ϗοάσʔλ͕ݩωλʹͳ͍ͬͯ·͢. • ΦΦλχαϯʂ˓߸ຊྥଧʂଧٿ଎౓180km/h, ඈڑ཭130m • ΦΦλχαϯʂ162km/hͷਅͬ௚͙Ͱݟಀ͠ࡾৼʂʂʂ • ໺ٿͷҰڍखҰ౤଍, ͢΂ͯͷ౤ٿɾଧٿσʔλ͕ه࿥͞ΕΔ. • ϨΪϡϥʔγʔζϯʢ30νʔϜɾ162ࢼ߹ʣͰ͓͓Αͦ70ʙ80ສٿલޙ. ϙετγʔζϯɾय़Ωϟϯϓσʔλ΋͋Δ. • σʔλ͸91ݸͷ߲໨ʢ!?ʣͰߏ੒͞ΕΔ, ϨΪϡϥʔγʔζϯ෼Ͱ͓͓Αͦ400MBʙ600MB͙Β͍ͷσʔλ. • baseballsavant.mlb.com ͱ͍͏αΠτͰ୭Ͱ΋Ӿཡɾμ΢ϯϩʔυʢCSV ϑΥʔϚοτʣͰ͖·͢.

Slide 5

Slide 5 text

ϝδϟʔϦʔάͷσʔλͰৼΓฦΔʮΦΦλχαϯͷ2022೥ʯ ͜ͷσʔλͰԿ͕Ͱ͖Δ͔঺հ͠·͢ ग़య: ͍Β͢ͱ΍ https://www.irasutoya.com/2013/12/blog-post_5056.html ग़య: ͍Β͢ͱ΍ https://www.irasutoya.com/2019/06/blog-post_512.html

Slide 6

Slide 6 text

2022೥ͷΦΦλχαϯ, εϥΠμʔͱ2γʔϜ, ΧοτϘʔϧܑ͞ΜʹͳΔ • ࠓ೥ͷΦΦλχαϯ, ΊͬͪΌ 
 εϥΠμʔ౤͍͛ͯΔ • ͓ؾ͖ͮͩΖ͏͔?ޙ൒ઓ͸ 
 2γʔϜʢσʔλ্͸Sinkerʣ͕ 
 ૿͍͑ͯΔ͜ͱʹ!? • εϥΠμʔ, 2γʔϜ, ΧοτϘʔϧͰ 
 บ͕ڧ͍ۂ͕Γٿ౤͛ΔϚϯʹΩϟϥม ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022 ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022

Slide 7

Slide 7 text

ͱ͋ΔΦΦλχαϯͷొ൘೔ʢ2022/9/29, 8ճ10ୣࡾৼແࣦ఺ʣ ൒෼ۙ͘εϥΠμʔΛ౤͛ͯ2γʔϜͱΧοτͰԡ͍ͯ͘͠Πϝʔδ ౤͛ͨ৔ॴʢัख໨ઢʣ ϦϦʔεϙΠϯτʢัख໨ઢʣ ٿछͷׂ߹ ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022ʢ্ه͢΂ͯʣ

Slide 8

Slide 8 text

ͪͳΈʹଧऀͱͯ͠ύϑΥʔϚϯε ඇৗʹڧ͍ଧٿΛଧͬͯϗʔϜϥϯʹ͍ͯ͠Δ͜ͱ͕Θ͔Δ ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022

Slide 9

Slide 9 text

ʁʁʮຖ೔ຖࢼ߹ݟΔ࢓૊Έཉ͍͠ʯ ී௨ͷσʔλ෼ੳͳΒʮGoogle ColabͰͪΐͬͱ৮͓ͬͯ͠·͍ʯͰ͕͢ ຖ೔ݟΔ࢓૊Έ͕ཉ͔ͬͨ͠ͷͰ࡞Γ·ͨ͠

Slide 10

Slide 10 text

ͱ͍͏Θ͚Ͱ, ͪΐͬ͜ͱ࡞ͬͯΈ·ͨ͠. ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022

Slide 11

Slide 11 text

PythonͱGoogle CloudͰ࡞Δ αʔόϨεͰ͍͍ײ͡ͳ σʔλج൫ʢ໺ٿฤʣ

Slide 12

Slide 12 text

ΞʔΩςΫνϟͷશମ૾

Slide 13

Slide 13 text

This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied.

Slide 14

Slide 14 text

ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ • ຖ೔σʔλ֬ೝɾຖ೔σʔλߋ৽Λ͍͍ײ͡ʹ࣮ݱ͢ΔͨΊ, 
 ʮϑϧϚωʔδυͳαʔόϨεܥΫϥ΢υαʔϏεʯΛશ໘తʹ׆༻ͯ͠ߏஙɾӡ༻. • αʔϏεબఆͷجຊํ਑ • ʮDWH͸BigQueryʯΛى఺ʹ, ETLͱΞϓϦέʔγϣϯຊମΛઃܭʢBQΛ࢖͍͍͔ͨΒʣ • ֤ίϯϙʔωϯτ͸ϚΠΫϩαʔϏεͱͯ͠ಠཱͤ͞Δ, ࡞Γ΍͍͢ɾςετ͠΍͍͢ͷͰ. • ҰͭҰͭͷཁૉ͸খ͍͞ΞϓϦͳͷͰ, Cloud Funcions or Cloud RunͰߏஙɾӡ༻ • GitHub Actions౳ͷCI/CDͷύΠϓϥΠϯʹ૊ΈࠐΜͰσϓϩΠɾεέʔϧͰ͖ͨΓ 
 جຊతʹ͸ʮ࢖ͬͨ෼͚ͩ՝ۚʯʹͳΔͷͰ͓ࡒ෍ʹ΋༏͍͠ʢ݄͋ͨΓ$5લޙʣ👛

Slide 15

Slide 15 text

Ϣʔεέʔε঺հ

Slide 16

Slide 16 text

• ΞϓϦຊମ͸Cloud RunͰϗεςΟϯά, ΞϓϦຊମ͸Dashͱ͍͏PythonͷFrameworkͰ࣮૷ • API GatewayΛ௨ͯ͠BackendʢCloud FunctionsʣʹΞΫηε. Backend͸Functions FrameworkͰ࡞ͬͨRESTful API • Database͸Firestore, ޙʹ঺հ͢ΔETLͰBigQuery͔ΒETLͯ͠ߏங μογϡϘʔυΞϓϦ This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied.

Slide 17

Slide 17 text

σʔλऩू&BigQueryอଘ • σʔλݩαΠτʢBaseball Savantʣ͔Βఆظతʹσʔλऩू͢ΔΫϩʔϥʔʢCloud Functionsʣ࣮ߦ • ࣮ߦ݁Ռ͸Google Cloud StorageʢGCSʣʹCSVͱͯ͠อଘ. ͜Ε͕ݯઘͷσʔλʢDatalakeʣ • GCS্ͷCSVΛαϚϦʔ͍͍ͯ͠ײ͡ʹͯ͠BigQueryʹอଘ͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied.

Slide 18

Slide 18 text

Firestore౤ೖʢDatabaseʹσʔλҠૹʣ • BigQueryσʔλΛμογϡϘʔυ༻σʔλͷܗࣜʢJSONʣʹม׵͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ • ࣮ߦ݁ՌʢGCS্ʹJSONܗࣜͰอଘʣΛFirestoreʹೖΕΔͨΊͷPythonεΫϦϓτΛ࣮ߦ • DataprocͱFirestoreͷॲཧ͸खݩͷεΫϦϓτΛΛखಈ࣮ߦʢ׬શࣗಈԽΛ્֐͢Δ੍໿͕͋ͬͨͨΊʣ This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied.

Slide 19

Slide 19 text

ӡ༻ͯ͠ͷৼΓฦΓ • ΞϓϦଆ͸Cloud Run & Cloud FunctionsͰ͍͍ײ͡ʹӡ༻Ͱ͖ͯΔ👏 
 અ໿ͷͨΊϦιʔεΛίʔϧυελϯόΠͷঢ়ଶͰӡ༻͍ͯ͠Δ͕ 
 ݸਓར༻ͳͷͰࢧো͸ͳ͍ʢ͔ͭinstance͸CIճͯ͠૿ݮͰ͖Δߏ੒ʣ • σʔλଆ͸ධՁ͕෼͔ΕΔ • Cloud FunctionsͱSchedulerͰͷϐλΰϥεΠονͳσʔλॲཧ͸˕ • BigQuery΋͍͍ײ͡, ूܭॲཧͳͲ΋ετϨεແ͘ߦ͚͍ͯΔ̋ • Dataproc serverlessΛࠓճͷن໛Ͱ࢖͏ͷ͸৑௕͔ͩͬͨ΋͠Εͳ͍ʁ

Slide 20

Slide 20 text

ࠓޙ͸ଟ෼͜͏ͳΔ (Ver 1.1 -> Ver 2.0)

Slide 21

Slide 21 text

This presentation makes reference to marks owned by third parties. Unless otherwise noted, all such third-party marks are the property of their respective owners. No sponsorship, endorsement or approval of this content by the owners of such marks is intended, expressed or implied.

Slide 22

Slide 22 text

ࠓޙ΍Ζ͏ͱࢥ͍ͬͯΔ͜ͱ • ETL͸Cloud Functions & Cloud SchedulerͷϐλΰϥεΠονʹ౷Ұ • Dataproc͸࣮ࡍ࢖ͬͨ݁Ռ, ࣗ෼ͷϢʔεέʔεʹ͸too much • γϯϓϧͳCSV͔ͭσʔλྔ΋গͳ͍ͷͰCloud FunctionsͰॲཧՄೳ • σʔλऩूɾॲཧϑϩʔͷ׬શࣗಈԽ, IaCʹΑΔΠϯϑϥ؅ཧ • BigQueryΛ࢖֤ͬͨछ౷ܭσʔλͷॆ࣮Խ • ࠓ͸΄΅ੜσʔλΛΫΤϦͯ͠Δ͚ͩʢඞཁʹԠͯ͡viewΛ࡞ΔͳͲʣ • Spark΋࢖͑Δ͠΋͏ͪΐͬͱؾͷར͍ͨDatamartΛॆ࣮ͤ͞Δ

Slide 23

Slide 23 text

ߨԋͷ·ͱΊ • ϝδϟʔϦʔάʹ͸Φʔϓϯσʔλ͕͋Γɺ 
 େ୩ᠳฏબखͳͲͷύϑΥʔϚϯε͕ 
 ֬ೝɾධՁͰ͖·͢ɻ • Φʔϓϯσʔλͷ෼ੳɾՄࢹԽΛ೔ৗతʹ 
 ׆༻͢ΔͨΊɺGoogle CloudͰ 
 σʔλ෼ੳج൫Λ࡞Γ·ͨ͠ɻ • αʔόϨεɾΞʔΩςΫνϟͷΈͰ 
 σʔλج൫ߏஙɾӡ༻͸࣮ݱՄೳɺ 
 ڧ͘Φεεϝ͍͖͍ͯͨ͠Ͱ͢ɻ ग़య: https://speakerdeck.com/shinyorke/pythonshi-inotamenosupotudetajie-xi-nokihon-pysparktomeziyarigudetawotian-ete-number-pyconjp-2022

Slide 24

Slide 24 text

ࠓճ͓࿩Ͱ͖ͳ͔ͬͨ͜ͱ • DataprocʢSparkʣͷৄ͍͠࿩. ಛʹServerlessΛ࢖ͬͨ݅. • Cloud Functionsୈೋੈ୅Λ࢖ͬͨόονॲཧͱόοΫΤϯυAPI Ͳ͔͜ͷػձͰ·ͨLTͳͲͰ͖ͨΒͱࢥ͓ͬͯΓ·͢👍

Slide 25

Slide 25 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠