Share
Embed
Python使いのためのスポーツデータ解析のきほん - PySparkとメジャーリーグデータを添えて #PyConJP 2022
PyCon JP 2022 10/15 Talk Session Material
# Reference
https://shinyorke.hatenablog.com/entry/baseball-data-visualization-app
https://shinyorke.hatenablog.com/entry/ohtani-san-pitch-2022
Shinichi Nakagawa PRO
October 15, 2022
Transcript
95 slides
-
No Baseball, No Engineering! High Performance Data Platform Knowledge of
PySpark, Cloud and ⚾ Python͍ͷͨΊͷεϙʔπσʔλղੳͷ͖΄Μ - PySparkͱϝδϟʔϦʔάσʔλΛఴ͑ͯ Shinichi Nakagawa@shinyorke 2022/10/15 PyConJP 2022 Talk Session -
Onboardingʢ͜ͷηογϣϯͷ͝Ҋʣ • PythonͱSparkʢPySparkʣͱύϒϦοΫΫϥυʢGoogle CloudʣͰ GBҎ্ͷσʔλΛ͍͍ײ͡ʹॲཧͯ͠ѻ͓͏ͥʂ, ͱ͍͏τʔΫͰ͢. • ༰తʹதڃऀʙ্ڃऀ͚Ͱ͢, ॳ৺ऀͷํͷࢦʹͳΔͱخ͍͠Ͱ͢. ʢ㲈Θ͔Βͳ͍ɾΒͳ͍͜ͱࣗ͝ͷʮ৳ͼ͠Ζʯͩͱࢥ͍ͬͯͩ͘͞ʣ
• σʔλͷࡐʮϝδϟʔϦʔάʯͰ͢⽁, εϙʔπσʔλͷগ͠. • ٿʹڵຯͳ͍ʢor͖͡Όͳ͍ʣํͱҰॹʹָ͠ΊͨΒ͍Ͱ͢. ࠓͷτʔΫΛ͖͔͚ͬʹٿʹڵຯͯΔΑ͏ͳΛؤுͬͯΓ·͢" -
օ༷ʹظ͢ΔલఏࣝͱϞνϕʔγϣϯ • ʲMustʳPandasSQLͰσʔλॲཧɾੳΛखΛಈ͔ͯͬͨ͜͠ͱ͕͋Δ. • ʲMustʳGoogle CloudʢGCPʣ, AWS, AzureͳͲͷPublic CloudͰ PythonΛͬͨ͜ͱ͕͋Δ.
※αʔϏεΘͣʢEC2, App Engine, etc…ʣ • ϑϧϚωʔδυͷαʔόϨεڥͰͷ։ൃܦݧʢ৮ͬͨ͜ͱ͋ΕOKʣ. AWS Lambda, AWS App Runner, App Engine, Cloud RunͳͲ͕֘. • ʢ͖ݏ͍ؔͳ͘ʣٿͷϧʔϧͱΦΦλχαϯѲ͍ͯ͠Δ. -
Who am ɹ? ʢ͓લ୭Α?ʣ • Shinichi Nakagawa@shinyorke • େख֎ࢿITίϯαϧاۀϚωʔδϟʔ ʢݩɾࣄۀձࣾͷϑϧαΠΫϧΤϯδχΞʣ
• ΫϥυΠϯϑϥΛѻ͏νʔϜͷϚωʔδϟʔ • झຯͱ࣮ӹΛ݉Ͷͯݸਓ։ൃͯ͠·͢# ʢओʹٿͱϑΟδΧϧέΞతʣ • ٿͱҿΈͳ͕Βͷϓϩάϥϛϯάେ͖. • ਪ͠: ৽ঙ߶ࢤ, ສதਖ਼, ୩ݪ݈ଠʢͷڧݞʣ #Python #Serverless #GoogleCloud #Baseball #DataScience #SABRmetrics -
ϝδϟʔϦʔάͷϏοάσʔλ • ϝδϟʔϦʔάʮStatcastʯͱ͍͏γεςϜͰ৭ΜͳσʔλΛه͍ͯ͠·͢. ※ΧϝϥɾϨʔμʔͰه, Ұ෦౷ܭɾਓྗͰه • ྫ͑, ͜ͷลͷ࣮گͷݩωλͯ͢͜ͷʮStatcastʯͱ͍͏Ϗοάσʔλ͕ݩωλʹͳ͍ͬͯ·͢. • ΦΦλχαϯʂ˓߸ຊྥଧʂଧٿ180km/h,
ඈڑ130m • ΦΦλχαϯʂ162km/hͷਅ͙ͬͰݟಀ͠ࡾৼʂʂʂ • ٿͷҰڍखҰ, ͯ͢ͷٿɾଧٿσʔλ͕ه͞ΕΔ. • ϨΪϡϥʔγʔζϯʢ30νʔϜɾ162ࢼ߹ʣͰ͓͓Αͦ70ʙ80ສٿલޙ. ϙετγʔζϯɾय़Ωϟϯϓσʔλ͋Δ. • σʔλ91ݸͷ߲ʢ!?ʣͰߏ͞ΕΔ, ϨΪϡϥʔγʔζϯͰ͓͓Αͦ400MBʙ600MB͙Β͍ͷσʔλ. • baseballsavant.mlb.com ͱ͍͏αΠτͰ୭ͰӾཡɾμϯϩʔυʢCSV ϑΥʔϚοτʣͰ͖·͢. -
2022ͷΦΦλχαϯ, εϥΠμʔͱ2γʔϜ, ΧοτϘʔϧܑ͞ΜʹͳΔ • ࠓͷΦΦλχαϯ, ΊͬͪΌ εϥΠμʔ͍͛ͯΔ • ͓ؾ͖ͮͩΖ͏͔?ޙઓ 2γʔϜʢσʔλ্Sinkerʣ͕
૿͍͑ͯΔ͜ͱʹ!? • εϥΠμʔ, 2γʔϜ, ΧοτϘʔϧͰ บ͕ڧ͍ۂ͕Γٿ͛ΔϚϯʹΩϟϥม -
StatcastσʔλΛJupyter Lab + PlotlyͰோΊΔ • ৭Μͳσʔλ͕͋ΔͷͰ݁ߏͳ͜ͱ͕Θ͔Γͦ͏. • ࣌ܥྻσʔλͳͷͰ, ύϑΥʔϚϯεͷมԽ͔ͭΊΔ. •
ٿ͕લͱҧ͏ͱ͔, ಥવ2γʔϜ૿͑ͨͳ?ͱ͔. • Ϙʔϧͷɾ࠲ඪܥσʔλ͕ἧ͍ͬͯΔ. • ؤுͬͯ࠲ඪܭࢉ͍͍ͯ͠ײ͡ʹͨ͠Β3Dඳըͱ͔͍͚Δ. ʢҙ༁ɾࠓճؤுΔ༨༟ͳ͔ͬͨͷͰͬͯ·ͤΜ$ʣ -
ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ • ຖσʔλ֬ೝɾຖσʔλߋ৽Λ͍͍ײ͡ʹ࣮ݱ͢ΔͨΊ, ʮϑϧϚωʔδυͳαʔόϨεܥΫϥυαʔϏεʯΛશ໘తʹ׆༻ͯ͠ߏஙɾӡ༻. • ʮϑϧϚωʔδυͳαʔόϨεܥΫϥυαʔϏεʯ #ͱ • CLIίϯιʔϧͰϙνϙν͢Δ͚ͩͰͻͱ·্ཱ͕ͣͪΔ •
Πϯϑϥɾαʔόʔͷϝϯςφϯε͕ෆཁʢࣗ͡Όͳͯ͘, ΫϥυαʔϏεଆ͕Δʣ • ΑΓ۩ମతʹ, ࣗͰK8sΫϥελVMΛݐͯͳͯ͘ྑ͍ʢωοτϫʔΫͷઃఆൃੜʣ • GitHub ActionsͷCI/CDͷύΠϓϥΠϯʹΈࠐΜͰσϓϩΠɾεέʔϧͰ͖ͨΓ جຊతʹʮ͚ͬͨͩ՝ۚʯʹͳΔͷͰ͓ࡒʹ༏͍͠% -
σʔλऩू&BigQueryอଘ • σʔλݩαΠτʢBaseball Savantʣ͔Βఆظతʹσʔλऩू͢ΔΫϩʔϥʔʢCloud Functionsʣ࣮ߦ • ࣮ߦ݁ՌGoogle Cloud StorageʢGCSʣʹCSVͱͯ͠อଘ. ͜Ε͕ݯઘͷσʔλʢDatalakeʣ
• GCS্ͷCSVΛαϚϦʔ͍͍ͯ͠ײ͡ʹͯ͠BigQueryʹอଘ͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ -
SparkͱPySpark • ʮେ͖͍σʔλΛ͍͍ײ͡ʹࢄͯ͠ॲཧ͢ΔʯͨΊͷFramework • Sparkຊମͷ࣮Java͕ͩ, PythonͷInterfaceͰ͋ΔʮPySparkʯΛ ͏ࣄ͕ଟ͍ʢଞͷݴޠͩͱR͑ͨΓ͢Δʣ. • σʔλॲཧόονͷϓϩάϥϜͱͯ͠ಈ͔͢ or
Jupyter Lab, ZeppelinͰnotebook࣮ߦ. • Python͍ʹඇৗʹೃછΈ͕ਂ͍, DataFrameͳInterface͕͋Δ. • SparkಠࣗͷDataframe. ͪͳΈʹPandas DataframeʹมՄೳ • Pandas APIʢSpark 3.2Ҏ߱ͰPandasͷػೳΛར༻Մೳ, Ұ෦੍༗Γʣ -
SparkΛͲ͜Ͱߏஙɾӡ༻͢Δ͔' ڥɾखஈ ߏஙͷखؒ ӡ༻͢͠͞ උߟ ΦϯϓϨϛεͰ શͯࣗલߏஙɾӡ༻ શͯࣗલͰઃఆ͢Δ ඞཁ͕͋Δ Կ͔ΒԿ·Ͱ
ࣗͰݟΔඞཁ͕͋Δ Ұ൪େมͳύλʔϯ ຊ৬ͷΠϯϑϥΤϯδχΞ Ͱ͖͍ͭࣄ Ϋϥυ্ͷ7.,Tʹ ࣗલͰߏஙɾӡ༻ શͯࣗલͰઃఆ͢Δ ඞཁ͕͋Δ ͋ΔఔΫϥυαʔϏε ͷԸܙʹत͔ΕΔ 4QBSLڥͷࣗલߏங ׂͱқ͕ߴ͍ ΫϥυαʔϏεఏڙͷ ϚωʔδυαʔϏεΛ͏ ˞࠷ਪ͢Δํࣜ (6*Ͱϙνϙν͢Δ ͘͠$-*"1*Ͱ ͍͍ײ͡ʹ࣮ߦ $16ͷϦιʔεΛࢹ ঢ়گʹԠͯ͡ϝϯςφϯε ࠷ָ͔ͭεϚʔτͳํ๏ "84 (PPHMF$MPVEଞ ֤ࣾαʔϏε༗ -
Google Cloudʹ͓͚ΔSparkӡ༻ͷબࢶ ڥɾखஈ ߏங ӡ༻ ͑Δػೳ උߟ ($&PS(,&ʹ ڥΛ࡞ͬͯӡ༻ ࣗલͰߏஙޙ
4QBSLΛಋೖ શͯࣗલͰӡ༻ ໘ΛݟΔඞཁ༗ શͯͷػೳ ݁ہͷॴ%BUBQSPDͰ ग़དྷΔ͜ͱͳͷͰ ͓͢͢Ί͠ͳ͍ %BUBQSPD HDMPVEίϚϯυ "1* ίϯιʔϧͷ ͲΕ͔Ͱߏங %BUBQSPD͕࡞ͬͨ (,&PS($&ڥ Λࢹɾӡ༻ શͯͷػೳ Ұ൪ඪ४తͳߏ %BUBQSPD 4FSWFSMFTT HDMPVEίϚϯυ "1* ίϯιʔϧ ্هͷͲΕ͔Ͱߏங ࣮ߦதͷࢹͷΈ ڥॲཧޙʹ ࣗಈআ όονॲཧͷΈରԠ OPUFCPPL͑ͳ͍ ఆظతͳόονॲཧ ͜Ε͕Ұ൪͍͍ ※Spark in BigQueryͱ͍͏, BigQueryͷετΞυͱͯ͠SparkΛ࣮ߦ͢Δػೳͷఏڙ༧ఆ༗Γʢby Google Cloud Next ‘22ʣ -
DataprocͱDataproc Serverless • Google CloudʹDataprocͱ͍͏SparkʢHadoopʣͷϚωʔδυαʔϏε͕ଘࡏ͢Δ. • ࠓ·ͰGCEGKEʢK8sʣͰʮϗετ͘͠Cluster͕ଘࡏʯલఏͷ ӡ༻͔͠Ͱ͖ͳ͔͕ͬͨ, ͍ͭ࠷ۙServerlessͱ͍͏બࢶ͕ര •
ʮ11ճʯʮ30͓͖ʯΈ͍ͨͳόονӡ༻Ͱ͋ΕServerless͕͑Δʂ ͳ͓, notebookͷ࣮ߦʢJupyterͳͲʣະରԠͳͷͰΞυϗοΫʹ͑ͳ͍. • Serverless͚ͬͨͩ՝ۚͳͷͰ͓ࡒʹ༏͍͠% • όʔδϣϯSpark 3.2, PySpark͔ΒPandas API͑·͢ʢ͕ࠓճͬͯ·ͤΜʣ. -
ʲ࠶ܝʳσʔλऩू&BigQueryอଘ • σʔλݩαΠτʢBaseball Savantʣ͔Βఆظతʹσʔλऩू͢ΔΫϩʔϥʔʢCloud Functionsʣ࣮ߦ • ࣮ߦ݁ՌGoogle Cloud StorageʢGCSʣʹCSVͱͯ͠อଘ. ͜Ε͕ݯઘͷσʔλʢDatalakeʣ
• GCS্ͷCSVΛαϚϦʔ͍͍ͯ͠ײ͡ʹͯ͠BigQueryʹอଘ͢ΔPySparkεΫϦϓτΛDataproc Serverless্Ͱ࣮ߦ -
σʔλऩू ʢnot Sparkʣ • WebεΫϨΠϐϯάSparkͰ Δ͖͜ͱͰͳ͍. • λεΫΛrequests-htmlͰ࣮, Cloud FunctionsͰӡ༻ͯ͠ରॲ.
• Cloud SchedulerͷCronઃఆͰ ఆظ࣮ߦ, GCSʹอଘ -
CSVσʔλΛ BigQueryʹೖ • Dataproc্ͰΔλεΫͱͯ͠ దͳൣғɾॲཧͷҰͭ • GCSͷύε͔ΒϑΝΠϧநग़ Spark SQLͰॲཧͯ͠BigQuery •
DataFrameͱSQL͕Θ͔Ε ͍͍ײ͡ʹ࣮ɾӡ༻Մೳ -
DataprocΛ͓͏ • Google CloudͷυΩϡϝϯτɾαϯϓϧΛࣸܦ͠ͳ͕ΒΔͱྑ͖ • https://cloud.google.com/dataproc • https://cloud.google.com/dataproc-serverless/docs • https://github.com/GoogleCloudDataproc/cloud-dataproc
• Serverlessͷ߹, ࣄલʹVPC subnetΛ࡞, ࣮ߦ࣌ʹࢦఆ͢Δඞཁ͋Γ. • ࣍ϖʔδ͔Β, PySparkΛͬͯΔ߹ͷαϯϓϧΛগ͠հ͠·͢. • Spark DataFrameΛݩʹ, ʮσʔλΛಡΜͰՃͯ͠ॻ͖ࠐΈʯతͳόονॲཧ. • ͲͷΫϥε͔Θ͔Γ͘͢͢ΔͨΊ, Type Hints͖Ͱ࣮͍ͯ͠·͢. -
ͻͱ·࣮ͣ 1. SessionΛ࡞Δ • DB connectionతͳͭ • SparkSessionͷObjectΛ࡞Δ • Object࡞࣌ʹParameterࢦఆ
• BigQueryΛ͏࣌ JARͷࢦఆ͕ඞਢͳͷͰҙ -
ͻͱ·࣮ͣ 2. SchemaΛ࡞Δ • CSVͷ߹SchemaΛ࡞Δ • ࡞͞ΕΔDataframeʹ ܕΛ͚ͭΔҝ, ઈରඞཁ •
ࠓճ91߲ͷSchema ؤுͬͯॻ͖·ͨ͠ྦ -
ͻͱ·࣮ͣ 5. BigQueryಡࠐ • อଘͱಉ͘͡BigQueryͷ JARΛࢦఆ • spark readͰBigQueryΛࢦఆ •
BigQueryͷViewʹରͯ͠ ߦ͏߹, Φϓγϣϯ͕ඞཁ -
PySparkͱDataproc Serverless • ʮ͍͍ͨͱ͖͚ͩSparkΛ͏ʯͱ͍͏ϢʔεέʔεΛ࣮ݱՄೳ. ͜Ε͕αʔόϨεαʔϏεΛ͏͖࠷େͷཧ༝. • ࠓճͷΞϓϦέʔγϣϯͷσʔλαΠζʢ1Ͱ1GB͍͔ͳ͍ʣͩͱ Ըܙʹत͔Εͳ͍͕, ʮGB/ఔͷσʔλΛαΫοͱόονॲཧʯ Έ͍ͨͳϢʔεέʔεʹͳΔͱ݁ߏศརͳؾ͕͠·͢ʢલॲཧɾΫϨϯδϯά͢Δͱ͔ʣ.
• ʮॲཧ͢Δͱ͖͚ͩಈ͔͢ʯͱ͍͏ײ͡ͷ͍ܰίʔυͳͷͰPySparkͱ૬ੑόπάϯ. • ͳ͓, ॲཧͷࣗಈԽͪΐͬͱบ͕͋Γ·͢, Cloud ComposerʢAirflowʣ͕ඞཁ. ※ৄࡉࢿྉͷAppendixΛࢀর -
2022ϓϩٿ, άοͱ͖ͨग़དྷࣄBEST 5 1. ϑΝΠλʔζ, ສɾਗ਼ٶɾాٶΒ, ମೳྗ༏ΕΔएख͕಄ 2. FIGHTERS GIRL
2022, ΩϨοΩϨͷΩπωμϯε͕େώοτ ύɾϦʔάTVͷಈը࠶ੜͰଟͷ࢝ٿࣜಈըʹѹউ 3. ύϫʔͱڧݞ, ढ़कͰҰ࣌Λங͍ͨࢳҪՅஉ, ཹհͷҾୀ 4. ٿʹډ࠲Δੜͷௗ, όοτΛৼΓճͨ͠ઍ༿ϩος֯த֎खʹෛ͚Δ 5. ࠤʑ࿕رશࢼ߹, ଜ্फོࡾףԦ + ຊྥଧه, Τά͘ͳ͍Ͱ͔͢? -
2022ϓϩٿ, άοͱ͖ͨग़དྷࣄBEST 5 1. ϑΝΠλʔζ, ສɾਗ਼ٶɾాٶΒ, ମೳྗ༏ΕΔएख͕಄ 2. FIGHTERS GIRL
2022, ΩϨοΩϨͷΩπωμϯε͕େώοτ ύɾϦʔάTVͷಈը࠶ੜͰଟͷ࢝ٿࣜಈըʹѹউ 3. ύϫʔͱڧݞ, ढ़कͰҰ࣌Λங͍ͨࢳҪՅஉ, ཹհͷҾୀ 4. ٿʹډ࠲Δੜͷௗ, όοτΛৼΓճͨ͠ઍ༿ϩος֯த֎खʹෛ͚Δ 5. ࠤʑ࿕رશࢼ߹, ଜ্फོࡾףԦ + ຊྥଧه, Τά͘ͳ͍Ͱ͔͢? -
Statcast ʢ&ࢲʣ͕ਪ͢ ʮΤά͍֎खͨͪʯ • ମೳྗ͓Խ͚ͰΩϨοΩϨ • ύϫʔ, ڧݞ, कͦͯ͠٭͕ചΓ •
όοτΛৼΓճ͢ੑͬΆ͞ • ଧٿΛݩʹਪ͠Λ3ਓհ • ݱ࣌ͷ৽ঙ߶ࢤͬΆ͍ ӉਓૉΒ͍͠֎खͰ͢( -
ຊ͝հ͢ΔΤά͍֎खͨͪ • Judge, AaronʢΞʔϩϯɾδϟοδʣ • Rodríguez, JulioʢϑϦΦɾϩυϦήεʣ • Buxton, ByronʢόΠϩϯɾόΫετϯʣ
300ଧ੮Ҏ্ཱ͍ͬͯΔ֎ख͔ͭ, ଧٿ͕ͯ͘ଧ͕ग़·͘Δ, ݪଇηϯλʔΛक͍ͬͯΔબखΛ3ਓհ͠·͢. -
ϑϦΦɾϩυϦήε ʢγΞτϧظͷʣ • ϚϦφʔζʹᰜͱݱΕͨظͷ ͪͳΈʹࠓͷϧʔΩʔ • एख࣌ͷBIG BOSSΈ͍ͨͳ ମೳྗΛੜ͔ͨ͠ϓϨʔ͕ັྗ •
ଧٿ্͕͕֯ͬͯόϨϧ૿͑ͨΒ Πνϩʔࢯʹগͣͭۙͮ͘͠ͷͰ? 10ܖʹԠ͑Δ׆༂Λظʂ -
όΠϩϯɾόΫετϯ ʢϛωιλͷສʣ • ϛωιλɾπΠϯζෆಈͷηϯλʔ • ٿ͡Όͳ͍ڝٕߦ͚ͦ͏? ͱ͍͏Τήπͳ͍٭ྗͱݞͷ࣋ͪओ, ͦͷׂʹଧٿ͕֯ύϫʔώολʔ • ৭ʑࡶͬΆ͍ॴͱελΠϧͷྑ͞Ͱ
ສதਖ਼ʢϑΝΠλʔζʣʹࣅ͍ͯΔ. Ϛϯνϡ, ͷόΫετϯʹͳͬͯ͘Εʂ -
ʲ࠶ܝʳຊͷελʔςΟϯάϝϯόʔ • ϝδϟʔϦʔάͷϏοάσʔλͰ༡΅͏ • PythonͱGoogle CloudͰ࡞ΔαʔόϨεͰ͍͍ײ͡ͳσʔλج൫ • PySpark + DataprocͰ࣮ݱ͢ΔαʔόϨεͳσʔλॲཧ
• Ϗοάσʔλ͕ਪ͢ʮΤά͍ΞεϦʔτܥ֎खʯ ָ͓͠Έ͍͚ͨͩ·͔ͨ͠?ใྔ͕ଟ͔ͬͨͷͰཧղ͢Δ·Ͱ͍͔͠ ࢿྉެ։͠·͢ͷͰ, ͥͻৼΓฦΓͱ͓ͯ͠ಡΈ͍ͩ͘͞) -
ࠓͷΛཁ͢Δͱ… • εϙʔπσʔλͷղੳɾੳͷ͓ͱͯ͠ٿ໘ന͍Αʂ Baseball Savantͱ͍͏τϥοΩϯάσʔλΛ͏ͱྑ͖. • PythonͰ͍͍ײ͡ʹσʔλॲཧΛ͢ΔͷʹPySparkྑ͍ͧ. • PySparkΫϥυͰಈ͔ͤ·͢, ࠓDataprocΛհ͠·ͨ͠.
• αʔόϨεʹΫϥυΛ͑ΔΑ͏ʹͳΔͱ,৭ʑͱָʢ੍ͨͩ͠ݶ͋Δʣ. • ϝδϟʔΤά͍֎ख͕͍Δ͕, εϥΠμʔͱ2γʔϜ͓Խ͚ͷΦΦλχαϯڧ͍. -
͓ࣄʢۀʣͰࢀߟʹ͠Α͏ͱࢥͬͨํ • ࠓճհͨ͠ΓํɾߏઈରతͳճɾϕεϓϥͰͳ͍Ͱ͢. ྫ͑αʔόϨεɾΞʔΩςΫνϟʹ͖͢/͖͢͡Όͳ͍ঢ়گ࣮֬ʹଘࡏ͠·͢. • ͜ͷࢲʢshinyorkeʣ͕Γ͍ͨࣄ, ͍͍ͱࢥͬͯΔࣄʢ&৮Γ͍ͨϞϊʣΛ ٧ΊࠐΜͰ࡞ͬͨ, ͕ࣗΓ͍ͨࣄͷूେͰ, ͋͘·Ͱͷग़͠ํͷҰͭͰ͢.
• ͬͱݴ͑, ʮαʔόϨεͱ͔PySparkͰͲ͜·Ͱ͍͍ײ͡ʹ࡞ΕΔ?ʯͱ͍͏ ϓϩτλΠϓͱͯ͠࡞Γ·ͨ͠&࣮ࠓޙSpark֎ͭ͢ΓͰ͢ʢৄ͘͠Appendixʹͯʣ. • ʢίϯςΩετͷཧղ͕த్ͳ··ʣͦͷ··ਅࣅ͢Δͱരࢮ͠·͢. ·ͣखΛಈ͔͠, ֶशͨ͠Γಈ͔ͨ͠Γ͍͍ͯ͠ͷΛݟ͚ͭΔࢀߟʹͲ͏ͧʂ -
ʲଓ͖ʳAppendix - ͏ͪΐͬͱৄ͍͠ • Dataproc ServerlessΛࣗಈ࣮ߦ͢Δ • AWSͳͲଞͷΫϥυͷSparkͳαʔϏεࣄ2022 • SparkΛΘͳ͍,
େ͖Ίͳσʔλॲཧͷ͖΄Μ for Google Cloud • Dash + Cloud RunΛ͍͍ͬͯײ͡ʹσʔλՄࢹԽΞϓϦΛ࡞Δ ؾʹͳΔํࢿྉͷଓ͖ΛಡΜͰ&ձͷํ࣭ٙԠͰ͠·͠ΐ͏ʂ -
Appendix - ͏ͪΐͬͱৄ͍͠ • Dataproc ServerlessΛࣗಈ࣮ߦ͢Δ • AWSͳͲଞͷΫϥυͷSparkͳαʔϏεࣄ2022 • SparkΛΘͳ͍,
େ͖Ίͳσʔλॲཧͷ͖΄Μ for Google Cloud • Dash + Cloud RunΛ͍͍ͬͯײ͡ʹσʔλՄࢹԽΞϓϦΛ࡞Δ • ࢀߟࢿྉ -
Dataproc ServerlessͷࣗಈԽ • ఆ͞ΕΔखஈҎԼͷ3ͭ. 1. APIΛ͍͍ͬͯײ͡ʹ࣮ߦ͢ΔҝͷDocker imageΛ࡞ ͜ΕΛԿ͔͠Βͷํ๏ͰContainerͱͯ͠ಈ͔͢ʢK8sͳͲʣ 2. CLIʢgcloudίϚϯυʣͰ࣮ߦͰ͖ΔͷͰ,
gcloudίϚϯυͷ Docker imageΛ࡞ʢҎԼ, 1.ͱಉจʣ 3. AirflowͷOperatorΛͬͯDataproc ServerlessΛಈ͔͢ • 1.ͱ2.ۤߦ͔ͭ, αʔόϨεͷҙຯ͕ແ͘ͳΔՄೳੑ͕͋Δʢ͔ͭ, 1ͱ2΄΅ಉ͜͡ͱΛݴ͍ͬͯΔʣ. Cloud RunͰಈ͔ͤΕΑ͍͕, ߏஙɾӡ༻ͱʹϦεΫ͕͋Γͦ͏ͳ༧ײ. • ϕεϓϥͬΆ͍ൣղʮ3.AirflowͷOperatorΛͬͯDataproc ServerlessΛಈ͔͢ʯͰ͢. -
Dataproc ServerlessͷॲཧࣗಈԽ • Google CloudͰcronతͳॲཧΛࣗಈԽ͢ΔͷʹPub/Sub + Schedulerʢ͘͠Cloud TaskʣΛ ͏ͱ͍͏ϕετɾϓϥΫςΟε͕ଘࡏ͢Δ. •
ͨͩ, 202210݄ݱࡏ, Dataproc ServerlessPub/SubΛInterfaceͱ࣮ͯ͠ߦ͢Δखஈ͕ແ͍ҝ, ೦ͳ͕Β͜ͷํ๏͑ͳ͍. • ͳͷͰ, ࠷εϚʔτͳํ๏AirflowͷDataprocܥOperatorΛ࣮ͬͯߦ͢Δ͜ͱʹͳΔ. Cloud ComposerͰAirflowΫϥελΛ্ཱͪ͛ͯӡ༻͢Δ. • https://cloud.google.com/composer/docs/composer-2/run-dataproc-workloads • ͪͳΈʹCloud ComposerαʔόϨεͰͳ͍Ͱ͢ʢϑϧϚωʔδυͰ͋Δ͕ʣ &K8sʢGKEʣΫϥελΛཱͯΔ͜ͱʹͳΔͷͰίετ໘ҙʢ࣮ͱ͔͘ݸਓͰ͏ʹߴ͍ʣ -
Google CloudҎ֎ͷSparkαʔϏεબࢶ • AWS, Azureͦͯ͠ʢ͋Δҙຯ͝ຊՈͰ͋ΔʣDatabricks͕ީิ. • ύϒϦοΫΫϥυΛΠϯϑϥͱͯ͠ѻ͏Ϣʔεέʔεͷ߹, Databricks͕࠷༗ྗީิʹʢϚϧνΫϥυԽ͍ͨ͠ͷέʔεʣ. • ࣮͜ͷ,
AWS͕ॆ࣮͍ͯͯ͠, EMRͱGlueͰϢʔεέʔεʹ ߹Θͤͯબ͢Δͱ͍͍Α͏ͳؾ͕͢Δ. • Azure৮ͬͨ͜ͱແ͍ͷͰΘ͔Βͳ͍…* -
Google CloudҎ֎ͷSparkαʔϏεબࢶ ΫϥυαʔϏε ˞શͯͰͳ͍Ͱ͢ 63- ֓ཁ %BUBCSJDLT IUUQTXXXEBUBCSJDLTDPNKQ ϚϧνΫϥυఆͩͱબࢶʹ 4QBSLͷੜΈͷ͕։ൃɾఏڙ
"84&.3 IUUQTBXTBNB[PODPNKQFNS "84ͷϚωʔδυ4QBSL)BEPPQ 4QBSLͱͯ͠͏ͳΒͬͪ͜ "84(MVF IUUQTBXTBNB[PODPNKQHMVF 4QBSLΛ&5-ͱͯ͠͏߹ &.3ΑΓ(MVFΛ͏ͷ͕ϕετ "[VSF)%*OTJHIU IUUQTB[VSFNJDSPTPGUDPNKBKQ TFSWJDFTIEJOTJHIUPWFSWJFX "[VSFʹ͓͚Δબࢶ ʢࢲ৮ͬͨ͜ͱͳ͍Ͱ͕͢ʜ -
͍͍ײ͡ͳσʔλॲཧ for Google Cloud • Dataflow • DataFusion • Dataprep
• Cloud Run • Cloud Functions -
༻్ʹ߹Θ͍͚ͤͯ·͠ΐ͏ʂ (PPHMF$MPVE4FSWJDF 63- ֓ཁ %BUBqPX IUUQTDMPVEHPPHMFDPNEBUBqPX IMKB "QBDIF#FBN͕ϕʔε ετϦʔϛϯάॲཧͳΒ͜Ε %BUB'VTJPO
IUUQTDMPVEHPPHMFDPNEBUB GVTJPOEPDT IMKB ΦϯϓϨΛؚΉɺطଘσʔλΛ औΓࠐΉ&5-తͳαʔϏε %BUBQSFQ IUUQTDMPVEHPPHMFDPNEBUBQSFQ IMKB σʔλલॲཧɾΫϨϯδϯάத৺ ͲͪΒ͔ͱ͍͑ϩʔίʔυ $MPVE3VO IUUQTDMPVEHPPHMFDPNSVO IMKB ͖ͳݴޠɾ'8Ͱ࡞ΔͳΒ͜Ε 1VC4VCͰτϦΨʔͯ͠ಈ͔͢ $MPVE'VODUJPOT IUUQTDMPVEHPPHMFDPNGVODUJPOT IMKB $MPVE3VOΑΓ੍͋Δ͕ αΫοͱ࡞ͬͯಈ͔͢ͳΒ -
ݱ࣮తͳબࢶɾצॴ • ϦΞϧλΠϜܥͷॲཧDataflow͕࠷༗ྗͷબࢶ. • طଘͷσʔλͱ౷߹ͨ͠Γ·ͱΊͨΓDataFusion. • ػցֶशͷσʔλલॲཧDataprep. • PythonʹݶΒͣ, ࣗͰ࡞ͬͯಈ͔͢ͳΒCloud
Run. • ʮPandasͱBigQuery, GCS͏ʯ͙Β͍ͳΒCloud FunctionsͰ αΫοͱΕ·͢ʢ࣮͜ͷϢʔεέʔεଟ͍ͷͰʁʣ. -
Dash + Cloud RunͰͷ ϗεςΟϯά • DashFlask͕ݩʹͳͬͯΔͷͰ gunicornͰ͍͍ײ͡ʹಈ͔͢తͳ ํ๏ͰϗεςΟϯάՄೳ. •
͜ΕαʔόϨεͳͷͰ, ͬͨ࣌ؒɾϦιʔε ͚ͩಈ͘ײ͡ʹͳΔ, ࣗલͰՄࢹԽΞϓϦΛ ࡞Γ͍ͨํͬͯΈΔͱྑ͍͔? • ͪͳΈʹAWSͷ߹, App RunnerͰಉ͡ ํ๏͕औΕΔͱࢥ͍·͢ʢࢼ͍ͯ͠·ͤΜ͕ʣ. -
ͳ͓, CI/CDϫʔΫϑϩʔ͜Μͳײ͡. • GitHub Repositoryʹpushͨ͠ΒGitHub Actions͕ൃՐ, ςετ -> Docker Build
-> Cloud RunσϓϩΠ • ςετpytest, flake8, mypyΛGitHub Actions্Ͱ࣮ࢪʢunit, integration·Ͱ୲อ͢ΔΠϝʔδʣ • Docker buildCloud Runͷඪ४తͳΓํʹै͏. • Cloud Build্ͰϏϧυ • Artifact Registryʹpush • Cloud RunͷσϓϩΠGithub ActionsͷެࣜΛ࣮ͬͯࢪ. -
Spark / PySparkؔ࿈ • PySpark Documents https://spark.apache.org/docs/latest/api/python/ • ೖPySparkɹ˞ͪΐͬͱݹ͍ॻ੶Ͱ͢, ༰ͱ͔ҙ.
https://www.oreilly.co.jp/books/9784873118185/ • PythonͰେྔσʔλॲཧʂ PySparkΛ༻͍ͨσʔλॲཧͱੳͷ͖΄Μ ʢPyCon JP 2017ʣ https://speakerdeck.com/chie8842/pythondeda-liang-detachu-li- pysparkwoyong-itadetachu-li-tofen-xi-falsekihon -
Google CloudʢDataprocʣ • ެࣜυΩϡϝϯτ https://cloud.google.com/dataproc/docs?hl=ja • PySparkͷެࣜαϯϓϧʢ͔͜͜Βࣸܦָ͕ʣ https://github.com/googleapis/python-dataproc • ެࣜαϯϓϧͦͷ2ʢΑΓ࣮ફతʣ
https://github.com/GoogleCloudDataproc/cloud-dataproc -
Google Cloudʢॳ৺ऀɾ͍͍ͨਓ͚ʣ • ެࣜυΩϡϝϯτ https://cloud.google.com/docs?hl=ja • ࢿ֨ https://cloud.google.com/certification?hl=ja • ΤϯλʔϓϥΠζͷͨΊͷGoogle
Cloudʢਪ͠ͷॻ੶Ͱ͢ʣ https://www.shoeisha.co.jp/book/detail/9784798175256 -
ࣗͷϒϩάهࣄʢPySpark/Dataؔ࿈ʣ • ٿͷϏοάσʔλΛGCPͱPySparkͰ͍͍ײ͡ʹ͍ͯ͘͢͠Έͨ https://shinyorke.hatenablog.com/entry/dataproc-baseball • SparkΛαʔόʔཧͤͣʹ͏ํ๏ https://shinyorke.hatenablog.com/entry/dataproc-serverless • Google CloudͰSparkΛ͏ڥΛαΫοͱखʹೖΕΔ
https://shinyorke.hatenablog.com/entry/dataproc-terraform • WebΞϓϦͱσʔλج൫ΛαΫοͱ্ཱͪ͛ΔͨΊͷϓϥΫςΟε https://shinyorke.hatenablog.com/entry/cloud-arch-serverless -
ٿؔͷࢀߟϒϩάɾίʔυ • ٿ͖ͱσʔλ͖ͷͨΊͷStatcastσʔλೖ https://shinyorke.hatenablog.com/entry/statcast-csv-docs-ja • StatcastσʔλͱPlotlyΛͬͯʮଧٿͷ౸ୡҐஔʯΛՄࢹԽ͢Δ https://shinyorke.hatenablog.com/entry/statcast-visualization-for-batting • Baseball SavantͰΦΦλχαϯͷσʔλΛோΊΔαϯϓϧ
https://github.com/Shinichi-Nakagawa/baseball-savant-shohei-ohtani2022 • RʹΑΔηΠόʔϝτϦΫεೖ https://gihyo.jp/book/2020/978-4-297-11684-2