Developers Summit 2023登壇資料
https://event.shoeisha.jp/devsumi/20230209/session/4196/
ਪ͠ਪͤΔ࣌ʹਪͤ.ͱ͍͏ԶͷDXΛ࣮ݱ͢ΔͨΊͷαʔόϨεͳσʔλج൫։ൃͱӡ༻.ٿσʔλΛఴ͑ͯ⽁@shinyorke 2023/02/10 Developers Summit 2023
View Slide
Onboardingʢ͜ͷηογϣϯͷ͝Ҋʣ• ࢲ͕झຯͰߏஙͨ͠ʮٿσʔλϓϥοτϑΥʔϜʯαʔϏεʹ͓͚Δ, ʮΫϥυωΠςΟϒͳσʔλΤϯδχΞϦϯάʯࣄྫͷհͰ͢.• Ϋϥυɾσʔλج൫Ͱඇৗʹ෯͍ൣғͷࣄྫͱͳΓ·͢. ࢿྉൃදޙʹެ։͠·͢&ؾʹͳΔՕॴࢿྉΛಡΉࣄΛਪ͠·͢.• ٿʹڵຯͳ͍ʢor͖͡Όͳ͍ʣํͱҰॹʹָ͠ΊͨΒ͍Ͱ͢.
αʔϏεͷߏஙͰͬͨϓϩάϥϛϯάݴޠɾαʔϏεɾؔ࿈ٕज़ͷʑΊͬͪΌଟ͍ͷͰؾʹͳΔͱ͜Ζ͚ͩಡΉɾ֮͑ΔͰେৎͰ͢ʂ
͜ΜͳํʹΦεεϝͰ͢.• DWHσʔλج൫ͱ͍ͬͨʮσʔλར׆༻ʯͷ͍͍ײ͡ͳߏஙࣄྫΛΓ͍ͨํ.• ύϒϦοΫΫϥυ, ओʹGoogle CloudΛͬͯԿ͔͠ΒͷࣄΛ͢Δํ.• ΫϥυαʔϏεΛ͍͍ͬͨײ͡ͳγεςϜઃܭɾߏஙʹڵຯ͕͋Δํ.• αʔόϨεͳΫϥυαʔϏεΛ͍ͬͯΔ or ͜Ε͔Β͏ํʢڵຯͰ͍͍ʣ. ※AWS Lambda, AWS App Runner, Google App Engine, Cloud RunͳͲ͕֘.• ʢ͖ݏ͍ؔͳ͘ʣٿͷϧʔϧٴͼΦΦλχαϯ͕୭͔͍ͩͬͯΔ.
Who am I ?ʢ͓લ୭Α?ʣ• Shinichi [email protected]• shinyorkeʮ͠ΜΑʔ͘ʯͱಡΈ·͢ ※໊લͷҰ෦ + ਪ͠όϯυͷϘʔΧϧ͔Βഈआ🎸• େख֎ࢿܥITίϯαϧاۀϚωʔδϟʔ ΫϥυɾΠϯϑϥѻ͏νʔϜͰ৭ʑͬͯΔਓ• ࣄझຯϑϧαΠΫϧͳΤϯδχΞ• ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ ʮٕज़ʯʮٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢. https://shinyorke.hatenablog.com/
ॏཁͳεϥΠυ͜ͷ৭Ͱ͢※10ຕͳ͍Ͱ͢, ࠷ѱ͜ͷ৭ͷεϥΠυ͚͍ͩͬͯͩ͘͞.
ຊͷελʔςΟϯάϝϯόʔ• ΫϥυͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX• αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ• ʮਪ͠ͷٿબखʯͷʮਪ࣌͠ʯͲ͜ͳͷ͔ΛݟۃΊΔ⽁
ͱ͜ΖͰ, ʮਪ͠ʯͬͯԿ?
ਓͱ͋Γ·͕͢, ରϞϊͰ͋ΕԿͰ͋Εʮਪ͠ʯͬͯݴ͏ؾ͕͢Δʢখʣ গͳ͘ͱࢲͦͷೝࣝʢҟೝΊΔʣ“ਪ͠ʢ͓͠ʣͱɺओʹΞΠυϧആ༏ʹ͍ͭͯ༻͍ΒΕΔຊޠͷଏޠͰ͋ΓɺਓʹનΊ͍ͨͱࢥ͏΄ͲʹײΛ͍࣋ͬͯΔਓͷ͜ͱΛ͍͏ɻ” by Wikipedia9
झຯʹ͓͚Δʮਪ͠ʯ͋Γ·͔͢?ྑ͔ͬͨΒͭͿ͍ͯΈ͍ͯͩ͘͞
ࢲͷʮझຯʹ͓͚Δਪ͠ʯ• ٿʢੲࣄʹ͍ͯͨ͠ʣ• ମೳྗ͕ߴͯ͘ෆࢥٞͳΩϟϥͷ֎ख• ηΠόʔϝτϦΫεʢٿ౷ܭֶʣϮλΫ• ͦͷଞͷझຯʢଞʹࢁ͋Δʣ• ڝഅΛݟΔͷ͕େ͖ʢ੨ࣛໟɾࠇࣛໟͷഅ͕ಛʹ͖ʣ🐴• 90s-00s͙Β͍ͷUK Rockେ͖🎸ʢblur, RADIOHEAD͕େʣ
ٕज़ʹ͓͚Δʮਪ͠ʯ͋Γ·͔͢?ྑ͔ͬͨΒͭͿ͍ͯΈ͍ͯͩ͘͞
ࢲͷʮਪٕ͠ज़ʯͷʑʢ΄ΜͷҰྫͰ͢, ͬͱͨ͘͞Μ͋Δʣ.ࣄझຯ͍ͭ͜ΕΒͷͷʹॿ͚ΒΕ͍ͯ·͢.
͜ͷઌʮٕज़ʯʮΫϥυʯʮٿʯͷʮਪ͠ʯͰ࣮ݱͨ͠ʮΫϥυͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDXʯͷΛ͠·͢ʂ
ਪ͠ͷٿબखʮΦΦλχαϯͷ2022ʯΛৼΓฦΔσʔλࢀরݩ: https://baseballsavant.mlb.com/statcast_search ※ΦʔϓϯσʔλͰ͢
2022ͷΦΦλχαϯ,εϥΠμʔͱ2γʔϜͰʮบ͕ੌ͍ʯϐονϟʔʹ• ࠓͷΦΦλχαϯ, ΊͬͪΌ εϥΠμʔ͍͛ͯΔ• ͓ؾ͖ͮͩΖ͏͔?ޙઓ 2γʔϜʢσʔλ্Sinkerʣ͕ ૿͍͑ͯΔ͜ͱʹ!?• γʔζϯޙ͔ΒͷΩϟϥมͰແঢ়ଶ. ਅ͙ͬओମͷελΠϧ͔Βมޭ.
ͱ͋ΔΦΦλχαϯͷొ൘ʢ2022/9/29, 8ճ10ୣࡾৼແࣦʣશٿͷ͏ͪ, ७ਮͳετϨʔτʢ4-Seamʣ͕Θ͔ͣ3.7%, ଞͯ͢ۂ͛ΔϘʔϧ.͛ͨॴʢัखઢʣ ϦϦʔεϙΠϯτʢัखઢʣٿछͷׂ߹
ϫΠʮਏ͍Ͱ͢…ຖճSQLͱίʔυΛॻ͖ଓ͚Δͷ͕ʯޢຎߦ͡Όͳ͍ͷͰͬͱؾָʹΓ͍ͨʢޢຎߦܦݧͳ͍Ͱ͕͢ʣ.≒
ϫΠʮͦ͏ͩ, ⚾σʔλج൫࡞Ζ͏ʯԶͷDXϓϩδΣΫτ, ര.σʔλʢhttps://baseballsavant.mlb.com/ ʣ͕͍ʹ͍͘, ίʔυॻ͘ͷਏ͍.ͩͬͨΒࣗͰ࡞Γ͍͢ͷ࡞ͬͯ͠·͑ʂͱ͍͏ΞΠσΞ͕߱ྟ.
࡞Γ·ͨ͠ʢ૯࡞ظؒɾʣ.
ΞϓϦͱσʔλج൫ͷΞʔΩςΫνϟʢʮԶͷDXʯੈքͷਤʣ
ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ• ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥυར༻• ͓ࡒʹ༏͍͠ߏͱ͍ํʹప͢Δ
ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥυαʔϏεͱ?Google CloudΛྫʹ͢Δͱ
ʲਤʳGoogle Cloudͷڞ༗ϞσϧʢComputeܥݶఆʣ
ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥυ• ʮϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥυαʔϏεʯͷಛ• Πϯϑϥɾαʔόʔͷϝϯςφϯε͕ෆཁʢࣗ͡Όͳͯ͘, ΫϥυαʔϏεଆ͕Δʣ• ΑΓ۩ମతʹ, ࣗͰK8sΫϥελVMΛݐͯͳͯ͘ྑ͍αʔϏεͷ͜ͱ• ʮϑϧϚωʔδυ͔ͭαʔόϨεʯʮख͕͔͔ؒΒͳ͍ʯ• ΄΅ϝϯςφϯεϑϦʔ.ϛυϧΣΞͷอकɾӡ༻͔Β։์͞ΕΔ.• εέʔϥϏϦςΟͷ୲อ͕͍͢͠. ඞཁʹԠͯ͡εέʔϧΞτɾεέʔϧΠϯָ͕.• Ұݟ͢ΔͱαʔόϨεߏύʔϑΣΫτʹݟ͑·͕͢, ϦΫΤετ͋ͨΓͷॲཧ࣌ؒ, ͑ΔϦιʔεྔͷ੍ݶɾϥϯλΠϜVersionͷ੍ݶ, ߟྀ͖͢ཹҙɾܽ͋Γ·͢.
ʁʁʁʮݸਓͰΫϥυαʔϏεΛ͕ͬͭΓ͏ͱ͓͕ۚ৺ʯͬͯࢥ͏͡Όͳ͍Ͱ͔͢, ΫϥυͱαʔόϨεͰ͍͍ײ͡ʹग़དྷ·͢.
αʔόϨεͳΫϥυ͓ࡒʹ༏͍͠• ʮ͏ͱ্ཱ͖͚͕ͩͪΕ͍͍ʯͷͰΞϓϦέʔγϣϯج൫αʔόϨεʹશৼΓ• WebΞϓϦέʔγϣϯόονॲཧίʔϧυελϯόΠ• ʮಈ͍͍ͯͳ͍͓͕͔͔࣌ۚΔʯͷDWHͱετϨʔδͷΈʢͦΕ͘͝গֹʣ• DatabaseʢBigQueryʣͱCloud Storage͍ํΛ͠ίετΛ͑Δ• ແྉͷϧʔϧΛཧղ͠, ͦͷൣᙝͰۃྗ͏ʢ㲈ඞཁͳͷʹ͓ۚΛ͏ʣ• σʔλͷظอϧʔϧͱετϨʔδλΠϓͷ, ҰׅॲཧͰͷσʔλΠϯϙʔτ
Ұϲ݄Ͱ$3ະຬ, ίʔώʔ3.34ഋఔͰࡁΈ·ͨ͠🐯 ※σʔλج൫ϓϩδΣΫτͷ࣮ίετΑΓࢉग़ʢίʔώʔίϯϏχίʔώʔج४ʣ, υϝΠϯऔಘྉ֎෦ίετΛআ͘
ΞʔΩςΫνϟղઆͷ·ͱΊ• ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥυαʔϏεத৺Ͱߏ• อकɾӡ༻࡞ۀ͔Βͷղ์, εέʔϥϏϦςΟͷ୲อʹ༗ར• αʔόϨεͳΒͰͷ੍ࣄ߲ʹҙʢॲཧ࣌ؒɾϦιʔεʣ• ͓ࡒʹ༏͍͍͠ํ͕Ͱ͖Δ• ʮ͚͓͕͔͔ͬͨ࣌ͩۚΔʯίετߏͷม• ແྉɾσʔλظอϧʔϧΛ׆༻ͯ͠ίετѹॖ
Ϣʔεέʔε͝ͱͷٕज़બఆμογϡϘʔυΞϓϦฤ
μογϡϘʔυΞϓϦͷϢʔεέʔε• σϞΞϓϦέʔγϣϯʢWebΞϓϦέʔγϣϯʣCloud Run্Ͱϗετ. ࣮PythonʢDashʣ.• όοΫΤϯυͷೝূAPI GatewayΛ༻͍ͯߦ͏, API KeyํࣜͷೝূʢSaaSతʹ͏ͨΊ͑ͯ͜͏͍ͬͯΔʣ.• BigQueryʹ֨ೲ͞ΕͨσʔλΛݕࡧ͢ΔόοΫΤϯυʢRESTful APIʣGoʢGinʣͰ࣮, Cloud RunͰϗετ.• RESTful APIͷResponse݅ผʹCloud StorageʹΩϟογϡʢಉ͡ΫΤϦΛԿ࣮ߦͤ͞ͳ͍ʣ.
ʮΫϥυαʔϏεબͼʯͷ͠͞Λղܾ͢Δ• αʔόϨεͳΫϥυαʔϏεͷબͼํ - Google Cloudฤ• App EngineͱFirebaseͱCloud Run, Կ͕ҧ͏ͷ?• Cloud Functions͍ͬͯͭ͏ͷ?ͦͦԿऀ??• …ͱ͍ͬͨΫΤενϣϯʹ͓͑͠·͢• ϓϩάϥϛϯάݴޠͷબͼํ
αʔόϨεͳΞϓϦέʔγϣϯಈ࡞ڥओͳαʔϏεʢ(PPHMFʣ 63- ֓ཁ"QQ&OHJOF IUUQTDMPVEHPPHMFDPNBQQFOHJOFαʔόϨε͔ͭϑϧϚωʔδυͷݩΈ͍ͨͳଘࡏ ͍͍͢'JSFCBTF IUUQTfiSFCBTFHPPHMFDPNΞϓϦͷΈͳΒͣ %#ɾ௨ͱΕΔ͜ͱ͕ଟ͍͔ͭศརͳ͓ݴޠ+BWB4DSJQUͷΈ$MPVE3VO IUUQTDMPVEHPPHMFDPNSVO͖ͳݴޠɾڥͰ࡞ΔͳΒ͜Ε ͻͱ·ͣ$POUBJOFS࡞ͬͯಈ͔ͤΔ$MPVE'VODUJPOT IUUQTDMPVEHPPHMFDPNGVODUJPOTͪΐͬͱͨؔ͠Λಈ͔͢ͳΒ4MBDL#PUͷখ͞ΊͳΞϓϦͳͲ
Cloud RunΛબͨ͠ཧ༝• σϞΞϓϦɾόοΫΤϯυڞʹContainerʢDockerʣϕʔεͰߏங͍ͯͨͨ͠Ί• ։ൃதґଘ͢ΔϥΠϒϥϦ͕ෆಁ໌ͩͬͨͨΊDocker ContainerఆͰ։ൃ• ContainerΛͦͷ··ಈ͔ͤΔCloud Run͕࠷ָ͔ͭ׳Ε͍ͯͨͷͰ࠾༻• Cloud RunΛબͨ͠ཧ༝ͱഎܠ1.ґଘϥΠϒϥϦͳͲͷ߹ͰPython or GoͰͷ։ൃ -> Firebase͕ީิ͔Βফ͑Δ2.Cloud FunctionsͰಈ͔͢ʹඍົͳ༷ͱͳͬͨͨΊ, Cloud Funcitonsফ͑Δ3.App EngineʢStandardʣContainer͡Όͳ͍ -> Cloud Runʹܾఆ ※ContainerͰಈ͔ͤΔApp EngineʢFlexibleʣΛΘͳ͔ͬͨཧ༝…ؾʹͳΔํฉ͍͍ͯͩ͘͞
ީิͱͯ͠ݕ౼ͨ͠ϓϩάϥϛϯάݴޠݴޠ ݕ౼ͷഎܠ ݁Ռ1ZUIPOʢ8FCΞϓϦɾόονʣҰ൪͍׳Ε͍ͯͯαΫοͱ࡞ΕΔςετɾσϓϩΠͷڥपΓ͕গʑ໘σϞ༻ͷσʔλΞϓϦ͓Αͼ σʔλج൫ͷؔΛ1ZUIPOʹͳ͓Ұ෦(PͰॻ͖͑Δ༧ఆ(Pʢ3&45GVM"1*ʣެࢲͱʹຆͲ͍ͬͯͳ͍͕͖ςετɾίʔυϑΥʔϚολʔ͕ޙൃݴޠ͚͍͍ͩ͋ͬͯ͢ϓϩάϥϛϯάݴޠͱͯ͠ͷಛੑ $POUBJOFS։ൃͱͷ૬ੑൈ܈ͩͬͨͷͰόοΫΤϯυͷݴޠͱͯ͠༻5ZQF4DSJQUʢࠓճෆ࠾༻ʣ1ZUIPOͷ࣍ʹ͍ͬͯΔϑϩϯτΤϯυ࡞ΔͳΒ͜ΕόοΫΤϯυଞʹީิ༗Γ%BTIʢ1ZUIPOͷ'SBNFXPSLʣଆ ͰϑϩϯτΤϯυΛੜ͢Δࣄʹͨ͠ͷͰࠓճΘͣ
ΞϓϦLow-code🐍• ͔݁Βݴ͏ͱʮDashʯͱ͍͏ PythonͷLow-codeͰ࣮.• ϓϩτλΠϓͰੳɾՄࢹԽʹ Jupyter LabͱPlotlyΛ͓ͬͯΓ, ͜ΕΛͦͷ··Ҡ২Ͱ͖ΔखஈͰ ࣮͔ͨͬͨ͠ʢબͨ͠എܠʣ• https://dash.plotly.com/
Dash for Pythonʢྫʣ• DashΛ͏ͱPython͚ͩͰϑϩϯτॻ͚·͢.• HTMLʹͨ͠ίϯϙʔωϯτΛPythonͰ ΰϦͬͱॻ͍ͯ࡞͢Δͱ͍͍ײ͡ʹಈ͘.• ΠϕϯτۦಈͰͷॻ͖͑Callbackͳ σίϨʔλʔͰ࣮ʢงғؾ΄΅Reactʣ.• ຊͷϑϩϯτΤϯυΑΓෆརͳଟ͍ͷͰ ͍ॴʹेҙΛʂʢύϑΥʔϚϯεʣ
GoΛͬͨόοΫΤϯυ։ൃ• ผʹPythonͰྑ͔ͬͨͷ͕ͩ, ͍ͨͯ͘GoΛͬͨʢ࠷େͷཧ༝ʣ.• RESTful APIΛ࡞ΔͱܾΊͨ࣌, ʮ͜ΕͬͯGoͳΒεϚʔτʹ࡞ΕΔ?ʯ ͱ͍͏ࣝϕʔεͷԾઆ͓Αͼ, ϓϩάϥϛϯάΛָ͠ΉͨΊܾߦ.• ࣗͰ࡞ͬͯಈ͔ͨ͠ॴ, ʮεϚʔτʹ࡞Εͦ͏ʯͱ͍͏ૂ͍ݟࣄతத.• ContainerαΠζͷॖখ, จ۟ͳ͍ύϑΥʔϚϯε.• go fmt, go testͷ͓ӄͰDevOpsʢCI/CDʣύΠϓϥΠϯ͕ચ࿅͞ΕͨϞϊʹ.
Ϣʔεέʔε͝ͱͷٕज़બఆσʔλऩूج൫ฤ
σʔλऩूج൫ͷϢʔεέʔεᶃ Cron Scheduler͕ຖܾ·ͬͨ࣌ؒʹᶄͷTriggerΛݺͼग़͢ᶄ TriggerσʔλऩूαʔϏεʢCrawlerʣʹඞཁͳύϥϝʔλʔΛ࡞ͬͯ͢ᶅ CrawlerTrigger͔ΒͷύϥϝʔλʔΛͬͯσʔλݩαΠτ͔ΒCSVΛμϯϩʔυ, Datalakeʹอଘᶆ Cron SchedulerͷτϦΨʔΛݩʹImporterCSVσʔλΛDWHʢBigQueryʣʹೖ
ΫϥυωΠςΟϒͳʮόονॲཧʯ• ΫϥυͰ͋Γ͕ͪͳόονॲཧํࣜ• Cloud Functions + Pub/SubΛͬͯϐλΰϥεΠονతʹ࡞Δ• BigQueryͱͷ͖߹͍ํ
ΫϥυͰ͋Γ͕ͪͳόονॲཧํࣜݴޠ ڧΈͱॴ ΫϥυωΠςΟϒͳํ๏ͱͯ͠ʲΓํ࣍ୈʳࣗલͰαʔόʔ࡞ΔʲڧΈʳDSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔʲॴʳϝϯςɾӡ༻͕໘ɾଐਓԽ͢Δةݥ"84&$4'BSHBUFͳΒ͍͍ײ͡7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒΫϥυ͏ҙຯ͕ݮΔ͔ʲΦεεϝʳ"QBDIF"JSflPXΛΫϥυαʔϏεͰ͏ʲڧΈʳ"JSflPXͷΈʹͬͯࢹɾӡ༻Մʲॴʳ"JSflPXͷֶशίετɾऔΓѻ͍ۀͰ͏ ͋Δఔͷن͕͋ΔόονॲཧͰྑ͍ײ͡ͷํ๏(PPHMF$MPVE "84྆ํ͍͚Δʲཁ͕݅߹͑Φεεϝʳ1VC4VCʹΑΔΠϕϯτۦಈʲڧΈʳϚΠΫϩαʔϏεతʹ࡞ΕΔʲॴʳઃܭɾ࣮Λཧղɾ׳ΕΔֶशίετΫϥυαʔϏεͳΒͰͷํ๏ઃܭɾ࣮ͷֶशίετ͔͔Δ͕׳Εͯ͠·͑݁ߏؾܰʹ࡞ΕΔ
ࠓճPub/SubΠϕϯτۦಈͰߏஙݴޠ ڧΈͱॴ ΫϥυωΠςΟϒͳํ๏ͱͯ͠ʲΓํ࣍ୈʳࣗલͰαʔόʔ࡞ΔʲڧΈʳDSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔʲॴʳϝϯςɾӡ༻͕໘ɾଐਓԽ͢Δةݥ"84&$4'BSHBUFͳΒ͍͍ײ͡7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒΫϥυ͏ҙຯ͕ݮΔ͔ʲΦεεϝʳ"QBDIF"JSflPXΛΫϥυαʔϏεͰ͏ʲڧΈʳ"JSflPXͷΈʹͬͯࢹɾӡ༻Մʲॴʳ"JSflPXͷֶशίετɾऔΓѻ͍ۀͰ͏ ͋Δఔͷن͕͋ΔόονॲཧͰྑ͍ײ͡ͷํ๏(PPHMF$MPVE "84྆ํ͍͚Δʲ࠾༻ʳ1VC4VCʹΑΔΠϕϯτۦಈʲڧΈʳϚΠΫϩαʔϏεతʹ࡞ΕΔʲॴʳઃܭɾ࣮Λཧղɾ׳ΕΔֶशίετΫϥυαʔϏεͳΒͰͷํ๏ઃܭɾ࣮ͷֶशίετ͔͔Δ͕׳Εͯ͠·͑݁ߏؾܰʹ࡞ΕΔ
Pub/Sub + Cloud FunctionsͳϐλΰϥεΠον• crontabͷΫϥυαʔϏεʢCloud Schedulerʣ͔ΒPub/Subܦ༝Ͱୟ͘ࣄͰόονॲཧΛ࣮ݱ• ֤Cloud Functionsؔʢᶄ, ᶅ, ᶆʣPub/Sub͔ΒͷϝοηʔδΛड͚ͯಈ͘ΈͰ࡞Δ• ͜ΕΒͷํ๏Google CloudެࣜʹQuick Start͕͋ΔͷͰਅࣅͨ͠Β࣮ݱՄೳʢਅࣅ͠·ͨ͠ʣ• ͪͳΈʹڥCloud RunͳͲ, ଞͷαʔϏε͑·͢&ݴޠPython͡Όͳͯ͘ߏ͍·ͤΜ
Pub/SubʹΑΔΠϕϯτۦಈΛબͨ͠ཧ༝• ӡ༻ɾίετ྆໘ͰαʔόϨεͷϝϦοτΛ׆͔ͨ͢Ί.• ࣗલαʔόʔΠϯϑϥΛࣗݾཧ͠ͳ͍ΞΧϯ&VMͩͱ͓͔͔ۚΔ.• AirflowʢCloud ComposerʣGKEΫϥελ͕ඞཁͰίετ໘Ͱͷෆ҆.• Cloud Functions + Cloud Scheduler + Pub/SubͰαʔόϨεԽ͕Ұ൪ཁٻ༷ʹ߹͏ͱஅ• ֤ॲཧΛϚΠΫϩαʔϏεͱ࣮ͯ͠Մೳʢೋ൪ͷཧ༝ʣ.• ಠཱͨ͠ॲཧ୯ҐͰCloud FunctionsͷؔΛ࡞Δ͜ͱͰૄ݁߹ͳϚΠΫϩαʔϏεʹͳΔ.• ֤ؔͰϓϩάϥϛϯάݴޠɾFrameworkͷมߋ͕ग़དྷΔ, ςετͷ͢͠͞ͱ͍͏ϝϦοτ.
• σʔλநग़ʢExtractʣ, มʢTransformʣ, ૹग़ʢLoadʣͦΕͧΕͰ͚ͯ࡞ΔͱεοΩϦ͠·͢. ※ࠓճ্༷TransformͱLoadҰॹʹ͍ͯ͠·͢ʢͦͷํ͕εοΩϦͨͨ͠Ίʣ• ॲཧͷτϦΨʔCloud SchedulerͷαʔϏεΛCronΘΓʹ͢Δͱ࣮͕͍͍ײ͡ʹলུͰ͖·͢.• Cloud Functionsʢ&ଞࣾΫϥυؚΊͨFaaSʣϦΫΤετ͋ͨΓͷॲཧ࣌ؒɾϦιʔε੍ݶ͕͋ΔͷͰҙ• Cloud FunctionsʢୈೋੈʣϦΫΤετ͋ͨΓͷ࠷େॲཧ࣌ؒ60ʢHTTPܦ༝ͷ߹ʣ, CPU/ϝϞϦ੍ݶ͋Γ·͢.• AWS LambdaͷαʔϏεͦΕͧΕ੍͕͋ΔͷͰ͝ҙΛ, େ͖͍σʔλDataflow, ઐ༻αʔϏεΛ͏͜ͱΛݕ౼. ※FaaS: Function as a ServiceʢؔϕʔεͷΫϥυαʔϏε, Cloud Functions, AWS LambdaͳͲʣCloud FunctionsΛͬͨόονॲཧͷ࣮
BigQueryͱͷ͖߹͍ํ• αΫοͱ͑ΔεέʔϧՄೳͳDWH͕ཉ͔ͬͨ͠ΒBigQuery͍͍ͧ• ʢࠓճͷج൫Ͱͳ͘ʣݱ࣮ͷۀͰεέʔϧՄೳͳॴʹԿ͔ٹΘΕ·ͨ͠.• ݸਓར༻Ͱ͍׳Ε͓ͯ͘ͱͳ͓ྑ͍Ͱ͢.• ݄ͨΓ, 10GBͷσʔλอ, 1TBͷΫΤϦ࣮ߦແྉ.• ݸਓͰͬͯͯ͜ͷྔͦ͏ͦ͏ߦ͔ͳ͍ϋζͳͷͰ͏ͱ͍͍͔.• ҰσʔλΛೖΕ͓ͯ͘ͱ, ΞϓϦ͔Β͏ɾΞυϗοΫੳ྆ํศརͰ͢. ࢲ͏BigQuery͔ΒಀΕΔ͜ͱग़དྷ·ͤΜʢਅإʣ.
ٕज़తͳͰർΕ͖ͯ·ͤΜ͔?ʮਪ͠ͷٿબखʯͷΛ͠·͢Ͷ⽁
WBCͰઈରݟಀͤͳ͍ʮ֤ࠃͷਓͨͪʯ• ମೳྗ͓Խ͚ͰΩϨοΩϨ• ύϫʔ, ڧݞ, कͦͯ͠٭͕ചΓ• ߽͛ͯٿorΤά͍มԽٿ• ͦ͏͍͏ਓͨͪΛ͝հ• ݱ࣌ͷ৽ঙ߶ࢤͬΆ͍ Ӊਓબख͍Δ͔🤘
WBCʹग़͢Δʢϋζʣ, ֤ࠃදͷਓϚϯ• ถࠃදʮϝδϟʔφϯόʔϫϯͷकඋྗΛތΔαʔυʯ• υϛχΧදʮະདྷͷΠνϩʔ or ৽ঙ??࿘ອ͋;ΕΔηϯλʔʯ• ࣆJAPANʮϝδϟʔϦʔά༗ג֎ख͕ຬΛ࣋ͯ͠ͷ͝ొʯ
ϊʔϥϯɾΞϨφυʢถࠃදʣ• 10࿈ଓΰʔϧυάϥϒͱ͍͏ ϝδϟʔ۶ࢦͷڧଧͷࡾྥख• ଧऀͱͯ͠ۃͳҾͬுΓϚϯ ӈଧͪͰଧ΄΅ࠨํ• ถࠃදओཁϝϯόʔͷҰਓ ࡾྥकඋͱϑϧεΠϯάʹ
ϑϦΦɾϩυϦήεʢυϛχΧදʣ• λϨϯτ܉ஂυϛχΧͷएखελʔ. 2022MLB৽ਓԦ, All MLB 2ndνʔϜೖΓ.• ࡾৼ͔ଧͱ͍͏ۃͳଧܸελΠϧ, ҙ֎ͱ֯ʹଧͯΔଧܸηϯε, Πϝʔδతʹ৽ঙ߶ࢤͬΆ͍.• ॴଐ͢ΔϚϦφʔζͰΠνϩʔࢯ͕ ࿅शύʔτφʔΛ͍ͯ͠Δ͜ͱͰ.
ϥʔζɾψʔτόʔʢࣆJAPANʣ• ࣆJAPANॳͷΞϝϦΧग़બख• ଧ͍ͷͷग़ྥͱଧ ͦͯ͠ଧٿ͕͘, 3ϙδγϣϯ ͦͭͳ͘कΕΔ֎कඋ˕• Ωϟϥཱ͍͍ͪͯͯ͠Ϡπͱ͍͏ᷚ, ࣆδϟύϯͷηϯλʔͱͯ͠ظʂ
ࣆJAPANͱ͍͑, ͜ͷೋਓݟಀͤͳ͍ਓώϯτ: ݩϋϜͷഎ൪߸11
μϧϏογϡ༗͜ͱ, ٿεϖγϟϦετͳ@faridyu͞Μୣࡾৼͷଟ͞ͱΧοτϘʔϧࢁ…WBCͰͲ͏͍͏ϓϨʔ͢Δ͔ظ͍ͨ͠.
େ୩ᠳฏʢࢦ໊ଧऀʣͷओཁͳଧٿσʔλҾͬுΓϚϯͰͳͯ֯͘ʹ͍ଧٿଧͬͯͯੌ͍ʢ͜ͳΈʣ
݁ͼ
ʲ࠶ܝʳຊͷελʔςΟϯάϝϯόʔ• ΫϥυͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX• αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ• ʮਪ͠ͷٿબखʯͷʮਪ࣌͠ʯͲ͜ͳͷ͔ΛݟۃΊΔ⽁
ࠓͷΛཁ͢Δͱ…• ϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥυ͚ͩͰσʔλج൫࡞ΕΔ.• ΫϥυαʔϏεͱϓϩάϥϛϯάݴޠ༻్ʹ߹Θͤͯબ΅͏.• ࣆJAPANʹڧྗͳϥΠόϧ͕ͨ͘͞Μ͍Δʢ͜ͳΈʣ.
ʮԶͷDXϓϩδΣΫτʯരޙͷޮՌ.• ٿσʔλαΠΤϯςΟετɾϓϩμΫτΦʔφʔͱ͍͏ࢹͰϝονϟ͑ͦ͏.• ؒ500MB, 91߲ͱ͍͏టष͍σʔλΛ͍͘͢͢Δ͚͍͍ͩϞϊΛ࡞Εͨʢࣄ࣮ʣ.• σʔλΛݟֶͯΜͩΓ, ϒϩάɾొஃͷωλʹ͢Δ͜ͱͰ։ൃͷࢿճऩ͕Ͱ͖ͦ͏.• ݸਓͱͯ͠ͷٕज़ݕূ͕ϝονϟḿΔ༧ײʢ&ࣗݾຬʣ• Google Cloudͷࢼ͍ͨ͠ػೳɾϞϊΛ࣮ݧ͢Δsandboxͱͯ͠ػೳͦ͠͏, AIܥαʔϏεͱ͔.• ΞʔΩςΫνϟΛͦͷ··ʹAWSAzureͰࢼ͢ͷΞϦ, ϚϧνΫϥυΠέΔΜ͡ΌͶ?• ࣄʹඞཁͳεΩϧɾࣝΛझຯʢࣗݾຬʣ͔ΒखʹೖΕΔج൫ͱͯ͠࠷ߴʹྑ͍.
͓ࣄʢۀʣͰࢀߟʹ͠Α͏ͱࢥͬͨํ• ࠓճհͨ͠ΓํɾߏઈରతͳճɾϕετϓϥΫςΟεͰͳ͍Ͱ͢. ྫ͑αʔόϨεɾΞʔΩςΫνϟʹ͖͢/͖͢͡Όͳ͍ঢ়گ࣮֬ʹଘࡏ͠·͢.• ͜ͷࢲʢshinyorkeʣ͕Γ͍ͨࣄ, ͍͍ͱࢥͬͯΔࣄʢ&৮Γ͍ͨϞϊʣΛ ٧ΊࠐΜͰ࡞ͬͨ, ͕ࣗΓ͍ͨࣄͷूେͰ, ͋͘·Ͱղͷग़͠ํͷҰͭͰ͢.• αʔόϨεʹͩ͜ΘΒͳ͔ͬͨΒʮGKEΫϥελཱͯͯͦ͜Ͱಈ͔͢ʯͰऴΘΔ. ͜ΕཱͳղͩͬͨΓ͠·͢, ϝϦσϝ࣍ୈͰ.• Infrastructure as CodeͰ͖ͯͳ͍, ςετίʔυΓͳ͍etc…࣮՝ࢁੵΈ.• ʢίϯςΩετͷཧղ͕த్ͳ··ʣͦͷ··ਅࣅ͢Δͱരࢮ͠·͢. ·ͣखΛಈ͔͠, ֶशͨ͠Γಈ͔ͨ͠Γ͍͍ͯ͠ͷΛݟ͚ͭΔࢀߟʹͲ͏ͧʂ
ʮਪ͠ਪͤΔ࣌ʹਪͤʯͱ͍͏ԶͷDXଓฤ.• ϝδϟʔϦʔάɾϓϩٿͷ༧ଌΤϯδϯ࡞ͬͯΔͷͰ, ͦΕΛࠓճͷج൫ʹ࣮͍ͨ͠, ૬ੑྑ͍ͣͳͷͰ.• ࠓޙͷలͱͯ͠, σʔλͷϥΠηϯεͷ՝Λղܾ͠, ͍͍ײ͡ͳٿσʔλαΠτΛҰൠެ։͢ΔʢઈରʹΓ͍ͨʣ.• ࠓޙٕज़Λ৭ʑࢼ͠, ٿͷ໘നָ͍͠ΈํΛൃ৴ͭͭ͠, ʮٿͱٕज़ͷਪ͠ΛϦίϝϯυʯ͢ΔԶͷDXΛ͍ͬͯ͘Ͱ.
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠⽁@shinyorke
shinyorke
fendo181
yamish123
dosukoi_android
daiksy
akkeylab
paper2
iwasiman
javiergs
mski_iksm
kubode
okuramasafumi
andpad
bermonpainter
searls
danielanewman
iamctodd
zakiwarfel
jennybc
roundedbygravity
addyosmani
michaelherold
tanoku
schacon
brianwarren