俺のDXを実現するためのサーバレスなデータ基盤開発と運用 / Serverless Data Platform and Baseball

by Shinichi Nakagawa

Slide 1

Slide 1 text

ਪ͠͸ਪͤΔ࣌ʹਪͤ. ͱ͍͏ԶͷDXΛ࣮ݱ͢ΔͨΊͷαʔόϨεͳσʔλج൫։ൃͱӡ༻. ໺ٿσʔλΛఴ͑ͯ⽁ @shinyorke 2023/02/10 Developers Summit 2023

Slide 2

Slide 2 text

Onboardingʢ͜ͷηογϣϯͷ͝Ҋ಺ʣ • ࢲ͕झຯͰߏஙͨ͠ʮ໺ٿσʔλϓϥοτϑΥʔϜʯαʔϏεʹ͓͚Δ,   ʮΫϥ΢υωΠςΟϒͳσʔλΤϯδχΞϦϯάʯࣄྫͷ঺հͰ͢. • Ϋϥ΢υɾσʔλج൫Ͱඇৗʹ෯޿͍ൣғͷࣄྫͱͳΓ·͢.   ࢿྉ͸ൃදޙʹެ։͠·͢&ؾʹͳΔՕॴ͸ࢿྉΛಡΉࣄΛਪ঑͠·͢. • ໺ٿʹڵຯͳ͍ʢor޷͖͡Όͳ͍ʣํͱ΋Ұॹʹָ͠ΊͨΒ޾͍Ͱ͢.

Slide 3

Slide 3 text

౰αʔϏεͷߏஙͰ࢖ͬͨϓϩάϥϛϯάݴޠɾαʔϏεɾؔ࿈ٕज़ͷ਺ʑ ΊͬͪΌଟ͍ͷͰؾʹͳΔͱ͜Ζ͚ͩಡΉɾ֮͑ΔͰ΋େৎ෉Ͱ͢ʂ

Slide 4

Slide 4 text

͜ΜͳํʹΦεεϝͰ͢. • DWH΍σʔλج൫ͱ͍ͬͨʮσʔλར׆༻ʯͷ͍͍ײ͡ͳߏஙࣄྫΛ஌Γ͍ͨํ. • ύϒϦοΫΫϥ΢υ, ओʹGoogle CloudΛ࢖ͬͯԿ͔͠ΒͷࣄΛ͢Δํ. • Ϋϥ΢υαʔϏεΛ࢖͍͍ͬͨײ͡ͳγεςϜઃܭɾߏஙʹڵຯ͕͋Δํ. • αʔόϨεͳΫϥ΢υαʔϏεΛ࢖͍ͬͯΔ or ͜Ε͔Β࢖͏ํʢڵຯͰ΋͍͍ʣ.   ※AWS Lambda, AWS App Runner, Google App Engine, Cloud RunͳͲ͕֘౰. • ʢ޷͖ݏ͍ؔ܎ͳ͘ʣ໺ٿͷϧʔϧٴͼΦΦλχαϯ͕୭͔ͩ஌͍ͬͯΔ.

Slide 5

Slide 5 text

Who am I ? ʢ͓લ୭Α?ʣ • Shinichi Nakagawa@shinyorke • shinyorke͸ʮ͠ΜΑʔ͘ʯͱಡΈ·͢   ※໊લͷҰ෦ + ਪ͠όϯυͷϘʔΧϧ͔Βഈआ🎸 • େख֎ࢿܥITίϯαϧاۀϚωʔδϟʔ   Ϋϥ΢υɾΠϯϑϥѻ͏νʔϜͰ৭ʑ΍ͬͯΔਓ • ࢓ࣄ΋झຯ΋ϑϧαΠΫϧͳΤϯδχΞ • ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ   ʮٕज़ʯʮ໺ٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢.   https://shinyorke.hatenablog.com/

Slide 6

Slide 6 text

ॏཁͳεϥΠυ͸͜ͷ഑৭Ͱ͢ ※10ຕ΋ͳ͍Ͱ͢, ࠷ѱ͜ͷ৭ͷεϥΠυ͚ͩ௥͍ͬͯͩ͘͞.

Slide 7

Slide 7 text

ຊ೔ͷελʔςΟϯάϝϯόʔ • Ϋϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX • αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ • ʮਪ͠ͷ໺ٿબखʯͷʮਪ࣌͠ʯ͸Ͳ͜ͳͷ͔ΛݟۃΊΔ⽁

Slide 8

Slide 8 text

ͱ͜ΖͰ, ʮਪ͠ʯͬͯԿ?

Slide 9

Slide 9 text

ਓ෺ͱ͋Γ·͕͢, ର৅͸ϞϊͰ͋ΕԿͰ͋Εʮਪ͠ʯͬͯݴ͏ؾ͕͢Δʢখ੠ʣ   গͳ͘ͱ΋ࢲ͸ͦͷೝࣝʢҟ࿦͸ೝΊΔʣ “ਪ͠ʢ͓͠ʣͱ͸ɺओʹΞΠυϧ΍ആ༏ʹ ͍ͭͯ༻͍ΒΕΔ೔ຊޠͷଏޠͰ͋Γɺਓ ʹનΊ͍ͨͱࢥ͏΄Ͳʹ޷ײΛ͍࣋ͬͯΔ ਓ෺ͷ͜ͱΛ͍͏ɻ” by Wikipedia 9

Slide 10

Slide 10 text

झຯʹ͓͚Δʮਪ͠ʯ͸͋Γ·͔͢? ྑ͔ͬͨΒͭͿ΍͍ͯΈ͍ͯͩ͘͞

Slide 11

Slide 11 text

ࢲͷʮझຯʹ͓͚Δਪ͠ʯ͸ • ໺ٿʢੲ͸࢓ࣄʹ͍ͯͨ͠ʣ • ਎ମೳྗ͕ߴͯ͘ෆࢥٞͳΩϟϥͷ֎໺ख • ηΠόʔϝτϦΫεʢ໺ٿ౷ܭֶʣϮλΫ • ͦͷଞͷझຯʢଞʹ΋୔ࢁ͋Δʣ • ڝഅΛݟΔͷ͕େ޷͖ʢ੨ࣛໟɾࠇࣛໟͷഅ͕ಛʹ޷͖ʣ🐴 • 90s-00s͙Β͍ͷUK Rockେ޷͖🎸ʢblur, RADIOHEAD͕େ޷෺ʣ

Slide 12

Slide 12 text

ٕज़ʹ͓͚Δʮਪ͠ʯ͸͋Γ·͔͢? ྑ͔ͬͨΒͭͿ΍͍ͯΈ͍ͯͩ͘͞

Slide 13

Slide 13 text

ࢲͷʮਪٕ͠ज़ʯͷ਺ʑʢ΄ΜͷҰྫͰ͢, ΋ͬͱͨ͘͞Μ͋Δʣ. ࢓ࣄ΋झຯ΋͍ͭ΋͜ΕΒͷ΋ͷʹॿ͚ΒΕ͍ͯ·͢.

Slide 14

Slide 14 text

͜ͷઌ͸ʮٕज़ʯʮΫϥ΢υʯʮ໺ٿʯͷʮਪ͠ʯͰ࣮ݱͨ͠ ʮΫϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDXʯͷ࿩Λ͠·͢ʂ

Slide 15

Slide 15 text

ਪ͠ͷ໺ٿબखʮΦΦλχαϯͷ2022೥ʯΛৼΓฦΔ σʔλࢀরݩ: https://baseballsavant.mlb.com/statcast_search ※ΦʔϓϯσʔλͰ͢

Slide 16

Slide 16 text

2022೥ͷΦΦλχαϯ, εϥΠμʔͱ2γʔϜͰ ʮบ͕ੌ͍ʯϐονϟʔʹ • ࠓ೥ͷΦΦλχαϯ, ΊͬͪΌ   εϥΠμʔ౤͍͛ͯΔ • ͓ؾ͖ͮͩΖ͏͔?ޙ൒ઓ͸   2γʔϜʢσʔλ্͸Sinkerʣ͕   ૿͍͑ͯΔ͜ͱʹ!? • γʔζϯޙ൒͔ΒͷΩϟϥมͰແ૒ঢ়ଶ.   ਅͬ௚͙ओମͷελΠϧ͔Βม਎੒ޭ.

Slide 17

Slide 17 text

ͱ͋ΔΦΦλχαϯͷొ൘೔ʢ2022/9/29, 8ճ10ୣࡾৼແࣦ఺ʣ શ౤ٿͷ͏ͪ, ७ਮͳετϨʔτʢ4-Seamʣ͕Θ͔ͣ3.7%, ଞ͸͢΂ͯۂ͛ΔϘʔϧ. ౤͛ͨ৔ॴʢัख໨ઢʣ ϦϦʔεϙΠϯτʢัख໨ઢʣ ٿछͷׂ߹

Slide 18

Slide 18 text

ϫΠʮਏ͍Ͱ͢…ຖճSQLͱίʔυΛॻ͖ଓ͚Δͷ͕ʯ ޢຎߦ͡Όͳ͍ͷͰ΋ͬͱؾָʹ΍Γ͍ͨʢޢຎߦܦݧͳ͍Ͱ͕͢ʣ. ≒

Slide 19

Slide 19 text

ϫΠʮͦ͏ͩ, ⚾σʔλج൫࡞Ζ͏ʯ ԶͷDXϓϩδΣΫτ, ര஀. σʔλʢhttps://baseballsavant.mlb.com/ ʣ͕࢖͍ʹ͍͘, ίʔυॻ͘ͷਏ͍. ͩͬͨΒࣗ෼Ͱ࡞Γ΍͍͢΋ͷ࡞ͬͯ͠·͑ʂͱ͍͏ΞΠσΞ͕߱ྟ.

Slide 20

Slide 20 text

࡞Γ·ͨ͠ʢ૯࡞੒ظؒɾ໿൒೥ʣ.

Slide 21

Slide 21 text

ΞϓϦͱσʔλج൫ͷΞʔΩςΫνϟ ʢʮԶͷDXʯੈքͷ஍ਤʣ

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υར༻ • ͓ࡒ෍ʹ༏͍͠ߏ੒ͱ࢖͍ํʹప͢Δ

Slide 24

Slide 24 text

ʮϑϧϚωʔδυ͔ͭαʔόϨεʯ ͳΫϥ΢υαʔϏεͱ͸? Google CloudΛྫʹ͢Δͱ

Slide 25

Slide 25 text

ʲਤʳGoogle Cloudͷ੹೚ڞ༗ϞσϧʢComputeܥݶఆʣ

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υ • ʮϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥ΢υαʔϏεʯͷಛ௃ • Πϯϑϥɾαʔόʔͷϝϯςφϯε͕ෆཁʢࣗ෼͡Όͳͯ͘, Ϋϥ΢υαʔϏεଆ͕΍Δʣ • ΑΓ۩ମతʹ͸, ࣗ෼ͰK8sΫϥελ΍VMΛݐͯͳͯ͘΋ྑ͍αʔϏεͷ͜ͱ • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯ͸ʮख͕͔͔ؒΒͳ͍ʯ • ΄΅ϝϯςφϯεϑϦʔ.ϛυϧ΢ΣΞͷอकɾӡ༻͔Β։์͞ΕΔ. • εέʔϥϏϦςΟͷ୲อ͕͠΍͍͢. ඞཁʹԠͯ͡εέʔϧΞ΢τɾεέʔϧΠϯָ͕. • Ұݟ͢ΔͱαʔόϨεߏ੒͸ύʔϑΣΫτʹݟ͑·͕͢, ϦΫΤετ͋ͨΓͷॲཧ࣌ؒ,   ࢖͑ΔϦιʔεྔͷ੍ݶɾϥϯλΠϜVersionͷ੍ݶ౳, ߟྀ͢΂͖ཹҙ఺ɾܽ఺΋͋Γ·͢.

Slide 28

Slide 28 text

ʁʁʁʮݸਓͰΫϥ΢υαʔϏεΛ͕ͬͭΓ࢖͏ͱ͓͕ۚ৺഑ʯ ͬͯࢥ͏͡Όͳ͍Ͱ͔͢, Ϋϥ΢υͱαʔόϨεͰ͍͍ײ͡ʹग़དྷ·͢.

Slide 29

Slide 29 text

αʔόϨεͳΫϥ΢υ͸͓ࡒ෍ʹ༏͍͠ • ʮ࢖͏ͱ্ཱ͖͚͕ͩͪΕ͹͍͍ʯͷͰΞϓϦέʔγϣϯج൫͸αʔόϨεʹશৼΓ • WebΞϓϦέʔγϣϯ΋όονॲཧ΋ίʔϧυελϯόΠ • ʮಈ͍͍ͯͳ͍࣌΋͓͕͔͔ۚΔʯͷ͸DWHͱετϨʔδͷΈʢͦΕ΋͘͝গֹʣ • DatabaseʢBigQueryʣͱCloud Storage͸࢖͍ํΛ޻෉͠ίετΛ཈͑Δ • ແྉ࿮ͷϧʔϧΛཧղ͠, ͦͷൣᙝͰۃྗ࢖͏ʢ㲈ඞཁͳ΋ͷʹ͸͓ۚΛ෷͏ʣ • σʔλͷ௕ظอ؅ϧʔϧͱετϨʔδλΠϓͷ޻෉, ҰׅॲཧͰͷσʔλΠϯϙʔτ

Slide 30

Slide 30 text

Ұϲ݄Ͱ$3ະຬ, ίʔώʔ3.34ഋ෼ఔ౓ͰࡁΈ·ͨ͠🐯   ※σʔλج൫ϓϩδΣΫτͷ࣮ίετΑΓࢉग़ʢίʔώʔ୅͸ίϯϏχίʔώʔج४ʣ, υϝΠϯऔಘྉ౳֎෦ίετΛআ͘

Slide 31

Slide 31 text

ΞʔΩςΫνϟղઆͷ·ͱΊ • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υαʔϏεத৺Ͱߏ੒ • อकɾӡ༻࡞ۀ͔Βͷղ์, εέʔϥϏϦςΟͷ୲อʹ༗ར • αʔόϨεͳΒͰ͸ͷ੍໿ࣄ߲ʹ஫ҙʢॲཧ࣌ؒɾϦιʔε౳ʣ • ͓ࡒ෍ʹ༏͍͠࢖͍ํ͕Ͱ͖Δ • ʮ࢖͚͓͕͔͔ͬͨ࣌ͩۚΔʯίετߏ଄΁ͷม׵ • ແྉ࿮ɾσʔλ௕ظอ؅ϧʔϧΛ׆༻ͯ͠ίετѹॖ

Slide 32

Slide 32 text

Ϣʔεέʔε͝ͱͷٕज़બఆ μογϡϘʔυΞϓϦฤ

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

μογϡϘʔυΞϓϦͷϢʔεέʔε • σϞΞϓϦέʔγϣϯʢWebΞϓϦέʔγϣϯʣ͸Cloud Run্Ͱϗετ. ࣮૷͸PythonʢDashʣ. • όοΫΤϯυͷೝূ͸API GatewayΛ༻͍ͯߦ͏, API KeyํࣜͷೝূʢSaaSతʹ࢖͏ͨΊ׶͑ͯ͜͏΍͍ͬͯΔʣ. • BigQueryʹ֨ೲ͞ΕͨσʔλΛݕࡧ͢ΔόοΫΤϯυʢRESTful APIʣ͸GoʢGinʣͰ࣮૷, Cloud RunͰϗετ. • RESTful APIͷResponse͸৚݅ผʹCloud StorageʹΩϟογϡʢಉ͡ΫΤϦΛԿ౓΋࣮ߦͤ͞ͳ͍ʣ.

Slide 35

Slide 35 text

ʮΫϥ΢υαʔϏεબͼʯͷ೉͠͞Λղܾ͢Δ • αʔόϨεͳΫϥ΢υαʔϏεͷબͼํ - Google Cloudฤ • App EngineͱFirebaseͱCloud Run, Կ͕ҧ͏ͷ? • Cloud Functions͍ͬͯͭ࢖͏ͷ?ͦ΋ͦ΋Կऀ?? • …ͱ͍ͬͨΫΤενϣϯʹ͓౴͑͠·͢ • ϓϩάϥϛϯάݴޠͷબͼํ

Slide 36

Slide 36 text

αʔόϨεͳΞϓϦέʔγϣϯಈ࡞؀ڥ ओͳαʔϏεʢ(PPHMFʣ 63- ֓ཁ "QQ&OHJOF IUUQTDMPVEHPPHMFDPNBQQFOHJOF αʔόϨε͔ͭϑϧϚωʔδυͷ ݩ૆Έ͍ͨͳଘࡏ ࢖͍΍͍͢ 'JSFCBTF IUUQT fi SFCBTFHPPHMFDPN ΞϓϦͷΈͳΒͣ %#ɾ௨஌ͱ ΍ΕΔ͜ͱ͕ଟ͍͔ͭศར ͳ͓ݴޠ͸+BWB4DSJQUͷΈ $MPVE3VO IUUQTDMPVEHPPHMFDPNSVO ޷͖ͳݴޠɾ؀ڥͰ࡞ΔͳΒ͜Ε   ͻͱ·ͣ$POUBJOFS࡞ͬͯಈ͔ͤΔ $MPVE'VODUJPOT IUUQTDMPVEHPPHMFDPNGVODUJPOT ͪΐͬͱͨؔ͠਺Λಈ͔͢ͳΒ 4MBDL#PU౳ͷখ͞ΊͳΞϓϦͳͲ

Slide 37

Slide 37 text

Cloud RunΛબ୒ͨ͠ཧ༝ • σϞΞϓϦɾόοΫΤϯυڞʹContainerʢDockerʣϕʔεͰߏங͍ͯͨͨ͠Ί • ։ൃத͸ґଘ͢ΔϥΠϒϥϦ͕ෆಁ໌ͩͬͨͨΊDocker Container૝ఆͰ։ൃ • ContainerΛͦͷ··ಈ͔ͤΔCloud Run͕࠷΋ָ͔ͭ׳Ε͍ͯͨͷͰ࠾༻ • Cloud RunΛબ୒ͨ͠ཧ༝ͱഎܠ 1.ґଘϥΠϒϥϦͳͲͷ౎߹ͰPython or GoͰͷ։ൃ -> Firebase͕ީิ͔Βফ͑Δ 2.Cloud FunctionsͰಈ͔͢ʹ͸ඍົͳ࢓༷ͱͳͬͨͨΊ, Cloud Funcitons΋ফ͑Δ 3.App EngineʢStandardʣ͸Container͡Όͳ͍ -> Cloud Runʹܾఆ   ※ContainerͰಈ͔ͤΔApp EngineʢFlexibleʣΛ࢖Θͳ͔ͬͨཧ༝͸…ؾʹͳΔํ͸ฉ͍͍ͯͩ͘͞

Slide 38

Slide 38 text

ީิͱͯ͠ݕ౼ͨ͠ϓϩάϥϛϯάݴޠ ݴޠ ݕ౼ͷഎܠ ݁Ռ 1ZUIPOʢ8FCΞϓϦɾόονʣ Ұ൪࢖͍׳Ε͍ͯͯαΫοͱ࡞ΕΔ ςετɾσϓϩΠ౳ͷ؀ڥपΓ͕ গʑ໘౗ σϞ༻ͷσʔλΞϓϦ͓Αͼ σʔλج൫ͷؔ਺Λ1ZUIPOʹ ͳ͓Ұ෦(PͰॻ͖׵͑Δ༧ఆ (Pʢ3&45GVM"1*ʣ ެࢲͱ΋ʹຆͲ࢖͍ͬͯͳ͍͕޷͖ ςετɾίʔυϑΥʔϚολʔ౳͕ ޙൃݴޠ͚ͩ͋ͬͯ࢖͍΍͍͢ ϓϩάϥϛϯάݴޠͱͯ͠ͷಛੑ   $POUBJOFS։ൃͱͷ૬ੑൈ܈ͩͬͨ ͷͰόοΫΤϯυͷݴޠͱͯ͠࢖༻ 5ZQF4DSJQUʢࠓճ͸ෆ࠾༻ʣ 1ZUIPOͷ࣍ʹ࢖͍ͬͯΔ ϑϩϯτΤϯυ࡞ΔͳΒ͜Ε όοΫΤϯυ͸ଞʹީิ༗Γ %BTIʢ1ZUIPOͷ'SBNFXPSLʣଆ   ͰϑϩϯτΤϯυΛੜ੒͢Δࣄʹ ͨ͠ͷͰࠓճ͸࢖Θͣ

Slide 39

Slide 39 text

ΞϓϦ͸Low-code🐍 • ݁࿦͔Βݴ͏ͱʮDashʯͱ͍͏   PythonͷLow-codeͰ࣮૷. • ϓϩτλΠϓͰ͸෼ੳɾՄࢹԽʹ   Jupyter LabͱPlotlyΛ࢖͓ͬͯΓ,   ͜ΕΛͦͷ··Ҡ২Ͱ͖ΔखஈͰ   ࣮૷͔ͨͬͨ͠ʢબ୒ͨ͠എܠʣ • https://dash.plotly.com/

Slide 40

Slide 40 text

Dash for Pythonʢྫʣ • DashΛ࢖͏ͱPython͚ͩͰϑϩϯτॻ͚·͢. • HTMLʹ໛ͨ͠ίϯϙʔωϯτΛPythonͰ   ΰϦͬͱॻ͍ͯ࡞੒͢Δͱ͍͍ײ͡ʹಈ͘. • ΠϕϯτۦಈͰͷॻ͖׵͑͸Callbackͳ   σίϨʔλʔͰ࣮૷ʢงғؾ΄΅Reactʣ. • ຊ෺ͷϑϩϯτΤϯυΑΓෆརͳ఺΋ଟ͍ͷͰ   ࢖͍ॴʹ͸े෼஫ҙΛʂʢύϑΥʔϚϯε౳ʣ

Slide 41

Slide 41 text

GoΛ࢖ͬͨόοΫΤϯυ։ൃ • ผʹPythonͰ΋ྑ͔ͬͨͷ͕ͩ, ࢖͍ͨͯ͘GoΛ࢖ͬͨʢ࠷େͷཧ༝ʣ. • RESTful APIΛ࡞ΔͱܾΊͨ࣌, ʮ͜ΕͬͯGoͳΒεϚʔτʹ࡞ΕΔ?ʯ   ͱ͍͏஌ࣝϕʔεͷԾઆ͓Αͼ, ϓϩάϥϛϯάΛָ͠ΉͨΊܾߦ. • ࣗ෼Ͱ࡞ͬͯಈ͔ͨ͠ॴ, ʮεϚʔτʹ࡞Εͦ͏ʯͱ͍͏ૂ͍͸ݟࣄతத. • ContainerαΠζͷॖখ, จ۟ͳ͍ύϑΥʔϚϯε. • go fmt, go test౳ͷ͓ӄͰDevOpsʢCI/CDʣύΠϓϥΠϯ͕ચ࿅͞ΕͨϞϊʹ.  

Slide 42

Slide 42 text

Ϣʔεέʔε͝ͱͷٕज़બఆ σʔλऩूج൫ฤ

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

σʔλऩूج൫ͷϢʔεέʔε ᶃ Cron Scheduler͕ຖ೔ܾ·ͬͨ࣌ؒʹᶄͷTriggerΛݺͼग़͢ ᶄ Trigger͸σʔλऩूαʔϏεʢCrawlerʣʹඞཁͳύϥϝʔλʔΛ࡞ͬͯ౉͢ ᶅ Crawler͸Trigger͔ΒͷύϥϝʔλʔΛ࢖ͬͯσʔλݩαΠτ͔ΒCSVΛμ΢ϯϩʔυ, Datalakeʹอଘ ᶆ Cron SchedulerͷτϦΨʔΛݩʹImporter͸CSVσʔλΛDWHʢBigQueryʣʹ౤ೖ

Slide 45

Slide 45 text

Ϋϥ΢υωΠςΟϒͳʮόονॲཧʯ • Ϋϥ΢υͰ͋Γ͕ͪͳόονॲཧํࣜ • Cloud Functions + Pub/SubΛ࢖ͬͯϐλΰϥεΠονతʹ࡞Δ • BigQueryͱͷ෇͖߹͍ํ

Slide 46

Slide 46 text

Ϋϥ΢υͰ͋Γ͕ͪͳόονॲཧํࣜ ݴޠ ڧΈͱ୹ॴ Ϋϥ΢υωΠςΟϒͳํ๏ͱͯ͠ ʲ΍Γํ࣍ୈʳࣗલͰαʔόʔ࡞Δ ʲڧΈʳ DSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔ ʲ୹ॴʳ ϝϯςɾӡ༻͕໘౗ɾଐਓԽ͢Δةݥ "84&$4'BSHBUFͳΒ͍͍ײ͡ 7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒ Ϋϥ΢υ࢖͏ҙຯ͕ݮΔ͔΋ ʲΦεεϝʳ"QBDIF"JS fl PXΛ Ϋϥ΢υαʔϏεͰ࢖͏ ʲڧΈʳ "JS fl PXͷ࢓૊Έʹ৐ͬͯ؂ࢹɾӡ༻Մ ʲ୹ॴʳ "JS fl PXͷֶशίετɾऔΓѻ͍ ۀ຿Ͱ࢖͏ ͋Δఔ౓ͷن໛͕͋Δ όονॲཧͰ͸ྑ͍ײ͡ͷํ๏ (PPHMF$MPVE "84྆ํ͍͚Δ ʲཁ͕݅߹͑͹Φεεϝʳ 1VC4VCʹΑΔΠϕϯτۦಈ ʲڧΈʳ ϚΠΫϩαʔϏεతʹ࡞ΕΔ ʲ୹ॴʳ ઃܭɾ࣮૷Λཧղɾ׳ΕΔֶशίετ Ϋϥ΢υαʔϏεͳΒͰ͸ͷํ๏ ઃܭɾ࣮૷ͷֶशίετ͸͔͔Δ͕ ׳Εͯ͠·͑͹݁ߏؾܰʹ࡞ΕΔ

Slide 47

Slide 47 text

ࠓճ͸Pub/SubΠϕϯτۦಈͰߏங ݴޠ ڧΈͱ୹ॴ Ϋϥ΢υωΠςΟϒͳํ๏ͱͯ͠ ʲ΍Γํ࣍ୈʳࣗલͰαʔόʔ࡞Δ ʲڧΈʳ DSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔ ʲ୹ॴʳ ϝϯςɾӡ༻͕໘౗ɾଐਓԽ͢Δةݥ "84&$4'BSHBUFͳΒ͍͍ײ͡ 7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒ Ϋϥ΢υ࢖͏ҙຯ͕ݮΔ͔΋ ʲΦεεϝʳ"QBDIF"JS fl PXΛ Ϋϥ΢υαʔϏεͰ࢖͏ ʲڧΈʳ "JS fl PXͷ࢓૊Έʹ৐ͬͯ؂ࢹɾӡ༻Մ ʲ୹ॴʳ "JS fl PXͷֶशίετɾऔΓѻ͍ ۀ຿Ͱ࢖͏ ͋Δఔ౓ͷن໛͕͋Δ όονॲཧͰ͸ྑ͍ײ͡ͷํ๏ (PPHMF$MPVE "84྆ํ͍͚Δ ʲ࠾༻ʳ 1VC4VCʹΑΔΠϕϯτۦಈ ʲڧΈʳ ϚΠΫϩαʔϏεతʹ࡞ΕΔ ʲ୹ॴʳ ઃܭɾ࣮૷Λཧղɾ׳ΕΔֶशίετ Ϋϥ΢υαʔϏεͳΒͰ͸ͷํ๏ ઃܭɾ࣮૷ͷֶशίετ͸͔͔Δ͕ ׳Εͯ͠·͑͹݁ߏؾܰʹ࡞ΕΔ

Slide 48

Slide 48 text

Pub/Sub + Cloud FunctionsͳϐλΰϥεΠον • crontab໾ͷΫϥ΢υαʔϏεʢCloud Schedulerʣ͔ΒPub/Subܦ༝Ͱୟ͘ࣄͰόονॲཧΛ࣮ݱ • ֤Cloud Functionsؔ਺ʢᶄ, ᶅ, ᶆʣ͸Pub/Sub͔ΒͷϝοηʔδΛड͚ͯಈ͘࢓૊ΈͰ࡞Δ • ͜ΕΒͷํ๏͸Google Cloudެࣜʹ΋Quick Start͕͋ΔͷͰਅࣅͨ͠Β࣮ݱՄೳʢਅࣅ͠·ͨ͠ʣ • ͪͳΈʹ؀ڥ͸Cloud RunͳͲ, ଞͷαʔϏε΋࢖͑·͢&ݴޠ͸Python͡Όͳͯ͘΋ߏ͍·ͤΜ

Slide 49

Slide 49 text

Pub/SubʹΑΔΠϕϯτۦಈΛબ୒ͨ͠ཧ༝ • ӡ༻ɾίετ྆໘ͰαʔόϨεͷϝϦοτΛ׆͔ͨ͢Ί. • ࣗલαʔόʔ͸ΠϯϑϥΛࣗݾ؅ཧ͠ͳ͍ΞΧϯ&VMͩͱ͓ۚ΋͔͔Δ. • Air fl owʢCloud Composerʣ͸GKEΫϥελ͕ඞཁͰίετ໘Ͱͷෆ҆. • Cloud Functions + Cloud Scheduler + Pub/SubͰαʔόϨεԽ͕Ұ൪ཁٻ࢓༷ʹ߹͏ͱ൑அ • ֤ॲཧΛϚΠΫϩαʔϏεͱ࣮ͯ͠૷Մೳʢೋ൪໨ͷཧ༝ʣ. • ಠཱͨ͠ॲཧ୯ҐͰCloud Functionsͷؔ਺Λ࡞Δ͜ͱͰૄ݁߹ͳϚΠΫϩαʔϏεʹͳΔ. • ֤ؔ਺ͰϓϩάϥϛϯάݴޠɾFrameworkͷมߋ͕ग़དྷΔ, ςετͷ͠΍͢͞ͱ͍͏ϝϦοτ.

Slide 50

Slide 50 text

• σʔλநग़ʢExtractʣ, ม׵ʢTransformʣ, ૹग़ʢLoadʣͦΕͧΕͰ෼͚ͯ࡞ΔͱεοΩϦ͠·͢.  ※ࠓճ͸࢓্༷TransformͱLoad͸Ұॹʹ͍ͯ͠·͢ʢͦͷํ͕εοΩϦͨͨ͠Ίʣ • ॲཧͷτϦΨʔ͸Cloud Scheduler౳ͷαʔϏεΛCron୅ΘΓʹ͢Δͱ࣮૷͕͍͍ײ͡ʹলུͰ͖·͢. • Cloud Functionsʢ&ଞࣾΫϥ΢υؚΊͨFaaSʣ͸ϦΫΤετ͋ͨΓͷॲཧ࣌ؒɾϦιʔε੍ݶ͕͋ΔͷͰ஫ҙ • Cloud Functionsʢୈೋੈ୅ʣ͸ϦΫΤετ͋ͨΓͷ࠷େॲཧ࣌ؒ͸60෼ʢHTTPܦ༝ͷ৔߹ʣ, CPU/ϝϞϦ΋੍ݶ͋Γ·͢. • AWS Lambda౳ͷαʔϏε΋ͦΕͧΕ੍໿͕͋ΔͷͰ͝஫ҙΛ, େ͖͍σʔλ͸Dataflow౳, ઐ༻αʔϏεΛ࢖͏͜ͱΛݕ౼.  ※FaaS: Function as a Serviceʢؔ਺ϕʔεͷΫϥ΢υαʔϏε, Cloud Functions, AWS LambdaͳͲʣ Cloud FunctionsΛ࢖ͬͨόονॲཧͷ࣮૷

Slide 51

Slide 51 text

BigQueryͱͷ෇͖߹͍ํ • αΫοͱ࢖͑ΔεέʔϧՄೳͳDWH͕ཉ͔ͬͨ͠ΒBigQuery͍͍ͧ • ʢࠓճͷج൫Ͱ͸ͳ͘ʣݱ࣮ͷۀ຿Ͱ΋εέʔϧՄೳͳॴʹԿ౓͔ٹΘΕ·ͨ͠. • ݸਓར༻Ͱ࢖͍׳Ε͓ͯ͘ͱͳ͓ྑ͍Ͱ͢. • ݄౰ͨΓ, 10GBͷσʔλอ؅, 1TBͷΫΤϦ࣮ߦ͸ແྉ. • ݸਓͰ΍ͬͯͯ͜ͷྔ͸ͦ͏ͦ͏ߦ͔ͳ͍ϋζͳͷͰ࢖͏ͱ͍͍͔΋. • Ұ౓σʔλΛೖΕ͓ͯ͘ͱ, ΞϓϦ͔Β࢖͏ɾΞυϗοΫ෼ੳ྆ํศརͰ͢.   ࢲ͸΋͏BigQuery͔ΒಀΕΔ͜ͱ͸ग़དྷ·ͤΜʢਅإʣ.

Slide 52

Slide 52 text

ٕज़తͳ࿩ͰർΕ͖ͯ·ͤΜ͔? ʮਪ͠ͷ໺ٿબखʯͷ࿩Λ͠·͢Ͷ⽁

Slide 53

Slide 53 text

WBCͰઈରݟಀͤͳ͍ ʮ֤ࠃͷ௒ਓͨͪʯ • ਎ମೳྗ͓Խ͚ͰΩϨοΩϨ • ύϫʔ, ڧݞ, ޷कͦͯ͠٭͕ചΓ • ౤͛ͯ͸߽଎ٿorΤά͍มԽٿ • ͦ͏͍͏௒ਓͨͪΛ͝঺հ • ݱ໾࣌୅ͷ৽ঙ߶ࢤͬΆ͍   Ӊ஦ਓબख΋͍Δ͔΋🤘

Slide 54

Slide 54 text

WBCʹग़৔͢Δʢϋζʣ, ֤ࠃ୅දͷ௒ਓϚϯ • ถࠃ୅දʮϝδϟʔφϯόʔϫϯͷकඋྗΛތΔαʔυʯ • υϛχΧ୅දʮະདྷͷΠνϩʔ or ৽ঙ??࿘ອ͋;ΕΔηϯλʔʯ • ࣆJAPANʮϝδϟʔϦʔά༗๬ג֎໺ख͕ຬΛ࣋ͯ͠ͷ͝ొ৔ʯ

Slide 55

Slide 55 text

ϊʔϥϯɾΞϨφυ ʢถࠃ୅දʣ • 10೥࿈ଓΰʔϧυάϥϒ৆ͱ͍͏   ϝδϟʔ۶ࢦͷڧଧͷࡾྥख • ଧऀͱͯ͠͸ۃ୺ͳҾͬுΓϚϯ   ӈଧͪͰ௕ଧ͸΄΅ࠨํ޲ • ถࠃ୅දओཁϝϯόʔͷҰਓ   ࡾྥकඋͱϑϧεΠϯάʹ஫໨

Slide 56

Slide 56 text

ϑϦΦɾϩυϦήε ʢυϛχΧ୅දʣ • λϨϯτ܉ஂυϛχΧ஫໨ͷएखελʔ.   2022೥MLB৽ਓԦ, All MLB 2ndνʔϜೖΓ. • ࡾৼ͔௕ଧͱ͍͏ۃ୺ͳଧܸελΠϧ,   ҙ֎ͱ޿֯ʹଧͯΔଧܸηϯε,   Πϝʔδతʹ͸৽ঙ߶ࢤͬΆ͍. • ॴଐ͢ΔϚϦφʔζͰ͸Πνϩʔࢯ͕   ࿅शύʔτφʔΛ͍ͯ͠Δ͜ͱͰ΋࿩୊.

Slide 57

Slide 57 text

ϥʔζɾψʔτόʔ ʢࣆJAPANʣ • ࣆJAPANॳͷΞϝϦΧग़਎બख • ଧ཰͸௿͍΋ͷͷग़ྥ཰ͱ௕ଧ཰   ͦͯ͠ଧٿ଎౓͕଎͘, 3ϙδγϣϯ   ͦͭͳ͘कΕΔ֎໺कඋ΋˕ • Ωϟϥཱ͍͍ͪͯͯ͠Ϡπͱ͍͏ᷚ,   ࣆδϟύϯͷηϯλʔͱͯ͠ظ଴ʂ

Slide 58

Slide 58 text

ࣆJAPANͱ͍͑͹, ͜ͷೋਓ΋ ݟಀͤͳ͍௒ਓ ώϯτ: ݩ೔ϋϜͷഎ൪߸11

Slide 59

Slide 59 text

μϧϏογϡ༗͜ͱ, ౤ٿεϖγϟϦετͳ@faridyu͞Μ ୣࡾৼͷଟ͞ͱΧοτϘʔϧ୔ࢁ…WBCͰͲ͏͍͏ϓϨʔ͢Δ͔ظ଴͍ͨ͠.

Slide 60

Slide 60 text

େ୩ᠳฏʢࢦ໊ଧऀʣͷओཁͳଧٿσʔλ ҾͬுΓϚϯͰ͸ͳͯ͘޿֯ʹ଎͍ଧٿଧͬͯͯੌ͍ʢ͜ͳΈʣ

Slide 61

Slide 61 text

݁ͼ

Slide 62

Slide 62 text

ʲ࠶ܝʳຊ೔ͷελʔςΟϯάϝϯόʔ • Ϋϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX • αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ • ʮਪ͠ͷ໺ٿબखʯͷʮਪ࣌͠ʯ͸Ͳ͜ͳͷ͔ΛݟۃΊΔ⽁

Slide 63

Slide 63 text

ࠓ೔ͷ࿩Λཁ໿͢Δͱ… • ϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥ΢υ͚ͩͰσʔλج൫͸࡞ΕΔ. • Ϋϥ΢υαʔϏεͱϓϩάϥϛϯάݴޠ͸༻్ʹ߹Θͤͯબ΅͏. • ࣆJAPANʹ͸ڧྗͳϥΠόϧ͕ͨ͘͞Μ͍Δʢ͜ͳΈʣ.

Slide 64

Slide 64 text

ʮԶͷDXϓϩδΣΫτʯര஀ޙͷޮՌ. • ໺ٿσʔλαΠΤϯςΟετɾϓϩμΫτΦʔφʔͱ͍͏ࢹ఺Ͱϝονϟ࢖͑ͦ͏. • ೥ؒ500MB, 91߲໨ͱ͍͏టष͍σʔλΛ࢖͍΍͘͢͢Δ͚͍͍ͩϞϊΛ࡞Εͨʢࣄ࣮ʣ. • σʔλΛݟֶͯΜͩΓ, ϒϩάɾొஃͷωλʹ͢Δ͜ͱͰ։ൃ෼ͷ౤ࢿճऩ͕Ͱ͖ͦ͏. • ݸਓͱͯ͠ͷٕज़ݕূ͕ϝονϟḿΔ༧ײʢ&ࣗݾຬ଍ʣ • Google Cloudͷࢼ͍ͨ͠ػೳɾϞϊΛ࣮ݧ͢Δsandboxͱͯ͠ػೳͦ͠͏, AIܥαʔϏεͱ͔. • ΞʔΩςΫνϟΛͦͷ··ʹAWS΍AzureͰࢼ͢ͷ΋ΞϦ, ϚϧνΫϥ΢υ΋ΠέΔΜ͡ΌͶ? • ࢓ࣄʹඞཁͳεΩϧɾ஌ࣝΛझຯʢࣗݾຬ଍ʣ͔ΒखʹೖΕΔج൫ͱͯ͠΋࠷ߴʹྑ͍.

Slide 65

Slide 65 text

͓࢓ࣄʢۀ຿ʣͰࢀߟʹ͠Α͏ͱࢥͬͨํ΁ • ࠓճ঺հͨ͠΍Γํɾߏ੒͸ઈରతͳճ౴ɾϕετϓϥΫςΟεͰ͸ͳ͍Ͱ͢.   ྫ͑͹αʔόϨεɾΞʔΩςΫνϟʹ͢΂͖/͢΂͖͡Όͳ͍ঢ়گ͸࣮֬ʹଘࡏ͠·͢. • ͜ͷ࿩͸ࢲʢshinyorkeʣ͕΍Γ͍ͨࣄ, ͍͍ͱࢥͬͯΔࣄʢ&৮Γ͍ͨϞϊʣΛ   ٧ΊࠐΜͰ࡞ͬͨ, ࣗ෼͕΍Γ͍ͨࣄͷूେ੒Ͱ, ͋͘·Ͱղͷग़͠ํͷҰͭͰ͢. • αʔόϨεʹͩ͜ΘΒͳ͔ͬͨΒʮGKEΫϥελཱͯͯͦ͜Ͱಈ͔͢ʯͰऴΘΔ.   ͜Ε΋ཱ೿ͳղ౴ͩͬͨΓ͠·͢, ϝϦσϝ࣍ୈͰ. • Infrastructure as CodeͰ͖ͯͳ͍, ςετίʔυ଍Γͳ͍etc…࣮͸՝୊ࢁੵΈ. • ʢίϯςΩετͷཧղ͕த్൒୺ͳ··ʣͦͷ··ਅࣅ͢Δͱരࢮ͠·͢.   ·ͣ͸खΛಈ͔͠, ֶशͨ͠Γಈ͔ͨ͠Γ͍͍ͯ͠΋ͷΛݟ͚ͭΔࢀߟʹͲ͏ͧʂ

Slide 66

Slide 66 text

ʮਪ͠͸ਪͤΔ࣌ʹਪͤʯͱ͍͏ԶͷDXଓฤ. • ϝδϟʔϦʔάɾϓϩ໺ٿͷ੒੷༧ଌΤϯδϯ΋࡞ͬͯΔͷͰ,   ͦΕΛࠓճͷج൫ʹ࣮૷͍ͨ͠, ૬ੑྑ͍͸ͣͳͷͰ. • ࠓޙͷల๬ͱͯ͠, σʔλͷϥΠηϯε౳ͷ՝୊Λղܾ͠,   ͍͍ײ͡ͳ໺ٿσʔλαΠτΛҰൠެ։͢Δʢઈରʹ΍Γ͍ͨʣ. • ࠓޙ΋ٕज़Λ৭ʑࢼ͠, ໺ٿͷ໘നָ͍͠ΈํΛൃ৴ͭͭ͠,   ʮ໺ٿͱٕज़ͷਪ͠ΛϦίϝϯυʯ͢ΔԶͷDXΛ΍͍ͬͯ͘΍Ͱ.

Slide 67

Slide 67 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠⽁ @shinyorke