Upgrade to Pro — share decks privately, control downloads, hide ads and more …

俺のDXを実現するためのサーバレスなデータ基盤開発と運用 / Serverless Data Platform and Baseball

俺のDXを実現するためのサーバレスなデータ基盤開発と運用 / Serverless Data Platform and Baseball

Shinichi Nakagawa

February 10, 2023
Tweet

More Decks by Shinichi Nakagawa

Other Decks in Programming

Transcript

  1. ਪ͠͸ਪͤΔ࣌ʹਪͤ.
    ͱ͍͏ԶͷDXΛ࣮ݱ͢ΔͨΊͷαʔόϨεͳσʔλج൫։ൃͱӡ༻.


    ໺ٿσʔλΛఴ͑ͯ⽁
    @shinyorke 2023/02/10 Developers Summit 2023

    View Slide

  2. Onboardingʢ͜ͷηογϣϯͷ͝Ҋ಺ʣ
    • ࢲ͕झຯͰߏஙͨ͠ʮ໺ٿσʔλϓϥοτϑΥʔϜʯαʔϏεʹ͓͚Δ,

    ʮΫϥ΢υωΠςΟϒͳσʔλΤϯδχΞϦϯάʯࣄྫͷ঺հͰ͢.


    • Ϋϥ΢υɾσʔλج൫Ͱඇৗʹ෯޿͍ൣғͷࣄྫͱͳΓ·͢.

    ࢿྉ͸ൃදޙʹެ։͠·͢&ؾʹͳΔՕॴ͸ࢿྉΛಡΉࣄΛਪ঑͠·͢.


    • ໺ٿʹڵຯͳ͍ʢor޷͖͡Όͳ͍ʣํͱ΋Ұॹʹָ͠ΊͨΒ޾͍Ͱ͢.

    View Slide

  3. ౰αʔϏεͷߏஙͰ࢖ͬͨϓϩάϥϛϯάݴޠɾαʔϏεɾؔ࿈ٕज़ͷ਺ʑ


    ΊͬͪΌଟ͍ͷͰؾʹͳΔͱ͜Ζ͚ͩಡΉɾ֮͑ΔͰ΋େৎ෉Ͱ͢ʂ

    View Slide

  4. ͜ΜͳํʹΦεεϝͰ͢.
    • DWH΍σʔλج൫ͱ͍ͬͨʮσʔλར׆༻ʯͷ͍͍ײ͡ͳߏஙࣄྫΛ஌Γ͍ͨํ.


    • ύϒϦοΫΫϥ΢υ, ओʹGoogle CloudΛ࢖ͬͯԿ͔͠ΒͷࣄΛ͢Δํ.


    • Ϋϥ΢υαʔϏεΛ࢖͍͍ͬͨײ͡ͳγεςϜઃܭɾߏஙʹڵຯ͕͋Δํ.


    • αʔόϨεͳΫϥ΢υαʔϏεΛ࢖͍ͬͯΔ or ͜Ε͔Β࢖͏ํʢڵຯͰ΋͍͍ʣ.

    ※AWS Lambda, AWS App Runner, Google App Engine, Cloud RunͳͲ͕֘౰.


    • ʢ޷͖ݏ͍ؔ܎ͳ͘ʣ໺ٿͷϧʔϧٴͼΦΦλχαϯ͕୭͔ͩ஌͍ͬͯΔ.

    View Slide

  5. Who am I ?


    ʢ͓લ୭Α?ʣ
    • Shinichi [email protected]


    • shinyorke͸ʮ͠ΜΑʔ͘ʯͱಡΈ·͢

    ※໊લͷҰ෦ + ਪ͠όϯυͷϘʔΧϧ͔Βഈआ🎸


    • େख֎ࢿܥITίϯαϧاۀϚωʔδϟʔ

    Ϋϥ΢υɾΠϯϑϥѻ͏νʔϜͰ৭ʑ΍ͬͯΔਓ


    • ࢓ࣄ΋झຯ΋ϑϧαΠΫϧͳΤϯδχΞ


    • ʮLean Baseballʯͱ͍͏ݸਓϒϩάͰ

    ʮٕज़ʯʮ໺ٿʯʮΩϟϦΞʯʹ͍ͭͯ৭ʑॻ͍ͯ·͢.

    https://shinyorke.hatenablog.com/

    View Slide

  6. ॏཁͳεϥΠυ͸͜ͷ഑৭Ͱ͢


    ※10ຕ΋ͳ͍Ͱ͢, ࠷ѱ͜ͷ৭ͷεϥΠυ͚ͩ௥͍ͬͯͩ͘͞.

    View Slide

  7. ຊ೔ͷελʔςΟϯάϝϯόʔ
    • Ϋϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX


    • αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ


    • ʮਪ͠ͷ໺ٿબखʯͷʮਪ࣌͠ʯ͸Ͳ͜ͳͷ͔ΛݟۃΊΔ⽁

    View Slide

  8. ͱ͜ΖͰ, ʮਪ͠ʯͬͯԿ?

    View Slide

  9. ਓ෺ͱ͋Γ·͕͢, ର৅͸ϞϊͰ͋ΕԿͰ͋Εʮਪ͠ʯͬͯݴ͏ؾ͕͢Δʢখ੠ʣ

    গͳ͘ͱ΋ࢲ͸ͦͷೝࣝʢҟ࿦͸ೝΊΔʣ
    “ਪ͠ʢ͓͠ʣͱ͸ɺओʹΞΠυϧ΍ആ༏ʹ
    ͍ͭͯ༻͍ΒΕΔ೔ຊޠͷଏޠͰ͋Γɺਓ
    ʹનΊ͍ͨͱࢥ͏΄Ͳʹ޷ײΛ͍࣋ͬͯΔ
    ਓ෺ͷ͜ͱΛ͍͏ɻ” by Wikipedia
    9

    View Slide

  10. झຯʹ͓͚Δʮਪ͠ʯ͸͋Γ·͔͢?


    ྑ͔ͬͨΒͭͿ΍͍ͯΈ͍ͯͩ͘͞

    View Slide

  11. ࢲͷʮझຯʹ͓͚Δਪ͠ʯ͸
    • ໺ٿʢੲ͸࢓ࣄʹ͍ͯͨ͠ʣ


    • ਎ମೳྗ͕ߴͯ͘ෆࢥٞͳΩϟϥͷ֎໺ख


    • ηΠόʔϝτϦΫεʢ໺ٿ౷ܭֶʣϮλΫ


    • ͦͷଞͷझຯʢଞʹ΋୔ࢁ͋Δʣ


    • ڝഅΛݟΔͷ͕େ޷͖ʢ੨ࣛໟɾࠇࣛໟͷഅ͕ಛʹ޷͖ʣ🐴


    • 90s-00s͙Β͍ͷUK Rockେ޷͖🎸ʢblur, RADIOHEAD͕େ޷෺ʣ

    View Slide

  12. ٕज़ʹ͓͚Δʮਪ͠ʯ͸͋Γ·͔͢?


    ྑ͔ͬͨΒͭͿ΍͍ͯΈ͍ͯͩ͘͞

    View Slide

  13. ࢲͷʮਪٕ͠ज़ʯͷ਺ʑʢ΄ΜͷҰྫͰ͢, ΋ͬͱͨ͘͞Μ͋Δʣ.


    ࢓ࣄ΋झຯ΋͍ͭ΋͜ΕΒͷ΋ͷʹॿ͚ΒΕ͍ͯ·͢.

    View Slide

  14. ͜ͷઌ͸ʮٕज़ʯʮΫϥ΢υʯʮ໺ٿʯͷʮਪ͠ʯͰ࣮ݱͨ͠


    ʮΫϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDXʯͷ࿩Λ͠·͢ʂ

    View Slide

  15. ਪ͠ͷ໺ٿબखʮΦΦλχαϯͷ2022೥ʯΛৼΓฦΔ


    σʔλࢀরݩ: https://baseballsavant.mlb.com/statcast_search ※ΦʔϓϯσʔλͰ͢

    View Slide

  16. 2022೥ͷΦΦλχαϯ,


    εϥΠμʔͱ2γʔϜͰ


    ʮบ͕ੌ͍ʯϐονϟʔʹ
    • ࠓ೥ͷΦΦλχαϯ, ΊͬͪΌ

    εϥΠμʔ౤͍͛ͯΔ


    • ͓ؾ͖ͮͩΖ͏͔?ޙ൒ઓ͸

    2γʔϜʢσʔλ্͸Sinkerʣ͕

    ૿͍͑ͯΔ͜ͱʹ!?


    • γʔζϯޙ൒͔ΒͷΩϟϥมͰແ૒ঢ়ଶ.

    ਅͬ௚͙ओମͷελΠϧ͔Βม਎੒ޭ.

    View Slide

  17. ͱ͋ΔΦΦλχαϯͷొ൘೔ʢ2022/9/29, 8ճ10ୣࡾৼແࣦ఺ʣ


    શ౤ٿͷ͏ͪ, ७ਮͳετϨʔτʢ4-Seamʣ͕Θ͔ͣ3.7%, ଞ͸͢΂ͯۂ͛ΔϘʔϧ.
    ౤͛ͨ৔ॴʢัख໨ઢʣ ϦϦʔεϙΠϯτʢัख໨ઢʣ
    ٿछͷׂ߹

    View Slide

  18. ϫΠʮਏ͍Ͱ͢…ຖճSQLͱίʔυΛॻ͖ଓ͚Δͷ͕ʯ


    ޢຎߦ͡Όͳ͍ͷͰ΋ͬͱؾָʹ΍Γ͍ͨʢޢຎߦܦݧͳ͍Ͱ͕͢ʣ.

    View Slide

  19. ϫΠʮͦ͏ͩ, ⚾σʔλج൫࡞Ζ͏ʯ


    ԶͷDXϓϩδΣΫτ, ര஀.


    σʔλʢhttps://baseballsavant.mlb.com/ ʣ͕࢖͍ʹ͍͘, ίʔυॻ͘ͷਏ͍.


    ͩͬͨΒࣗ෼Ͱ࡞Γ΍͍͢΋ͷ࡞ͬͯ͠·͑ʂͱ͍͏ΞΠσΞ͕߱ྟ.

    View Slide

  20. ࡞Γ·ͨ͠ʢ૯࡞੒ظؒɾ໿൒೥ʣ.

    View Slide

  21. ΞϓϦͱσʔλج൫ͷΞʔΩςΫνϟ


    ʢʮԶͷDXʯੈքͷ஍ਤʣ

    View Slide

  22. View Slide

  23. ΞʔΩςΫνϟղઆʢ㲈ͩ͜ΘΓϙΠϯτʣ
    • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υར༻


    • ͓ࡒ෍ʹ༏͍͠ߏ੒ͱ࢖͍ํʹప͢Δ

    View Slide

  24. ʮϑϧϚωʔδυ͔ͭαʔόϨεʯ


    ͳΫϥ΢υαʔϏεͱ͸?


    Google CloudΛྫʹ͢Δͱ

    View Slide

  25. ʲਤʳGoogle Cloudͷ੹೚ڞ༗ϞσϧʢComputeܥݶఆʣ

    View Slide

  26. View Slide

  27. ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υ
    • ʮϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥ΢υαʔϏεʯͷಛ௃


    • Πϯϑϥɾαʔόʔͷϝϯςφϯε͕ෆཁʢࣗ෼͡Όͳͯ͘, Ϋϥ΢υαʔϏεଆ͕΍Δʣ


    • ΑΓ۩ମతʹ͸, ࣗ෼ͰK8sΫϥελ΍VMΛݐͯͳͯ͘΋ྑ͍αʔϏεͷ͜ͱ


    • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯ͸ʮख͕͔͔ؒΒͳ͍ʯ


    • ΄΅ϝϯςφϯεϑϦʔ.ϛυϧ΢ΣΞͷอकɾӡ༻͔Β։์͞ΕΔ.


    • εέʔϥϏϦςΟͷ୲อ͕͠΍͍͢. ඞཁʹԠͯ͡εέʔϧΞ΢τɾεέʔϧΠϯָ͕.


    • Ұݟ͢ΔͱαʔόϨεߏ੒͸ύʔϑΣΫτʹݟ͑·͕͢, ϦΫΤετ͋ͨΓͷॲཧ࣌ؒ,

    ࢖͑ΔϦιʔεྔͷ੍ݶɾϥϯλΠϜVersionͷ੍ݶ౳, ߟྀ͢΂͖ཹҙ఺ɾܽ఺΋͋Γ·͢.

    View Slide

  28. ʁʁʁʮݸਓͰΫϥ΢υαʔϏεΛ͕ͬͭΓ࢖͏ͱ͓͕ۚ৺഑ʯ


    ͬͯࢥ͏͡Όͳ͍Ͱ͔͢, Ϋϥ΢υͱαʔόϨεͰ͍͍ײ͡ʹग़དྷ·͢.

    View Slide

  29. αʔόϨεͳΫϥ΢υ͸͓ࡒ෍ʹ༏͍͠
    • ʮ࢖͏ͱ্ཱ͖͚͕ͩͪΕ͹͍͍ʯͷͰΞϓϦέʔγϣϯج൫͸αʔόϨεʹશৼΓ


    • WebΞϓϦέʔγϣϯ΋όονॲཧ΋ίʔϧυελϯόΠ


    • ʮಈ͍͍ͯͳ͍࣌΋͓͕͔͔ۚΔʯͷ͸DWHͱετϨʔδͷΈʢͦΕ΋͘͝গֹʣ


    • DatabaseʢBigQueryʣͱCloud Storage͸࢖͍ํΛ޻෉͠ίετΛ཈͑Δ


    • ແྉ࿮ͷϧʔϧΛཧղ͠, ͦͷൣᙝͰۃྗ࢖͏ʢ㲈ඞཁͳ΋ͷʹ͸͓ۚΛ෷͏ʣ


    • σʔλͷ௕ظอ؅ϧʔϧͱετϨʔδλΠϓͷ޻෉, ҰׅॲཧͰͷσʔλΠϯϙʔτ

    View Slide

  30. Ұϲ݄Ͱ$3ະຬ, ίʔώʔ3.34ഋ෼ఔ౓ͰࡁΈ·ͨ͠🐯

    ※σʔλج൫ϓϩδΣΫτͷ࣮ίετΑΓࢉग़ʢίʔώʔ୅͸ίϯϏχίʔώʔج४ʣ, υϝΠϯऔಘྉ౳֎෦ίετΛআ͘

    View Slide

  31. ΞʔΩςΫνϟղઆͷ·ͱΊ
    • ʮϑϧϚωʔδυ͔ͭαʔόϨεʯͳΫϥ΢υαʔϏεத৺Ͱߏ੒


    • อकɾӡ༻࡞ۀ͔Βͷղ์, εέʔϥϏϦςΟͷ୲อʹ༗ར


    • αʔόϨεͳΒͰ͸ͷ੍໿ࣄ߲ʹ஫ҙʢॲཧ࣌ؒɾϦιʔε౳ʣ


    • ͓ࡒ෍ʹ༏͍͠࢖͍ํ͕Ͱ͖Δ


    • ʮ࢖͚͓͕͔͔ͬͨ࣌ͩۚΔʯίετߏ଄΁ͷม׵


    • ແྉ࿮ɾσʔλ௕ظอ؅ϧʔϧΛ׆༻ͯ͠ίετѹॖ

    View Slide

  32. Ϣʔεέʔε͝ͱͷٕज़બఆ


    μογϡϘʔυΞϓϦฤ

    View Slide

  33. View Slide

  34. μογϡϘʔυΞϓϦͷϢʔεέʔε
    • σϞΞϓϦέʔγϣϯʢWebΞϓϦέʔγϣϯʣ͸Cloud Run্Ͱϗετ. ࣮૷͸PythonʢDashʣ.
    • όοΫΤϯυͷೝূ͸API GatewayΛ༻͍ͯߦ͏, API KeyํࣜͷೝূʢSaaSతʹ࢖͏ͨΊ׶͑ͯ͜͏΍͍ͬͯΔʣ.
    • BigQueryʹ֨ೲ͞ΕͨσʔλΛݕࡧ͢ΔόοΫΤϯυʢRESTful APIʣ͸GoʢGinʣͰ࣮૷, Cloud RunͰϗετ.
    • RESTful APIͷResponse͸৚݅ผʹCloud StorageʹΩϟογϡʢಉ͡ΫΤϦΛԿ౓΋࣮ߦͤ͞ͳ͍ʣ.

    View Slide

  35. ʮΫϥ΢υαʔϏεબͼʯͷ೉͠͞Λղܾ͢Δ
    • αʔόϨεͳΫϥ΢υαʔϏεͷબͼํ - Google Cloudฤ


    • App EngineͱFirebaseͱCloud Run, Կ͕ҧ͏ͷ?


    • Cloud Functions͍ͬͯͭ࢖͏ͷ?ͦ΋ͦ΋Կऀ??


    • …ͱ͍ͬͨΫΤενϣϯʹ͓౴͑͠·͢


    • ϓϩάϥϛϯάݴޠͷબͼํ

    View Slide

  36. αʔόϨεͳΞϓϦέʔγϣϯಈ࡞؀ڥ
    ओͳαʔϏεʢ(PPHMFʣ 63- ֓ཁ
    "QQ&OHJOF IUUQTDMPVEHPPHMFDPNBQQFOHJOF
    αʔόϨε͔ͭϑϧϚωʔδυͷ
    ݩ૆Έ͍ͨͳଘࡏ ࢖͍΍͍͢
    'JSFCBTF IUUQT
    fi
    SFCBTFHPPHMFDPN
    ΞϓϦͷΈͳΒͣ %#ɾ௨஌ͱ
    ΍ΕΔ͜ͱ͕ଟ͍͔ͭศར
    ͳ͓ݴޠ͸+BWB4DSJQUͷΈ
    $MPVE3VO IUUQTDMPVEHPPHMFDPNSVO
    ޷͖ͳݴޠɾ؀ڥͰ࡞ΔͳΒ͜Ε

    ͻͱ·ͣ$POUBJOFS࡞ͬͯಈ͔ͤΔ
    $MPVE'VODUJPOT IUUQTDMPVEHPPHMFDPNGVODUJPOT
    ͪΐͬͱͨؔ͠਺Λಈ͔͢ͳΒ
    4MBDL#PU౳ͷখ͞ΊͳΞϓϦͳͲ

    View Slide

  37. Cloud RunΛબ୒ͨ͠ཧ༝
    • σϞΞϓϦɾόοΫΤϯυڞʹContainerʢDockerʣϕʔεͰߏங͍ͯͨͨ͠Ί


    • ։ൃத͸ґଘ͢ΔϥΠϒϥϦ͕ෆಁ໌ͩͬͨͨΊDocker Container૝ఆͰ։ൃ


    • ContainerΛͦͷ··ಈ͔ͤΔCloud Run͕࠷΋ָ͔ͭ׳Ε͍ͯͨͷͰ࠾༻


    • Cloud RunΛબ୒ͨ͠ཧ༝ͱഎܠ


    1.ґଘϥΠϒϥϦͳͲͷ౎߹ͰPython or GoͰͷ։ൃ -> Firebase͕ީิ͔Βফ͑Δ


    2.Cloud FunctionsͰಈ͔͢ʹ͸ඍົͳ࢓༷ͱͳͬͨͨΊ, Cloud Funcitons΋ফ͑Δ


    3.App EngineʢStandardʣ͸Container͡Όͳ͍ -> Cloud Runʹܾఆ

    ※ContainerͰಈ͔ͤΔApp EngineʢFlexibleʣΛ࢖Θͳ͔ͬͨཧ༝͸…ؾʹͳΔํ͸ฉ͍͍ͯͩ͘͞

    View Slide

  38. ީิͱͯ͠ݕ౼ͨ͠ϓϩάϥϛϯάݴޠ
    ݴޠ ݕ౼ͷഎܠ ݁Ռ
    1ZUIPOʢ8FCΞϓϦɾόονʣ
    Ұ൪࢖͍׳Ε͍ͯͯαΫοͱ࡞ΕΔ
    ςετɾσϓϩΠ౳ͷ؀ڥपΓ͕
    গʑ໘౗
    σϞ༻ͷσʔλΞϓϦ͓Αͼ
    σʔλج൫ͷؔ਺Λ1ZUIPOʹ
    ͳ͓Ұ෦(PͰॻ͖׵͑Δ༧ఆ
    (Pʢ3&45GVM"1*ʣ
    ެࢲͱ΋ʹຆͲ࢖͍ͬͯͳ͍͕޷͖
    ςετɾίʔυϑΥʔϚολʔ౳͕
    ޙൃݴޠ͚ͩ͋ͬͯ࢖͍΍͍͢
    ϓϩάϥϛϯάݴޠͱͯ͠ͷಛੑ

    $POUBJOFS։ൃͱͷ૬ੑൈ܈ͩͬͨ
    ͷͰόοΫΤϯυͷݴޠͱͯ͠࢖༻
    5ZQF4DSJQUʢࠓճ͸ෆ࠾༻ʣ
    1ZUIPOͷ࣍ʹ࢖͍ͬͯΔ
    ϑϩϯτΤϯυ࡞ΔͳΒ͜Ε
    όοΫΤϯυ͸ଞʹީิ༗Γ
    %BTIʢ1ZUIPOͷ'SBNFXPSLʣଆ

    ͰϑϩϯτΤϯυΛੜ੒͢Δࣄʹ
    ͨ͠ͷͰࠓճ͸࢖Θͣ

    View Slide

  39. ΞϓϦ͸Low-code🐍
    • ݁࿦͔Βݴ͏ͱʮDashʯͱ͍͏

    PythonͷLow-codeͰ࣮૷.


    • ϓϩτλΠϓͰ͸෼ੳɾՄࢹԽʹ

    Jupyter LabͱPlotlyΛ࢖͓ͬͯΓ,

    ͜ΕΛͦͷ··Ҡ২Ͱ͖ΔखஈͰ

    ࣮૷͔ͨͬͨ͠ʢબ୒ͨ͠എܠʣ


    • https://dash.plotly.com/

    View Slide

  40. Dash for Pythonʢྫʣ
    • DashΛ࢖͏ͱPython͚ͩͰϑϩϯτॻ͚·͢.


    • HTMLʹ໛ͨ͠ίϯϙʔωϯτΛPythonͰ

    ΰϦͬͱॻ͍ͯ࡞੒͢Δͱ͍͍ײ͡ʹಈ͘.


    • ΠϕϯτۦಈͰͷॻ͖׵͑͸Callbackͳ

    σίϨʔλʔͰ࣮૷ʢงғؾ΄΅Reactʣ.


    • ຊ෺ͷϑϩϯτΤϯυΑΓෆརͳ఺΋ଟ͍ͷͰ

    ࢖͍ॴʹ͸े෼஫ҙΛʂʢύϑΥʔϚϯε౳ʣ

    View Slide

  41. GoΛ࢖ͬͨόοΫΤϯυ։ൃ
    • ผʹPythonͰ΋ྑ͔ͬͨͷ͕ͩ, ࢖͍ͨͯ͘GoΛ࢖ͬͨʢ࠷େͷཧ༝ʣ.


    • RESTful APIΛ࡞ΔͱܾΊͨ࣌, ʮ͜ΕͬͯGoͳΒεϚʔτʹ࡞ΕΔ?ʯ

    ͱ͍͏஌ࣝϕʔεͷԾઆ͓Αͼ, ϓϩάϥϛϯάΛָ͠ΉͨΊܾߦ.


    • ࣗ෼Ͱ࡞ͬͯಈ͔ͨ͠ॴ, ʮεϚʔτʹ࡞Εͦ͏ʯͱ͍͏ૂ͍͸ݟࣄతத.


    • ContainerαΠζͷॖখ, จ۟ͳ͍ύϑΥʔϚϯε.


    • go fmt, go test౳ͷ͓ӄͰDevOpsʢCI/CDʣύΠϓϥΠϯ͕ચ࿅͞ΕͨϞϊʹ.

    View Slide

  42. Ϣʔεέʔε͝ͱͷٕज़બఆ


    σʔλऩूج൫ฤ

    View Slide

  43. View Slide

  44. σʔλऩूج൫ͷϢʔεέʔε
    ᶃ Cron Scheduler͕ຖ೔ܾ·ͬͨ࣌ؒʹᶄͷTriggerΛݺͼग़͢
    ᶄ Trigger͸σʔλऩूαʔϏεʢCrawlerʣʹඞཁͳύϥϝʔλʔΛ࡞ͬͯ౉͢
    ᶅ Crawler͸Trigger͔ΒͷύϥϝʔλʔΛ࢖ͬͯσʔλݩαΠτ͔ΒCSVΛμ΢ϯϩʔυ, Datalakeʹอଘ
    ᶆ Cron SchedulerͷτϦΨʔΛݩʹImporter͸CSVσʔλΛDWHʢBigQueryʣʹ౤ೖ

    View Slide

  45. Ϋϥ΢υωΠςΟϒͳʮόονॲཧʯ
    • Ϋϥ΢υͰ͋Γ͕ͪͳόονॲཧํࣜ


    • Cloud Functions + Pub/SubΛ࢖ͬͯϐλΰϥεΠονతʹ࡞Δ


    • BigQueryͱͷ෇͖߹͍ํ

    View Slide

  46. Ϋϥ΢υͰ͋Γ͕ͪͳόονॲཧํࣜ
    ݴޠ ڧΈͱ୹ॴ Ϋϥ΢υωΠςΟϒͳํ๏ͱͯ͠
    ʲ΍Γํ࣍ୈʳࣗલͰαʔόʔ࡞Δ
    ʲڧΈʳ
    DSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔ
    ʲ୹ॴʳ
    ϝϯςɾӡ༻͕໘౗ɾଐਓԽ͢Δةݥ
    "84&$4'BSHBUFͳΒ͍͍ײ͡
    7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒ
    Ϋϥ΢υ࢖͏ҙຯ͕ݮΔ͔΋
    ʲΦεεϝʳ"QBDIF"JS
    fl
    PXΛ
    Ϋϥ΢υαʔϏεͰ࢖͏
    ʲڧΈʳ
    "JS
    fl
    PXͷ࢓૊Έʹ৐ͬͯ؂ࢹɾӡ༻Մ
    ʲ୹ॴʳ
    "JS
    fl
    PXͷֶशίετɾऔΓѻ͍
    ۀ຿Ͱ࢖͏ ͋Δఔ౓ͷن໛͕͋Δ
    όονॲཧͰ͸ྑ͍ײ͡ͷํ๏
    (PPHMF$MPVE "84྆ํ͍͚Δ
    ʲཁ͕݅߹͑͹Φεεϝʳ
    1VC4VCʹΑΔΠϕϯτۦಈ
    ʲڧΈʳ
    ϚΠΫϩαʔϏεతʹ࡞ΕΔ
    ʲ୹ॴʳ
    ઃܭɾ࣮૷Λཧղɾ׳ΕΔֶशίετ
    Ϋϥ΢υαʔϏεͳΒͰ͸ͷํ๏
    ઃܭɾ࣮૷ͷֶशίετ͸͔͔Δ͕
    ׳Εͯ͠·͑͹݁ߏؾܰʹ࡞ΕΔ

    View Slide

  47. ࠓճ͸Pub/SubΠϕϯτۦಈͰߏங
    ݴޠ ڧΈͱ୹ॴ Ϋϥ΢υωΠςΟϒͳํ๏ͱͯ͠
    ʲ΍Γํ࣍ୈʳࣗલͰαʔόʔ࡞Δ
    ʲڧΈʳ
    DSPOUBCͱԿ͔͠ΒͷݴޠͰ࡞ΕΔ
    ʲ୹ॴʳ
    ϝϯςɾӡ༻͕໘౗ɾଐਓԽ͢Δةݥ
    "84&$4'BSHBUFͳΒ͍͍ײ͡
    7.ܥͩͬͨΓϚγϯ͕ඞཁͳΒ
    Ϋϥ΢υ࢖͏ҙຯ͕ݮΔ͔΋
    ʲΦεεϝʳ"QBDIF"JS
    fl
    PXΛ
    Ϋϥ΢υαʔϏεͰ࢖͏
    ʲڧΈʳ
    "JS
    fl
    PXͷ࢓૊Έʹ৐ͬͯ؂ࢹɾӡ༻Մ
    ʲ୹ॴʳ
    "JS
    fl
    PXͷֶशίετɾऔΓѻ͍
    ۀ຿Ͱ࢖͏ ͋Δఔ౓ͷن໛͕͋Δ
    όονॲཧͰ͸ྑ͍ײ͡ͷํ๏
    (PPHMF$MPVE "84྆ํ͍͚Δ
    ʲ࠾༻ʳ
    1VC4VCʹΑΔΠϕϯτۦಈ
    ʲڧΈʳ
    ϚΠΫϩαʔϏεతʹ࡞ΕΔ
    ʲ୹ॴʳ
    ઃܭɾ࣮૷Λཧղɾ׳ΕΔֶशίετ
    Ϋϥ΢υαʔϏεͳΒͰ͸ͷํ๏
    ઃܭɾ࣮૷ͷֶशίετ͸͔͔Δ͕
    ׳Εͯ͠·͑͹݁ߏؾܰʹ࡞ΕΔ

    View Slide

  48. Pub/Sub + Cloud FunctionsͳϐλΰϥεΠον
    • crontab໾ͷΫϥ΢υαʔϏεʢCloud Schedulerʣ͔ΒPub/Subܦ༝Ͱୟ͘ࣄͰόονॲཧΛ࣮ݱ
    • ֤Cloud Functionsؔ਺ʢᶄ, ᶅ, ᶆʣ͸Pub/Sub͔ΒͷϝοηʔδΛड͚ͯಈ͘࢓૊ΈͰ࡞Δ
    • ͜ΕΒͷํ๏͸Google Cloudެࣜʹ΋Quick Start͕͋ΔͷͰਅࣅͨ͠Β࣮ݱՄೳʢਅࣅ͠·ͨ͠ʣ
    • ͪͳΈʹ؀ڥ͸Cloud RunͳͲ, ଞͷαʔϏε΋࢖͑·͢&ݴޠ͸Python͡Όͳͯ͘΋ߏ͍·ͤΜ

    View Slide

  49. Pub/SubʹΑΔΠϕϯτۦಈΛબ୒ͨ͠ཧ༝
    • ӡ༻ɾίετ྆໘ͰαʔόϨεͷϝϦοτΛ׆͔ͨ͢Ί.


    • ࣗલαʔόʔ͸ΠϯϑϥΛࣗݾ؅ཧ͠ͳ͍ΞΧϯ&VMͩͱ͓ۚ΋͔͔Δ.


    • Air
    fl
    owʢCloud Composerʣ͸GKEΫϥελ͕ඞཁͰίετ໘Ͱͷෆ҆.


    • Cloud Functions + Cloud Scheduler + Pub/SubͰαʔόϨεԽ͕Ұ൪ཁٻ࢓༷ʹ߹͏ͱ൑அ


    • ֤ॲཧΛϚΠΫϩαʔϏεͱ࣮ͯ͠૷Մೳʢೋ൪໨ͷཧ༝ʣ.


    • ಠཱͨ͠ॲཧ୯ҐͰCloud Functionsͷؔ਺Λ࡞Δ͜ͱͰૄ݁߹ͳϚΠΫϩαʔϏεʹͳΔ.


    • ֤ؔ਺ͰϓϩάϥϛϯάݴޠɾFrameworkͷมߋ͕ग़དྷΔ, ςετͷ͠΍͢͞ͱ͍͏ϝϦοτ.

    View Slide

  50. • σʔλநग़ʢExtractʣ, ม׵ʢTransformʣ, ૹग़ʢLoadʣͦΕͧΕͰ෼͚ͯ࡞ΔͱεοΩϦ͠·͢.

    ※ࠓճ͸࢓্༷TransformͱLoad͸Ұॹʹ͍ͯ͠·͢ʢͦͷํ͕εοΩϦͨͨ͠Ίʣ
    • ॲཧͷτϦΨʔ͸Cloud Scheduler౳ͷαʔϏεΛCron୅ΘΓʹ͢Δͱ࣮૷͕͍͍ײ͡ʹলུͰ͖·͢.
    • Cloud Functionsʢ&ଞࣾΫϥ΢υؚΊͨFaaSʣ͸ϦΫΤετ͋ͨΓͷॲཧ࣌ؒɾϦιʔε੍ݶ͕͋ΔͷͰ஫ҙ
    • Cloud Functionsʢୈೋੈ୅ʣ͸ϦΫΤετ͋ͨΓͷ࠷େॲཧ࣌ؒ͸60෼ʢHTTPܦ༝ͷ৔߹ʣ, CPU/ϝϞϦ΋੍ݶ͋Γ·͢.
    • AWS Lambda౳ͷαʔϏε΋ͦΕͧΕ੍໿͕͋ΔͷͰ͝஫ҙΛ, େ͖͍σʔλ͸Dataflow౳, ઐ༻αʔϏεΛ࢖͏͜ͱΛݕ౼.

    ※FaaS: Function as a Serviceʢؔ਺ϕʔεͷΫϥ΢υαʔϏε, Cloud Functions, AWS LambdaͳͲʣ
    Cloud FunctionsΛ࢖ͬͨόονॲཧͷ࣮૷

    View Slide

  51. BigQueryͱͷ෇͖߹͍ํ
    • αΫοͱ࢖͑ΔεέʔϧՄೳͳDWH͕ཉ͔ͬͨ͠ΒBigQuery͍͍ͧ


    • ʢࠓճͷج൫Ͱ͸ͳ͘ʣݱ࣮ͷۀ຿Ͱ΋εέʔϧՄೳͳॴʹԿ౓͔ٹΘΕ·ͨ͠.


    • ݸਓར༻Ͱ࢖͍׳Ε͓ͯ͘ͱͳ͓ྑ͍Ͱ͢.


    • ݄౰ͨΓ, 10GBͷσʔλอ؅, 1TBͷΫΤϦ࣮ߦ͸ແྉ.


    • ݸਓͰ΍ͬͯͯ͜ͷྔ͸ͦ͏ͦ͏ߦ͔ͳ͍ϋζͳͷͰ࢖͏ͱ͍͍͔΋.


    • Ұ౓σʔλΛೖΕ͓ͯ͘ͱ, ΞϓϦ͔Β࢖͏ɾΞυϗοΫ෼ੳ྆ํศརͰ͢.

    ࢲ͸΋͏BigQuery͔ΒಀΕΔ͜ͱ͸ग़དྷ·ͤΜʢਅإʣ.

    View Slide

  52. ٕज़తͳ࿩ͰർΕ͖ͯ·ͤΜ͔?


    ʮਪ͠ͷ໺ٿબखʯͷ࿩Λ͠·͢Ͷ⽁

    View Slide

  53. WBCͰઈରݟಀͤͳ͍


    ʮ֤ࠃͷ௒ਓͨͪʯ
    • ਎ମೳྗ͓Խ͚ͰΩϨοΩϨ


    • ύϫʔ, ڧݞ, ޷कͦͯ͠٭͕ചΓ


    • ౤͛ͯ͸߽଎ٿorΤά͍มԽٿ


    • ͦ͏͍͏௒ਓͨͪΛ͝঺հ


    • ݱ໾࣌୅ͷ৽ঙ߶ࢤͬΆ͍

    Ӊ஦ਓબख΋͍Δ͔΋🤘

    View Slide

  54. WBCʹग़৔͢Δʢϋζʣ, ֤ࠃ୅දͷ௒ਓϚϯ
    • ถࠃ୅දʮϝδϟʔφϯόʔϫϯͷकඋྗΛތΔαʔυʯ


    • υϛχΧ୅දʮະདྷͷΠνϩʔ or ৽ঙ??࿘ອ͋;ΕΔηϯλʔʯ


    • ࣆJAPANʮϝδϟʔϦʔά༗๬ג֎໺ख͕ຬΛ࣋ͯ͠ͷ͝ొ৔ʯ

    View Slide

  55. ϊʔϥϯɾΞϨφυ


    ʢถࠃ୅දʣ
    • 10೥࿈ଓΰʔϧυάϥϒ৆ͱ͍͏

    ϝδϟʔ۶ࢦͷڧଧͷࡾྥख


    • ଧऀͱͯ͠͸ۃ୺ͳҾͬுΓϚϯ

    ӈଧͪͰ௕ଧ͸΄΅ࠨํ޲


    • ถࠃ୅දओཁϝϯόʔͷҰਓ

    ࡾྥकඋͱϑϧεΠϯάʹ஫໨

    View Slide

  56. ϑϦΦɾϩυϦήε


    ʢυϛχΧ୅දʣ
    • λϨϯτ܉ஂυϛχΧ஫໨ͷएखελʔ.

    2022೥MLB৽ਓԦ, All MLB 2ndνʔϜೖΓ.


    • ࡾৼ͔௕ଧͱ͍͏ۃ୺ͳଧܸελΠϧ,

    ҙ֎ͱ޿֯ʹଧͯΔଧܸηϯε,

    Πϝʔδతʹ͸৽ঙ߶ࢤͬΆ͍.


    • ॴଐ͢ΔϚϦφʔζͰ͸Πνϩʔࢯ͕

    ࿅शύʔτφʔΛ͍ͯ͠Δ͜ͱͰ΋࿩୊.

    View Slide

  57. ϥʔζɾψʔτόʔ


    ʢࣆJAPANʣ
    • ࣆJAPANॳͷΞϝϦΧग़਎બख


    • ଧ཰͸௿͍΋ͷͷग़ྥ཰ͱ௕ଧ཰

    ͦͯ͠ଧٿ଎౓͕଎͘, 3ϙδγϣϯ

    ͦͭͳ͘कΕΔ֎໺कඋ΋˕


    • Ωϟϥཱ͍͍ͪͯͯ͠Ϡπͱ͍͏ᷚ,

    ࣆδϟύϯͷηϯλʔͱͯ͠ظ଴ʂ

    View Slide

  58. ࣆJAPANͱ͍͑͹, ͜ͷೋਓ΋


    ݟಀͤͳ͍௒ਓ


    ώϯτ: ݩ೔ϋϜͷഎ൪߸11

    View Slide

  59. μϧϏογϡ༗͜ͱ, ౤ٿεϖγϟϦετͳ@faridyu͞Μ


    ୣࡾৼͷଟ͞ͱΧοτϘʔϧ୔ࢁ…WBCͰͲ͏͍͏ϓϨʔ͢Δ͔ظ଴͍ͨ͠.

    View Slide

  60. େ୩ᠳฏʢࢦ໊ଧऀʣͷओཁͳଧٿσʔλ


    ҾͬுΓϚϯͰ͸ͳͯ͘޿֯ʹ଎͍ଧٿଧͬͯͯੌ͍ʢ͜ͳΈʣ

    View Slide

  61. ݁ͼ

    View Slide

  62. ʲ࠶ܝʳຊ೔ͷελʔςΟϯάϝϯόʔ
    • Ϋϥ΢υͱσʔλΤϯδχΞϦϯάͰ࣮ݱ͢ΔԶͷDX


    • αʔόϨεͳΞʔΩςΫνϟͰ࣮ݱ͢Δʮ͍͍ײ͡ͳσʔλج൫ʯ


    • ʮਪ͠ͷ໺ٿબखʯͷʮਪ࣌͠ʯ͸Ͳ͜ͳͷ͔ΛݟۃΊΔ⽁

    View Slide

  63. ࠓ೔ͷ࿩Λཁ໿͢Δͱ…
    • ϑϧϚωʔδυ͔ͭαʔόϨεͳΫϥ΢υ͚ͩͰσʔλج൫͸࡞ΕΔ.


    • Ϋϥ΢υαʔϏεͱϓϩάϥϛϯάݴޠ͸༻్ʹ߹Θͤͯબ΅͏.


    • ࣆJAPANʹ͸ڧྗͳϥΠόϧ͕ͨ͘͞Μ͍Δʢ͜ͳΈʣ.

    View Slide

  64. ʮԶͷDXϓϩδΣΫτʯര஀ޙͷޮՌ.
    • ໺ٿσʔλαΠΤϯςΟετɾϓϩμΫτΦʔφʔͱ͍͏ࢹ఺Ͱϝονϟ࢖͑ͦ͏.


    • ೥ؒ500MB, 91߲໨ͱ͍͏టष͍σʔλΛ࢖͍΍͘͢͢Δ͚͍͍ͩϞϊΛ࡞Εͨʢࣄ࣮ʣ.


    • σʔλΛݟֶͯΜͩΓ, ϒϩάɾొஃͷωλʹ͢Δ͜ͱͰ։ൃ෼ͷ౤ࢿճऩ͕Ͱ͖ͦ͏.


    • ݸਓͱͯ͠ͷٕज़ݕূ͕ϝονϟḿΔ༧ײʢ&ࣗݾຬ଍ʣ


    • Google Cloudͷࢼ͍ͨ͠ػೳɾϞϊΛ࣮ݧ͢Δsandboxͱͯ͠ػೳͦ͠͏, AIܥαʔϏεͱ͔.


    • ΞʔΩςΫνϟΛͦͷ··ʹAWS΍AzureͰࢼ͢ͷ΋ΞϦ, ϚϧνΫϥ΢υ΋ΠέΔΜ͡ΌͶ?


    • ࢓ࣄʹඞཁͳεΩϧɾ஌ࣝΛझຯʢࣗݾຬ଍ʣ͔ΒखʹೖΕΔج൫ͱͯ͠΋࠷ߴʹྑ͍.

    View Slide

  65. ͓࢓ࣄʢۀ຿ʣͰࢀߟʹ͠Α͏ͱࢥͬͨํ΁
    • ࠓճ঺հͨ͠΍Γํɾߏ੒͸ઈରతͳճ౴ɾϕετϓϥΫςΟεͰ͸ͳ͍Ͱ͢.

    ྫ͑͹αʔόϨεɾΞʔΩςΫνϟʹ͢΂͖/͢΂͖͡Όͳ͍ঢ়گ͸࣮֬ʹଘࡏ͠·͢.


    • ͜ͷ࿩͸ࢲʢshinyorkeʣ͕΍Γ͍ͨࣄ, ͍͍ͱࢥͬͯΔࣄʢ&৮Γ͍ͨϞϊʣΛ

    ٧ΊࠐΜͰ࡞ͬͨ, ࣗ෼͕΍Γ͍ͨࣄͷूେ੒Ͱ, ͋͘·Ͱղͷग़͠ํͷҰͭͰ͢.


    • αʔόϨεʹͩ͜ΘΒͳ͔ͬͨΒʮGKEΫϥελཱͯͯͦ͜Ͱಈ͔͢ʯͰऴΘΔ.

    ͜Ε΋ཱ೿ͳղ౴ͩͬͨΓ͠·͢, ϝϦσϝ࣍ୈͰ.


    • Infrastructure as CodeͰ͖ͯͳ͍, ςετίʔυ଍Γͳ͍etc…࣮͸՝୊ࢁੵΈ.


    • ʢίϯςΩετͷཧղ͕த్൒୺ͳ··ʣͦͷ··ਅࣅ͢Δͱരࢮ͠·͢.

    ·ͣ͸खΛಈ͔͠, ֶशͨ͠Γಈ͔ͨ͠Γ͍͍ͯ͠΋ͷΛݟ͚ͭΔࢀߟʹͲ͏ͧʂ

    View Slide

  66. ʮਪ͠͸ਪͤΔ࣌ʹਪͤʯͱ͍͏ԶͷDXଓฤ.
    • ϝδϟʔϦʔάɾϓϩ໺ٿͷ੒੷༧ଌΤϯδϯ΋࡞ͬͯΔͷͰ,

    ͦΕΛࠓճͷج൫ʹ࣮૷͍ͨ͠, ૬ੑྑ͍͸ͣͳͷͰ.


    • ࠓޙͷల๬ͱͯ͠, σʔλͷϥΠηϯε౳ͷ՝୊Λղܾ͠,

    ͍͍ײ͡ͳ໺ٿσʔλαΠτΛҰൠެ։͢Δʢઈରʹ΍Γ͍ͨʣ.


    • ࠓޙ΋ٕज़Λ৭ʑࢼ͠, ໺ٿͷ໘നָ͍͠ΈํΛൃ৴ͭͭ͠,

    ʮ໺ٿͱٕज़ͷਪ͠ΛϦίϝϯυʯ͢ΔԶͷDXΛ΍͍ͬͯ͘΍Ͱ.

    View Slide

  67. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠⽁


    @shinyorke

    View Slide