Upgrade to Pro — share decks privately, control downloads, hide ads and more …

実践的データ基盤への処方箋_2-9_2-12

Tomoya Koike
February 03, 2022

 実践的データ基盤への処方箋_2-9_2-12

Tomoya Koike

February 03, 2022
Tweet

More Decks by Tomoya Koike

Other Decks in Programming

Transcript

  1. ࣮ફత
    デ
    ʔλج൫΁ͷॲํᝦ


    ྠಡձ 2-9 ~ 2-12
    খ஑ஐ࠸

    View full-size slide

  2. 2−9 ϩάऩू͸ΤʔδΣϯτͷΩϟύγςΟʹ஫ҙ

    View full-size slide

  3. ϩάͱ͸
    3
    Α͘෼ੳ͞ΕΔϩά
    1. WebαʔόͷΞΫηεϩά


    2. ΞϓϦέʔγϣϯͷϩά
    WebαʔόͷΞΫηεϩάͷྫ
    • ΞΫηεͨ࣌ؒ͠ɺURL


    • ΞΫηεݩͷIPΞυϨε


    • ΞΫηεʹ༻͍ͨ୺຤৘ใ


    • Web App͕ઃఆͨ͠ɺϢʔβࣝผ৘ใ

    View full-size slide

  4. ϩά͸ϩάऩूΤʔδΣϯτͰऩू͢Δ
    4
    ϩάऩूΤʔδΣϯτ
    • όοϑΝʹΑΓϩάऩूϚωʔδϟͷෛՙΛҰఆʹͰ͖Δ


    • όοϑΝ͕ᷓΕͳ͍Α͏ʹαΠζΛ֬อ͢Δ͜ͱʹ஫ҙ͕ඞཁ

    View full-size slide

  5. ϩάऩू͕Ͱ͖Δ੡඼
    5
    ੡඼໊ ఏڙํ๏
    ఏڙɾαϙʔτ


    ͍ͯ͠Δձࣾ
    fluentd, fluent-bit OSS Treasure Data
    Logstash OSS Elastic
    CloudWatch Cloud AWS
    Cloud Logging Agent Cloud GCP

    View full-size slide

  6. 2−10 ୺຤σʔλͷऩू͸೉қ౓͕ߴ͍ͨΊ
    Ͱ͖Δ͚ͩ੡඼Λར༻͠ແཧͳΒࣗ࡞͢Δ

    View full-size slide

  7. ୺຤σʔλ͸େྔ͕ͩ༗༻
    7
    ୅දతͳ୺຤σʔλ
    ϒϥ΢βΠϕϯτ
    εϚϗΞϓϦΠϕϯτ
    IoTσόΠεσʔλ
    ը໘ͷεΫϩʔϧ΍Ϛ΢εͷي੻౳ɺϢʔβͷϒϥ΢β্Ͱͷૢ࡞σʔλ
    εϚϗΞϓϦ্ͰͷϢʔβͷૢ࡞σʔλ
    ंࡌηϯαʔͷσʔλ΍ɺ؀ڥηϯαɺ΢ΤΞϥϒϧσόΠε౳

    View full-size slide

  8. ϒϥ΢βΠϕϯτ΍εϚϗΞϓϦΠϕϯτ͸σʔλऩू੡඼Λར༻
    8
    ੡඼
    ཉ͍͠σʔλΛऩूͰ͖Δ੡඼͕ͳ͍͔Λ୳͢
    Ϣʔβͷ୺຤্ͷߦಈ෼ੳπʔϧ͸ଟ͘ଘࡏ͢Δ
    ϒϥ΢βΠϕϯτ
    εϚϗΞϓϦΠϕϯτ
    Adobe Analytics΍Google Analytics౳
    Google Analytics For Firebase౳

    View full-size slide

  9. ࣗ࡞͢Δ৔߹͸෼ࢄϝοηʔδΩϡʔΛ࢖͏
    9
    ϩάऩूπʔϧΛࣗ࡞͢Δ৔໘
    • IoTσόΠεͷσʔλͰɺΫϥ΢υαʔϏεʹ͍͍΋ͷ͕ͳ͍ͱ͖


    • ΞΫηεղੳπʔϧͰ͸े෼ͳσʔλ͕ಘΒΕͳ͍৔߹


    • ϩά಺༰ΛϦΞϧλΠϜʹ׆༻͢Δඞཁ͕͋Δͱ͖I
    ෼ࢄϝοηʔδΩϡʔʹϩάΛૹ৴͢Δ

    View full-size slide

  10. ࣗ࡞͢Δ৔߹͸෼ࢄϝοηʔδΩϡʔΛ࢖͏
    10
    ෼ࢄϝοηʔδΩϡʔ
    Ωϡʔͱ͸ɺઌೖΕઌग़͠ͷσʔλߏ଄ͷ͜ͱɻ


    ΩϡʔΠϯά͢Δϓϩσϡʔαͱɺpop͢ΔίϯγϡʔϚͷ2ͭͷ໾ׂ͕͋Δɻ

    View full-size slide

  11. ෼ࢄϝοηʔδΩϡʔͷ஫ҙ͢΂͖ಛ௃ͱӡ༻ͷίπ
    11
    ෼ࢄϝοηʔδΩϡʔͷ஫ҙ͢΂͖ಛ௃
    1. ॱংੑอূͷ༗ແ…ॱং͕ඞཁͳ৔߹͸λΠϜελϯϓΛೖΕͯฒͼସ͑Δ౳͕ඞཁ


    2. ϝοηʔδͷॏෳ༗ແ…ॲཧΛႈ౳ʹ͢Δ͔ɺॲཧ༗ແͷ൑அϩδοΫΛೖΕΔ


    3. ՄࢹੑλΠϜΞ΢τ…ॲཧ࣌ؒΑΓ୹͍ͱɺॲཧ͕2ճҎ্૸Δ
    ӡ༻ͷίπ
    • ίϯγϡʔϚ͕ॲཧʹࣦഊ͠ଓ͚ͨσουϨλʔΛઐ༻ͷΩϡʔʹೖΕΔ


    • ϓϩσϡʔαʔ͕ੜ੒ྔΛ཈੍͢ΔόοΫϓϨογϟʔͱ͍͏࢓૊ΈΛೖΕΔ

    View full-size slide

  12. ۩ମతͳγεςϜͷ࡞Γํ
    12
    ෼ࢄϝοηʔδΩϡʔ

    View full-size slide

  13. 2−11 ETL੡඼ΛબͿϙΠϯτ͸ར༻͢Δ


    ίωΫλͷػೳੑͱσόοάͷ͠΍͢͞

    View full-size slide

  14. ETL੡඼ͱ͸
    14
    ఏڙܗଶͷҧ͍
    ETL…Extract Transform LoadͷུͰɺσʔλͷநग़ɺՃ޻ɺϩʔυΛҙຯ͢Δɻ
    ෳࡶͳՃ޻͕Ͱ͖Δ͔Ͳ͏͔
    ҟͳΔσʔλιʔε͔ΒͷσʔλΛՃ޻ɺ݁߹ͯ͠ϩʔυ͢Δ౳ɻ


    Apache Nifi, DataSpider, Glue, Cloud Data Fusion
    OSSɺ༗ঈ੡඼ɺΫϥ΢υͷ3छྨɻ


    OSS…embulk΍fluentd, Apache Sqoop, ༗ঈ੡඼ͱͯ͠DataSpider, ASTERIAͳͲɻ


    Ϋϥ΢υͰ͸AWSͷGlue΍GCPͷCloud Data Fusion, ྆αʔϏεͷDMS,


    embulkͷϚωʔδυαʔϏεͰ͋ΔtroccoͳͲɻ ※DMS…Database Migration Service

    View full-size slide

  15. ࢖͏ίωΫλͷػೳΛॏࢹ͢Δ
    15
    MySQLίωΫλͰ͋Ε͹ɺWHERE۟΍ࠩ෼ͷΈऩू౳͕Ͱ͖Δ͔ɻ


    ϏοάσʔλͰ͋Ε͹ɺ෼ࢄॲཧͰ͖Δ͔ʹ஫ҙ͢Δɻ

    View full-size slide

  16. ιʔείʔυϨϕϧͰσόοά͠΍͍͢΋ͷΛར༻͢Δ
    16
    όά͕ى͖ͨͱ͖ɺσʔλιʔεɾ֨ೲઌɾऩू͠Α͏ͱ͢Δσʔλͷ૊Έ߹
    ΘͤʹΑͬͯ͸࠶ݱྫ͕ͳ͍͜ͱ΋ɻ
    ίωΫλͷιʔείʔυΛݟʹߦ͚Δͷ͕େࣄɻ


    ETL੡඼ͷαϙʔτʹௐࠪͯ͠΋Β͏ͱ͖͸ɺσʔλج൫ʹೖͬͯ΋Β͏Α͏४උɻ
    σʔλʹґଘͯ͠ى͖Δόά
    ૝ఆ͍ͯ͠ͳ͍จࣈίʔυ΍੍ޚจࣈɺվߦίʔυ


    null஋Λظ଴͢Δͱ͜Ζʹۭจࣈྻ

    View full-size slide

  17. ΤϯδχΞ͕͍ͳ͚Ε͹ϓϩάϥϛϯάϨεͷETL੡඼΋બ୒ࢶͷ1ͭ
    17
    ઐ༻ͷը໘্Ͱσʔλιʔε΍֨
    ೲઌͷΞΠίϯΛͭͳ͛ͯETLॲ
    ཧΛఆٛɺσϓϩΠͰ͖Δɻ
    Apache Nifi, Talend, DataSpider,
    ASTERIA, Glue, Cloud Data Fusion
    ͳͲɻ

    View full-size slide

  18. 2−12 σʔλϨΠΫͰ͸ऩूͨ͠σʔλ


    Λͳ͘͞ͳ͍Α͏ʹ͢Δ

    View full-size slide

  19. ऩूͨ͠σʔλΛݪଇͦͷ··஝ੵ͢Δ
    19
    σʔλϨΠΫʹ͸ऩूͨ͠σʔλΛՃ޻ͤͣʹ֨ೲ͢Δ
    σʔλϨΠΫʹ͸৑௕ԽͰ͖༰ྔ͕֦ுͰ͖Δ੡඼ΛબͿ
    ऩूͨ͠σʔλΛͳ͘͞ͳ͍ͨΊʹ৑௕Խ͢Δ͜ͱɺ


    σʔλ༰ྔΛ૿΍ͤΔΑ͏ʹ͓ͯ͘͜͠ͱ͕ॏཁɻ
    ϑΝΠϧ΍JSONܗࣜɺςʔϒϧߏ଄ͳͲΛͦͷ··อଘɻ
    Ճ޻ʹࣦഊͯ͠σʔλଛࣦ͢Δ͜ͱΛ๷͙ͨΊɻ


    ػີ৘ใ΍ݸਓ৘ใ͸ಗ໊ԽΛߦͬͯ஝ੵ͢Δɻ

    View full-size slide

  20. ϑΝΠϧ͸ΦϒδΣΫτετϨʔδʹ஝ੵ͢Δ
    20
    ෳ਺ͷσʔληϯλʔͰෳ਺σόΠεʹ
    ৑௕Խͯ͠อଘ͢Δ͜ͱͰɺ


    ΠϨϒϯφΠϯͷݎ࿚ੑͱ
    99.99%ͷՄ༻ੑΛ࣮ݱ͍ͯ͠Δɻ
    σʔλΛʮΦϒδΣΫτʯͱ͍͏୯ҐͰѻ͏هԱ૷ஔɻ


    Ϋϥ΢υαʔϏεͱͯ͠S3΍Cloud StorageͳͲ͕͋Δɻ
    ΦϒδΣΫτετϨʔδ
    ΦϯϓϨͷ৔߹͸෼ࢄετϨʔδΛར༻ɻ


    OSSͱͯ͠HDFSͳͲɻ

    View full-size slide

  21. CSV΍JSONσʔλ͸σʔλϕʔεʹೖΕͯ΋OK
    21
    CSV΍JSONσʔλΛDWH༻෼ੳDBʹೖΕΔ
    ෼ੳ༻DBͷதͰɺੜͷσʔλ
    Λ֨ೲ͢ΔσʔλϨΠΫ૚ͱ


    Ճ޻͞ΕͨσʔλΛ࣋ͭDWH
    ૚ʹ෼͚Δ
    JSONʹ͍ͭͯ͸ɺจࣈྻܕ·ͨ͸JSONܕͱͯ֨͠ೲ͢Δ

    View full-size slide

  22. σʔλ͕ΦϯϓϨϛεʹ͋ͬͯ΋σʔλϨΠΫ͸Ϋϥ΢υʹ͢Δ
    22
    3ͭͷཧ༝
    ैྔ՝ۚͰར༻Ͱ͖ΔͨΊ
    ଱ٱੑ͕ߴ͍ͨΊ
    ӡ༻ਓ݅අ͕͍҆
    ج൫ߏங࣌ʹσʔλྔΛਖ਼֬ʹݟੵ΋Δͷ͸ࠔ೉ͳͨΊ
    AWS S3ͷ଱ٱੑ͸ΠϨϒϯφΠϯͰ͋ΓɺΦϯϓϨͰ͸ఢΘͳ͍
    αʔόͷߏ੒Λؾʹ͠ͳͯ͘Α͍ͷͰɺඞཁͳٕज़ྗ͕Լ͕Δ
    σϝϦοτͱͯ͠ɺࡉ҆͘ఆ͠ͳ͍ωοτճઢ͔ɺߴ͍ઐ༻ઢ͔Λ༻͍Δඞཁ͕͋Δ

    View full-size slide