$30 off During Our Annual Pro Sale. View Details »

etl-in-the-media-company

 etl-in-the-media-company

ETLの実際
- 分析基盤刷新PoC
- ETL実例紹介
- ETLでぶつかった壁

Masatoshi Shimada

September 19, 2017
Tweet

More Decks by Masatoshi Shimada

Other Decks in Technology

Transcript

  1. ಈըܥϝσΟΞاۀͰ
    ߦΘΕ͍ͯΔ&5-ͷ࣮ࡍ
    ౡాխ೥!TNENUT

    View Slide

  2. ࣗݾ঺հ
    w ౡాխ೥!TNENUT
    w ιϑτ΢ΣΞΤϯδχΞ
    w 4DBMBc1ZUIPOc+BWBc(PMBOHc%%%

    w גࣜձࣾΤϒϦʔॴଐ
    w %&-*4),*5$)&/σʔλ෼ੳܥશൠ୲౰

    View Slide

  3. ΞϓϦ঺հ

    View Slide

  4. ໨࣍
    w ෼ੳج൫࡮৽1P$
    w &5-࣮ྫ঺հ
    w &5-ͰͿ͔ͭͬͨน

    View Slide

  5. ෼ੳج൫࡮৽1P$

    View Slide

  6. 4JOHMFTPVSDFPGUSVUI
    The state of your whole application is stored in an object tree within a single store. by redux.
    w ΞϓϦέʔγϣϯͷঢ়ଶ͸Ұݩ؅ཧ͞ΕΔ΂͖
    Ͱ͋Δ
    w ϏοάσʔλͰ΋Ұͭͷ৔ॴʹू໿ͯ͠ɺ42-
    ͰݕࡧՄೳʹ͓ͯ͘͠΂͖Ͱ͋Δ
    w ݱ࣮తʹ͸اۀ಺ͷσʔλ͸༷ʑͳܗͰଘࡏ͠
    ͓ͯΓ݁߹͢Δ͜ͱ͸೉͍͠

    View Slide

  7. γεςϜߏ੒ਤʢ౰ॳʣ
    "1*αʔό
    ޿ࠂޮՌଌఆ
    Πϕϯτܭଌ
    "UIFOB
    #JH2VFSZ
    ݁߹͕ग़དྷͳ͍ͨΊɺ
    4405͕ຬͨͤͳ͍
    ՄࢹԽ
    %8)

    View Slide

  8. ౰ॳͷ෼ੳج൫ʹ͓͚Δ՝୊
    w εΩϟϯྔʢΫΤϦ՝ۚʣ͕೔ʑ૿େ͍ͯͨ͠
    w ΫΤϦɺ5#ͷεΩϟϯ͕ଟ਺
    w #2ͷσʔλߏ଄͕ෳࡶͰ42-͕೉ಡঢ়ଶ
    w Ϩίʔυܕͷ෼ղ͕ओͳݪҼ
    w "UIFOBͱ#JH2VFSZͷ+0*/͕ग़དྷͳ͍
    w SFEBTIͳΒ͹݁߹Մೳ
    w ͔͠͠σʔλ࿈܎Λॏࢹͨͨ͠Ίߟྀ֎ͱͨ͠
    w ޿ࠂޮՌଌఆͷϩά͕࿈ܞ͞Ε͍ͯͳ͍
    "UIFOB
    #JH2VFSZ

    View Slide

  9. w 4405͕࣮ݱ͞Ε͍ͯΔ͜ͱ
    w εΩϟϯྔΛؾʹ͢Δ͜ͱແ͍෼ੳج൫Ͱ͋Δ͜ͱ
    w ෼ੳ༻42-͸ɺͳΔ΂͘؆қͳ42-ͱ͍ͨ͠
    w ಛघͳ8JOEPXؔ਺΍+0*/ͷআڈΛ૝ఆ
    w ඇΤϯδχΞ޲͚ʹ෼ੳͷෑډΛԼ͍͛ͨ
    w Ϩίʔυͷߏ଄͸ՄมͰεΩʔϚϨεʹ͍ͨ͠
    w શͯϑϧϚωʔδυͰ࣮ݱ͞ΕΔ͜ͱ
    ৽෼ੳج൫ʹ޲͚ͨ1P$

    View Slide

  10. ৽෼ੳج൫ʹ޲͚ͨ1P$
    w Ϧιʔε՝ۚͰ͋Δ͜ͱ
    w ϑϧϚωʔδυɾδϣϒεέδϡʔϥ͕ଘࡏ͢Δ͜ͱ
    w ϑϧϚωʔδυɾόϧΫΠϯϙʔτ͕ଘࡏ͢Δ͜ͱ
    w εΩʔϚϨε%#ͰεΩʔϚࣗಈߋ৽͕ଘࡏ͢Δ͜ͱ
    w 3&45Ͱσʔλ౤ೖՄೳͳίʔϧόοΫ༻ͷΤϯυϙ
    Πϯτ͕ଘࡏ͢Δ͜ͱ
    w σʔλ࿈ܞʢ*NQPSU&YQPSU
    ͕༰қͰ͋Δ͜ͱ

    View Slide

  11. "1*αʔό
    ޿ࠂޮՌଌఆ
    Πϕϯτܭଌ
    %BUB-BLF
    %8)
    4405 1P$
    ୡ੒
    σʔλ͕Ұݩ؅ཧ͞ΕΔͨΊ
    ͲͷΑ͏ͳσʔλͰ΋݁߹Մೳ
    #JH2VFSZ
    ՄࢹԽ
    3&45"1*
    $PODMVTJPO

    View Slide

  12. &5-࣮ྫ঺հ

    View Slide

  13. &5-࣮ྫ঺հ
    "1*αʔό
    ޿ࠂޮՌଌఆ
    Πϕϯτܭଌ
    %BUB-BLF
    %8)
    #JH2VFSZ
    ՄࢹԽ
    3&45"1*
    ͜ͷ෦෼ͷ࿩

    View Slide

  14. 'JSFCBTF"OBMZUJDT
    w #BB4πʔϧ
    w ओʹ"#ςετɺϓογϡ௨஌ɺΠϕϯτϩ
    άܭଌͰར༻͍ͯ͠Δ
    w #JH2VFSZʹΠϕϯτܭଌϩά͕࿈ܞՄೳ
    w #JH2VFSZͰੜϩάΛ෼ੳʹճ͍͕ͯͨ͠ɺɹ
    εΩϟϯྔ૿େ໰୊ͱ42-೉ಡԽ໰୊͕͋ͬͨ

    View Slide

  15. 'JSFCBTF
    4%,
    #JH2VFSZ 5SFBTVSF
    %BUB
    ($4 4
    &.3
    5SFBTVSF8PSLqPX
    'JSFCBTF
    "OBMZUJDT
    &5-"SDIJUFDUVSF
    .3FDPSE%BZ

    View Slide

  16. 'JSFCBTF
    4%,
    &5-"SDIJUFDUVSF
    'JSFCBTF
    "OBMZUJDT
    #JH2VFSZ 5SFBTVSF
    %BUB
    ($4 4
    &.3
    ϑϧϚωʔδυ
    .3FDPSE%BZ
    5SFBTVSF8PSLqPX

    View Slide

  17. 5SFBTVSF8PSLqPX
    w %JHEBHͷϑϧϚʔωδυδϣϒεέδϡʔϥ
    w 3VCZ1ZUIPO4IFMM͕࢖͑ͳ͍
    w Πϯϑϥૢ࡞͸"84-BNCEBΛར༻͢Δ
    w "84΍($1͸$-*Λ-BNCEBͰىಈ͢Δ
    w &.3ΦϖϨʔλΑΓ΋ॊೈʹ੍ޚ͔ͨͬͨ͠
    w ॲཧܥ͸ϑϧϚωʔδυͳαʔόϨεߏ੒

    View Slide

  18. %JHEBH8PSLqPX
    CR@FYUSBDU
    #JH2VFSZ ($4
    TUFQ
    IUUQ
    "1*(BUFXBZ -BNCEB &.3
    ($4 4 5%
    TUFQ
    45&1
    #JH2VFSZ͔Β($4ʹμϯϓ
    45&1
    ($4͔Β&.3Ͱ5%ʹ࿈ܞ

    View Slide

  19. &NCVMLPO&.3
    &.3
    w .BQ3FEVDF&YFDVUPSΛར༻͢Ε͹ɺɹ
    )BEPPQΫϥελʔ্Ͱ෼ࢄ࣮ߦͰ͖Δ
    IUUQTHJTUHJUIVCDPNTNENUTDGEFCFCEEDBCCCE
    IUUQRJJUBDPNKPLFSJUFNTECBFCGBBF
    w Լهͷ̎εςοϓͰར༻ՄೳʢUIBOLT!KPLFS


    w MPH༻ͷ+"3Λ&.3֤ϊʔυʹࣄલʹϩʔυ
    w ϚελʔϊʔυͰ.BQ3FEVDF&YFDVUPSΛ
    ར༻ͯ͠&NCVMLΛىಈ͢Δ

    View Slide

  20. ($4 4
    4 5%
    4UFQ"730ʢ#JH2VFSZʣ+40/ʢ3FDPSE&WFOUʣ
    4UFQ+40/ʢ3FDPSE&WFOUʣ1MB[NB%# 5%

    "730ʢ3FDPSE.BOZ&WFOUʣΛɺ
    +40/ʢ3FDPSE&WFOUʣʹม׵͍ͯ͠Δ
    %BUB$POOFDUPSͰ+40/಺ͷ+40/ܕΛ෼ղ͢ΔͨΊɺ
    5%ΦϓγϣϯͷFYQMPEF@KTPO@DPMVNOTΛར༻͍ͯ͠Δ
    FYQMPEF@KTPO@DPMVNOT͸%BUB$POOFDUPSઐ༻Ͱ&.3͔Βར༻Ͱ͖ͳ͍
    &.3Ͱ࣮ߦͯ͠΋ͬͱߴ଎Խ͍ͨ͠ͷͰ044൛Ͱ͜ͷΦϓγϣϯΛެ։ͯ͠ཉ͍͠ʢཁ๬ʣ
    &NCVML࣮ߦܭը

    View Slide

  21. Ϩίʔυͷঢ়ଶભҠ
    $ %
    " #
    '
    $ %
    " #
    &
    '
    Ћ
    Ћ
    Ћ
    Ћ
    Ћ
    Ћ
    Ћ
    Ϩίʔυ

    ܕ"
    ຌྫ
    ෼ׂ
    &
    " #
    $ % &
    '
    Ћ
    Ћ
    Ћ
    " #
    $ % &
    '
    ൃࢄ
    ܕ#
    ܕ$
    4UFQ"730+40/ʢ3FDPSE&WFOUʣ
    4UFQ+40/ʢ3FDPSE&WFOUʣ1MB[NB%# 5%

    View Slide

  22. $PODMVTJPO
    w .BQ3FEVDF&YFDVUPSΛར༻ͯ͠&.3Ͱ෼ࢄ࣮
    ߦ͢Δ͜ͱͰେྔσʔλͷऔΓࠐΈΛՄೳͱͨ͠
    w ϨίʔυΠϕϯτʹ෼ׂ͢Δ͜ͱͰ෼ੳ༻42-
    ͷ೉қ౓ΛԼ͛ΔࣄΛ࣮ݱͨ͠
    w ྻࢦ޲%#Λੜ͔ͯ͠+40/Λൃࢄͤ͞Δ͜ͱͰɺε
    ΩʔϚ؅ཧίετͷ࡟ݮΛ࣮ݱͨ͠
    w ϑϧϚωʔδυͰ&5-Λ࣮ݱ͢Δ͜ͱͰ؅ཧίετ
    ࡟ݮΛ࣮ݱͨ͠

    View Slide

  23. &5-ͰͿ͔ͭͬͨน

    View Slide

  24. 'JSFCBTFऔΓࠐΈ༻
    &NCVMLϓϥάΠϯ͕ͳ͍

    View Slide

  25. &NCVMLϓϥάΠϯ͕ͳ͍
    'JSFCBTFऔΓࠐΈ༻
    ແ͔ͬͨΒ࡞Δ

    View Slide

  26. &NCVMLϓϥάΠϯ͕ͳ͍
    'JSFCBTFऔΓࠐΈ༻
    ແ͔ͬͨΒ࡞Δ
    ͦͯ͠ɺ૊Έ߹ΘͤΔ

    View Slide

  27. ੒Ռ෺Ұཡ
    w FNCVMLpMUFSLFZ@JO@SFEJT
    w FNCVMLpMUFSKTPO@LFZ@KPJOFS
    w FNCVMLGPSNBUUFSGBTU@KTPOM
    w FNCVMLQBSTFSpSFCBTF@BWSP
    w FNCVMLEFDPEFSSFNPWF@OPOTUBOEBSE@VUG@CZUFT
    w FNCVMLPVUQVUqVFOUEʢFYQFSJNFOUBMBTXBJUJOHGPS+BWB


    w FNCVMLPVUQVULFZ@UP@SFEJT
    ͱͷ֨ಆͷ຤ɺੜ੒͞ΕͨϓϥάΠϯ

    View Slide

  28. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    &NCVMLΛར༻ͨ͠
    େྔϨίʔυࠩ෼࿈ܞͷ঺հ

    View Slide

  29. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    w ຊ೔Ϩίʔυ͸#2ͷຊ೔ςʔϒϧ JOUSBEBZ
    ʹ࿈ܞ͞ΕΔ
    w ఆࠁʢே࣌ࠒʣʹຊ೔ςʔϒϧ͔Βࡢ೔ςʔϒϧ͕࡞੒͞
    Εͯຊ೔Ϩίʔυ͕ࡢ೔ςʔϒϧʹҠಈ͢Δ
    w ࠷େͰաڈ೔લ·ͰͷΠϕϯτϩά͕೔෇ςʔϒϧʹఆࠁ
    ʢே࣌ࠒʣͰ࿈ܞ͞ΕΔ
    w ࡢ೔͕࡞੒ɺ̎೔લɾ̏೔લ͕௥هͷ̏छྨ
    w ͲͷϨίʔυ͕௥ه͞Ε͔ͨ͸42-Ͱ͸෼͔Βͳ͍
    #JH2VFSZ

    View Slide

  30. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    ຊ೔
    ࡢ೔
    ೔લ
    ೔લ
    աڈϩά
    ੵΈ্͛
    ఆࠁʢே࣌ࠒͷڍಈʣ
    ࡞੒
    ௥ه
    ௥ه
    JOUSBEBZ
    #JH2VFSZ

    View Slide

  31. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    Ұࡢ೔
    ຊ೔࿈ܞ͍ͨ͠σʔλ
    ࡢ೔࿈ܞࡁΈ
    w ຊ೔࿈ܞ͍ͨ͠σʔλΛநग़͢Δʹ͸ɺશྔσʔλͱࡢ೔࿈
    ܞࡁΈσʔλͱ%J⒎ΛऔΔඞཁ͕͋Δ
    ɹʢ3FETIJGUʢ42-ʣͰ΋࣮ݱՄೳ͕ͩ&NCVMLͰ࣮ݱ͔ͨͬͨ͠ʣ
    શྔ
    ͷϨίʔυʹ͍ͭͯ෼ղ͢ΔͱԼهͷΑ͏ʹͳΔ
    ʹ

    View Slide

  32. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    ࿈ܞࡁΈσʔλ
    4UFQ FNCVMLPVUQVULFZ@UP@SFEJT
    4UFQ
    શྔσʔλʢ.%ʣ —
    ʹ ຊ೔࿈ܞ෼Ϩίʔυ
    FNCVMLpMUFSLFZ@JO@SFEJT
    #JH2VFSZ
    4UFQ FNCVMLQBSTFSpSFCBTF@BWSP
    ࿈ܞࡁΈσʔλʢ.%ʣ
    ࿈ܞࡁΈσʔλʢ.%ʣ
    શྔσʔλ

    View Slide

  33. &NCVMLϓϥάΠϯ૊Έ߹Θͤ঺հ
    w ૊Έ߹Θͤͯ࢖͏&NCVMLϓϥάΠϯ
    w େྔσʔλͷࠩ෼Λൈͨ͘Ίʹ͸ɺϩʔυࡁΈͷঢ়ଶ͕ඞཁͱ
    ͳΓ3FEJTͷ4FUΛར༻ͨ͠
    w BLLBΛར༻্ͯ͠ྲྀ͔ΒྲྀΕΔϨίʔυ 1BHF
    ΛόοϑΝϦ
    ϯάͯ͠ɺҰఆྔஷ·͔ͬͯΒϑϥογϡ͢Δ͜ͱͰɺ3FEJT
    ΁ͷ໰͍߹Θͤޮ཰޲্Λૂͬͨ
    FNCVMLPVUQVULFZ@UP@SFEJT
    FNCVMLpMUFSLFZ@JO@SFEJT

    View Slide

  34. $PODMVTJPO
    w &NCVMLϓϥάΠϯΛ૊Έ߹Θ࣮ͤͯߦܭըΛߟྀ͢
    Ε͹ɺଟछଟ༷ͳ&5-͕࣮ݱՄೳͱͳΔ
    w ྑ͍&NCVMLϓϥάΠϯ͕ແ͔ͬͨΒ13͢Δ͔ɹ
    ࡞ͬͯ044Ͱެ։͠Α͏
    ࠔͬͨΒ5XJUUFSͰͭͿ΍͘ͱ৭ʑͱڭ͑ͯ͘ΕΔʢUIBOLT!IJSPZTBUP

    w &NCVMLϓϥάΠϯ։ൃʹ4DBMB͸ڧྗͳϒΩʹͳΔ
    BLLBͱ͔DJSDFͱ͔DBUTͱ͔TDBMB[ͱ͔TIBQFMFTTͱ͔

    View Slide

  35. ձࣾ֓ཁ
    FOOD FAMILY BEAUTY NEWS
    ಈըΛ௨ͯ͡

    ੈքΛ΋ͬͱָ͘͠

    ΋ͬͱॆ࣮ͨ͠ຖ೔ʹ

    View Slide

  36. https://corp.every.tv/recruits
    We are hiring!
    Chief Dog Officer
    EVE

    View Slide

  37. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide