Upgrade to Pro — share decks privately, control downloads, hide ads and more …

セプテーニで分析基盤(Treasure Data)を導入した話

kimutyam
September 19, 2017

セプテーニで分析基盤(Treasure Data)を導入した話

新宿Geek Lounge#3 分析基盤Meetup 登壇資料

kimutyam

September 19, 2017
Tweet

More Decks by kimutyam

Other Decks in Technology

Transcript

  1. @৽॓Geek Lounge#3 ෼ੳج൫Meetup
    @kimutyam
    2017/09/19
    ηϓςʔχͰ෼ੳج൫
    (Treasure Data)Λಋೖͨ͠࿩

    View Slide

  2. ໊લ໦ଜজ޺
    5XJUUFS!LJNVUZBN
    ձࣾηϓςʔχɾΦϦδφϧ
    %%%ͱ*$0/*9Λ࢖ͬͨ෼ੳઃܭΛ޷Έͱ͍ͯ͠Δ4DBMBΤϯδχΞ
    ۀ຿ࢧԉͷΞʔΩςΫνϟΛߟҊ͢Δ͏ͪʹ෼ੳج൫ܥͷۀ຿ʹ୧Γண͖·ͨ͠
    ࣗݾ঺հ

    View Slide

  3. PYXISͷྺ࢙ͱ෼ੳج൫ಋೖ·Ͱͷഎܠ

    View Slide

  4. PYXISͱ͸
    ޿ࠂ഑৴
    ࠷దԽ
    ୅ཧళۀ
    ຿վળ
    σʔλ෼

    View Slide

  5. σʔλ෼ੳॳظ
    PYXISͷσʔλ ηϓςʔχͷ֤෦໳DB
    ഔମࣾͷσʔλ
    ސ٬ͷσʔλ
    ࣾһͷσʔλ
    ֎෦ϕϯμʔͷσʔλ
    ಛఆͷ໨తʹԊͬͯߏ଄Խ͞Εͨσʔλ͕෼ࢄ؅ཧ͞Εɺ࣍ར༻͞Ε͍ͯͨ

    View Slide

  6. σʔλͷ࣍ར༻͕ͮ͠Β͍
    σʔλ෼ੳॳظͰͷ໰୊఺
    ύϑΥʔϚϯε͕ѱ͍
    σʔλ͕෼ࢄ؅ཧ͞Ε͍ͯΔ
    &5-ॲཧ"11ͷཚཱ

    View Slide

  7. σʔλͷ࣍ར༻͕ͮ͠Β͍
    վળ΁
    ύϑΥʔϚϯε͕ѱ͍ &5-ॲཧ"11ͷཚཱ
    ඇߏ଄Խσʔλͷ஝ੵ
    ෼ࢄॲཧ "11ͷ؆ུԽπʔϧར༻
    σʔλͷू໿Խ
    σʔλ͕෼ࢄ؅ཧ͞Ε͍ͯΔ

    View Slide

  8. σʔλͷ࣍ར༻͕ͮ͠Β͍
    ιϦϡʔγϣϯ: σʔλϨΠΫ
    ύϑΥʔϚϯε͕ѱ͍ &5-ॲཧ"11ͷཚཱ
    ඇߏ଄Խσʔλͷ஝ੵ
    ෼ࢄॲཧ "11ͷ؆ུԽπʔϧར༻
    σʔλͷू໿Խ
    σʔλ͕෼ࢄ؅ཧ͞Ε͍ͯΔ

    View Slide

  9. σʔλͷ࣍ར༻͕ͮ͠Β͍
    ιϦϡʔγϣϯ: σʔλ΢ΣΞϋ΢ε
    ύϑΥʔϚϯε͕ѱ͍
    σʔλ͕෼ࢄ؅ཧ͞Ε͍ͯΔ
    &5-ॲཧ"11ͷཚཱ
    ඇߏ଄Խσʔλͷ஝ੵ
    ෼ࢄॲཧ "11ͷ؆ུԽπʔϧར༻
    σʔλͷू໿Խ

    View Slide

  10. σʔλͷ࣍ར༻͕ͮ͠Β͍
    ιϦϡʔγϣϯ: Treasure Data
    ύϑΥʔϚϯε͕ѱ͍
    σʔλ͕෼ࢄ؅ཧ͞Ε͍ͯΔ
    &5-ॲཧ"11ͷཚཱ
    ඇߏ଄Խσʔλͷ஝ੵ
    ෼ࢄॲཧ "11ͷ؆ུԽπʔϧར༻
    σʔλͷू໿Խ

    View Slide

  11. Treasure DataΛར༻ͨ͠ΞʔΩςΫνϟ

    View Slide

  12. ΞʔΩςΫνϟߏ੒
    1MB[NB%# %BUB5BOL

    View Slide

  13. S3Λ1࣍੥͚ʹ͢Δ
    ετϨʔδ͸ͳ͍ͷͰ4Λར༻
    ੜσʔλͷσʔλܗࣜ +40/$479.-ͳͲ
    ͷ··อଘ
    4ϑΝʔετʹ͢Δ͜ͱͰ"UIFOB3FETIJGU&.3ͳͲͷ
    αʔϏεͱͷ࿈ܞ͕ඞཁʹԠͯ͡Մೳʹ
    ੜσʔλอଘͷࡍʹσʔλεΩʔϚΛߟྀ͠ͳ͍

    View Slide

  14. ߏ଄Խσʔλ͸ڥք͚ͮΒΕͨίϯςΩετʹด͡Δ
    ޿ࠂ഑৴
    ࠷దԽ
    ୅ཧళۀ
    ຿վળ
    σʔλ෼

    ʮ୅ཧళۀ຿վળʯ΍ʮ޿ࠂ഑৴࠷దԽʯͷίϯς
    Ωετ಺ʹσʔλ͸1:9*4಺ʹݶఆ
    ʮσʔλ෼ੳʯͷίϯςΩετͰ͸࣍ར༻͢Δͨ
    ΊʹੜσʔλΛ4ʹอଘ

    View Slide

  15. S3ͷϨΪϡϨʔγϣϯ
    #VDLFU1:9*4ͷΞϓϦέʔγϣϯڥքͷ໋໊ʹ͢Δ
    1BUI\5%ͷσʔλϕʔε໊^\5%ͷςʔϒϧ໊^[email protected]
    ΞϓϦέʔγϣϯؒͷڝ߹Λճආ͢ΔͨΊ
    &NCVMLଆͰ[email protected]Ͱ೔࣌ΛಈతʹࢦఆͰ͖ΔΑ͏ʹ
    "UIFOBͷύʔςΟγϣϯͰ೔෇ࢦఆ͠΍͘͢ͳΔΑ͏ʹ

    View Slide

  16. S3 -> TD

    View Slide

  17. Treasure DataͷྖҬ
    1MB[NB%# %BUB5BOL
    5SFBTVSF%BUBͷϑϧϚωʔδͷαʔϏεΛར༻
    5SFBTVSF8PSLqPX %JHEBH

    %BUB$POOFDUPS &NCVML

    5SFBTVSF8PSLqPXΛར༻͢Δ͜ͱͰɺಠࣗʹ%JHEBH
    4FSWFSΛཱͯΔߏஙɾอकίετΛ࡟ݮ
    5SFBTVSF8PSLqPX͔Βར༻͠ɺඞཁʹԠͯ͡%JHEBH
    4FSWFSΛཱͯΔΑ͏ʹϓϩδΣΫτελʔτ

    View Slide

  18. Treasure DataͷྖҬ
    1MB[NB%# %BUB5BOL
    4͔Β[email protected]
    1MB[NB%#͔Β1SFTUPͰूܭͯ͠൚ԽσʔλΛ1MB[NB%#
    ʹอଘ
    %BUB5BOL σʔλϚʔτ
    ʹར༻༻్ʹԊͬͨಛԽ͞Εͨ
    σʔλΛ֨ೲ
    ˞%BUB5BOL͸5SFBTVSF%BUB͕ఏڙ͍ͯ͠Δ1PTUHSFTRM

    View Slide

  19. ࣗલͰDigdag ServerΛݕ౼ͨ͠ྫ
    1MB[NB%# %BUB5BOL
    42-͚ͩͰ͸ूܭग़དྷͳ͍৔߹
    %JHEBHͷQZUIPOSVCZTIFMMΦϖϨʔλʔΛར༻
    ͯ͠ɺಠࣗʹ࣮૷ 5SFBTVSF8PSLqPX͸ະରԠ

    %BUB$POOFDUPSͰ͸ཉ͍͠ूܭཻ౓͕ಘΒΕͳ͍৔߹
    &NCVMLϓϥάΠϯΛ૊ΈࠐΈPS։ൃ

    View Slide

  20. Treasure DataΛར༻ͨ͠ۀ຿ϑϩʔ
    σʔλ஝ੵ
    σʔλ෼ੳɾՁ஋ݕূ
    %JHEBHʹΑΔΦʔτϝʔγϣϯԽ
    ։ൃऀ͕5SFBTVSF%BUBʹσʔλ֨ೲ
    σʔλΞφϦετ ฐࣾͰ͸ඇΤϯδχΞ͕ଟ͍
    ͕5SFBTVSF%BUBͷ(6*Λར༻
    ։ൃऀ͕؅ཧ͢Δ
    5SFBTVSF%BUBΛհͯ͠։ൃऀͱσʔλΞφϦετͷۀ຿࿈ܞΛ͢Δ

    View Slide

  21. Treasure DataΛબ୒ͨ͠Α͔ͬͨ఺

    View Slide

  22. ϑϧϚωʔδυͳ෼ੳج൫ΛεϐʔσΟʔʹखʹೖΓɺ෼ੳۀ຿ʹूத͕Ͱ͖Δ
    "84΍($1Ͱ෼ੳج൫Λࣗલߏங͢Δ৔߹͸ɺ
    αʔϏεͷ஌ݟΛ্࣋ͬͨͰอकɾӡ༻Λߦ͏ඞཁ͕͋Δ
    ˞ಋೖޙʹඇػೳཁٻ͕໌֬Խͨ͠Βࣗલ΋ݕ౼΋
    ෼ੳۀ຿ʹूதͰ͖Δ

    View Slide

  23. σʔλͷΠϯϙʔτΤΫεϙʔτͷΤίγεςϜͷॆ࣮͍ͯ͠·͢
    ֤छσʔλϕʔε΍ϑΝΠϧαʔόʔͱͷૄ௨ͷόϦΤʔγϣϯ͕ଟ͘ɺ
    ਺ଟ͘ͷ֎෦γεςϜͱͷ࿈ܞΛ༰қͰ͢
    ֎෦γεςϜͱͷ࿈ܞ͕༰қ

    View Slide

  24. 5SFBTVSF%BUBʹσʔλ஝ੵΛ͢Ε͹ɺσʔλΞφϦετ͕෼ੳͰ͖Δঢ়ଶʹ ۃ࿦

    ΫΤϦ՝ۚ΍సૹྔ՝ۚͰ͸ͳ͍ͷͰ42-ʹਫ਼௨͍ͯ͠ͳ͍͕ΫΤϦൃߦͯ͠΋Ϋϥ΢υഁ࢈͠ͳ
    ͍
    σʔλ෼ੳɾՁ஋ݕূ·Ͱͷεϐʔυײͱ҆৺ײ

    View Slide

  25. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide