Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ZOZOTOWNのバッチデータ転送基盤紹介 / ZOZOTOWN's data transfer batch

ZOZOTOWNのバッチデータ転送基盤紹介 / ZOZOTOWN's data transfer batch

Takehiro Shiozaki

December 08, 2018
Tweet

More Decks by Takehiro Shiozaki

Other Decks in Technology

Transcript

  1. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    %JHEBHͱ3VCZͰ࣮ݱ͢ΔϫʔΫϑϩʔ؅ཧ
    ;0;0508/ͷ
    όονσʔλసૹج൫঺հ
    גࣜձࣾ;0;0ςΫϊϩδʔζ
    ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ
    Ԙ࡚݈߂

    View Slide

  2. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ࣗݾ঺հ
    ৽ଔͰ7"4*-:ʹೖࣾޙɺ."Λܦ
    ͯ;0;0ςΫϊϩδʔζ΁ɻੲ͸
    3BJMTͰXFC "1*Λ࡞ͬͨΓɺݕࡧ
    αʔόʔͷύϑΥʔϚϯεɾνϡʔ
    χϯάΛ΍ͬͨΓɻ࠷ۙͷ͓࢓ࣄ͸
    σʔλج൫ͷ੔ཧͱ͔ɻ
    ;0;0ςΫϊϩδʔζ
    ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ
    Ԙ࡚݈߂

    View Slide

  3. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ͓͜ͱΘΓ
    3BJMTͷ࿩͸͠·ͤΜʂ

    View Slide

  4. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ໨࣍
    αʔϏε঺հ
    σʔλج൫ͷ঺հ
    σʔλϑϩʔ
    Πϯϑϥ
    Ͳ͜Ͱ3VCZΛ͔͍ͭͬͯΔͷ͔
    σʔλϚʔτͷࠩ෼ߋ৽ͷϩδοΫ
    STQFDͰ42-ϑΝΠϧͷςετ

    View Slide

  5. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ˓ Ҏ্ͷγϣοϓɺ Ҏ্
    ͷϒϥϯυͷऔΓѻ͍ʢ೥݄຤
    ࣌఺ʣ
    ˓ৗ࣌ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱ
    ຖ೔ฏۉ ఺Ҏ্ͷ৽ண঎඼Λܝࡌ
    ˓ଈ೔഑ૹαʔϏεΪϑτϥοϐϯ
    άαʔϏεπέ෷͍ͳͲ
    IUUQ[P[PKQ

    View Slide

  6. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ˓ ສμ΢ϯϩʔυಥഁɺίʔσΟ
    ωʔτ౤ߘ૯਺͸ສ݅ʢ೥
    ݄຤࣌఺ʣ
    ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ
    ͕ར༻ՄೳͳશͯͷࠃʣͰμ΢ϯϩʔυ
    ͕Մೳ
    ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔ
    βʔʢ8&"3*45"ʣ΋஀ੜ
    IUUQTXFBSKQ

    View Slide

  7. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ˓౰͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔ
    εʔπ
    ˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛε
    ϚʔτϑΥϯΧϝϥͰ౓ࡱӨ͢Δ͜
    ͱͰɺମܕσʔλΛܭଌ
    ˓ମܕσʔλ͸ɺॠ࣌ʹ%ϞσϧԽ͞
    Εɺ;0;0508/ΞϓϦʹอଘɻ
    IUUQ[P[PKQ[P[PTVJU

    View Slide

  8. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔ
    λΛ΋ͱʹɺҰਓͻͱΓͷମܕʹ߹ͬ
    ͨʮ͋ͳͨαΠζʯͷΞΠςϜ
    ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔ
    γοΫΞΠςϜΛఏڙ
    ˓άϩʔόϧαΠτʮ;0;0DPNʯ
    Ͱւ֎ల։
    IUUQ[P[PKQQC

    View Slide

  9. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ
    ;0;0508/%#
    8&"3%#
    1#%#
    1PXFS#*
    4 #JH2VFSZ
    όονసૹ
    ($4
    "OBMZTU
    σʔλूܭ
    σʔλ׆༻

    View Slide

  10. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ
    ;0;0508/%#
    8&"3%#
    1#%#
    1PXFS#*
    4 #JH2VFSZ
    όονసૹ
    ($4
    "OBMZTU
    σʔλूܭ
    σʔλ׆༻
    ࠓճ࿩͞ͳ͍෦෼

    View Slide

  11. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ཁૉٕज़

    View Slide

  12. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    &NCVMLͱ͸
    w&5- &YUSBDU5SBOTGFS-PBE
    πʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷσʔλΛόονసૹ
    wσʔλͷೖྗɾՃ޻ɾग़ྗΛ͢Δ෦෼ΛϓϥάΠϯͱͯ͠ఏڙ
    wઃఆϑΝΠϧ͸:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷγϟν͕ՄѪ͍

    View Slide

  13. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    %JHEBHͱ͸
    wϫʔΫϑϩʔ؅ཧπʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷ&NCVMLδϣϒΛޮ཰తʹ؅ཧ
    wฒྻ࣮ߦ
    wϦτϥΠ
    wλεΫͷ࣮ߦঢ়گͷՄࢹԽ
    w࣮ߦϩάͷදࣔ
    wઃఆϑΝΠϧ͸΄΅:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷϞάϥ͕ՄѪ͍

    View Slide

  14. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ
    ;0;0508/%#
    8&"3%#
    1#%#
    4 #JH2VFSZ
    &NCVMLʹΑΔ
    όονసૹ
    ($4
    σʔλूܭ
    ϫʔΫϑϩʔ؅ཧ

    View Slide

  15. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ

    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH

    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#

    View Slide

  16. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ

    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH

    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#
    δϣϒΩϡʔ
    λεΫͷ࣮ߦঢ়گ
    ࣮ߦϩά
    αʔόʔΛ૿΍ͯ͠
    ฒྻ౓Ξοϓ
    λεΫͷ࣮ߦ؀ڥ͸
    %PDLFSίϯςφ

    View Slide

  17. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ

    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH

    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#
    ࣮ߦϩά

    View Slide

  18. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    %JHEBHͷϫʔΫϑϩʔ঺հ
    +my_task_1:
    sh>: "echo this task runs first."
    +my_task_2:
    sh>: "echo this task runs next."
    ஞ࣮࣍ߦ
    +run_if_param_is_true:
    if>: ${param}
    _do:
    echo>: ${param} == true
    ৚݅෼ذ
    +repeat:
    for_each>:
    fruit: [apple, orange]
    _do:
    echo>: ${fruit}
    ϧʔϓ
    +repeat:
    for_each>:
    fruit: [apple, orange]
    _parallel: true
    _do:
    echo>: ${fruit}
    ฒྻ࣮ߦ

    View Slide

  19. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Ͳ͜Ͱ3VCZ͔ͭͬͯΔͷ͔ʁ
    w%JHEBHͷ3VCZΦϖϨʔλʔ
    w3VCZεΫϦϓτͰϫʔΫϑϩʔΛಈతʹ࡞Δ͜ͱ͕Ͱ͖Δ
    wIUUQTEPDTEJHEBHJPPQFSBUPSTSCIUNM

    View Slide

  20. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ෳ਺ςʔϒϧͷಉظ
    wFNCVMLσΟϨΫτϦҎԼͷશϑΝΠϧʹରͯ͠FNCVMLΛىಈ
    w3VCZͰϑΝΠϧ໊ҰཡΛऔಘ͠ɺ%JHEBHͰฒྻ࣮ߦ
    timezone: UTC
    +set_tables:
    require: tasks.rb
    rb>: set_tables
    +transfer_tables:
    for_each>:
    table: ${tables}
    _parallel: true
    _do:
    sh>: embulk run embulk/${table}.yml
    def set_tables
    tables = Dir.glob('embulk/*.yml')
    Digdag.env.store(tables: tables)
    end
    TBNQMFEJH UBTLTSC

    View Slide

  21. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ
    w΍Γ͍ͨ͜ͱ
    ूܭΫΤϦͷ࣮ߦ݁ՌΛಛఆͷςʔϒϧʹ೔࣍Ͱग़ྗͯ͠ੵΈ্͛ςʔϒϧʹ
    w❌ႈ౳Ͱ͸ͳ͍ྫ
    # INSERT INTO SELECTのBigQuery版
    query("集計クエリ", destination_table: 'table_name', write: 'append')
    w⭕ႈ౳ͳྫ
    table_yesterday = table('table_name_20181201')
    table_yesterday.copy('table_name_20181202')
    table_today = table('table_name_20181202')
    query("集計クエリ", destination_table: 'table_name_20181202', write: 'append')
    table_today.copy('table_name')
    [email protected]@ [email protected]@ [email protected]
    ᶃDPQZ ᶅDPQZ
    ᶄBQQFOE

    View Slide

  22. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ
    wҙ֎ͱΠϨΪϡϥʔύλʔϯ͕ଟ͘ɺ৚݅෼ذ͕ଟ͍
    w%JHEBHͷ੍ޚߏจ͚ͩͰ͸ࣗ༝౓͕গͳ͍
    wॲཧΛ੾Γସ͑ΔͨΊʹ3VCZΛར༻
    wHFNJOTUBMMHPPHMFDMPVEͰ($1ͷ"1*͕࢖͑Δ
    [email protected]@ [email protected]@ [email protected]
    ᶃDPQZ ᶅDPQZ
    ᶄBQQFOE
    ৽نσʔλϚʔτͷ࣌ʹ͸
    ࡢ೔ͷςʔϒϧ͕ͳ͍
    5JNF1BSUJUJPOFEςʔϒϧͷ৔߹͸
    ௨ৗͷDPQZ͸࢖͑ͳ͍

    View Slide

  23. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    wঢ়گ͋ΔσʔλϚʔτ͕ผͷσʔλϚʔτͷूܭ݁ՌΛར༻͍ͯ͠Δ
    wσʔλϚʔτͷूܭॲཧʹґଘؔ܎͕ൃੜʂ
    w΍Γ͍ͨ͜ͱґଘؔ܎Λຬͨͭͭ͠΋ɺฒྻ౓͸ՄೳͳݶΓ্͍͛ͨ
    EBUBNBSU
    EBUBNBSU
    EBUBNBSU
    4&-&$5ʙ'30.EBUBNBSU
    +0*/EBUBNBSUʙ
    EBUBNBSUͷूܭΑΓ΋ઌʹूܭ

    View Slide

  24. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"(
    ʹ͢Δ
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i)
    %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ
    " #
    #EFQFOETPO"

    View Slide

  25. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"(
    ʹ͢Δ
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i)
    %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ
    " #
    #EFQFOETPO"

    View Slide

  26. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    STQFDͰ42-ϑΝΠϧͷνΣοΫ
    w42-ϑΝΠϧΛॻ͘ͷ͸σʔλΞφϦετ
    wຊ൪ʹग़͔ͯ͠Βόον͕མͪΔ͜ͱ͕͠͹͠͹
    wจࣈίʔυ͕4+*4
    w#JH2VFSZͷXFC6*ͱ"1*Ͱͷࠩҟ
    wσʔλϚʔτͷґଘؔ܎͕॥؀
    w$JSDMF$*Ͱશ42-ϑΝΠϧͷจ๏νΣοΫΛߦ͍ݕ஌
    w͋ͱɺ$JSDMF$*ͰΠϯςάϨʔγϣϯςετ΋ߦ͍ͬͯΔ

    View Slide

  27. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    3VCZ࢖ͬͯΈͯΑ͔ͬͨ
    w࠷ॳ͸਺ߦͷͪΐͬͱͨ͠εΫϦϓτ
    wͩΜͩΜංେԽ͢ΔʹͭΕͯΦϒδΣΫτࢦ޲తͳઃܭʹ
    wΦϒδΣΫτࢦ޲ɾؔ਺ܕɾखଓ͖ܕͳͲͷελΠϧΛબ΂Δ
    wHFNΛ࢖͏͜ͱͰBXT΍($1ͷ"1*΋ୟ͚Δ

    View Slide

  28. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    8FSF)JSJOH
    w͍ͭ·Ͱ΋3BJMT΍͍͍ͬͯͷ͔ΩϟϦΞͰ೰ΜͰ͍Δํʂʂ
    wͦͷ3VCZܦݧɺฐࣾͷσʔλج൫ͳΒ׆͔ͤ·͢Α
    IUUQTUFDI[P[PDPNSFDSVJU

    View Slide