ZOZOTOWNのバッチデータ転送基盤紹介 / ZOZOTOWN's data transfer batch

Transcript

1. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   %JHEBHͱ3VCZͰ࣮ݱ͢ΔϫʔΫϑϩʔ؅ཧ
   ;0;0508/ͷ
   όονσʔλసૹج൫঺հ
   גࣜձࣾ;0;0ςΫϊϩδʔζ
   ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ
   Ԙ࡚݈߂
   

   View Slide

2. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ࣗݾ঺հ
   ৽ଔͰ7"4*-:ʹೖࣾޙɺ."Λܦ
   ͯ;0;0ςΫϊϩδʔζ΁ɻੲ͸
   3BJMTͰXFC "1*Λ࡞ͬͨΓɺݕࡧ
   αʔόʔͷύϑΥʔϚϯεɾνϡʔ
   χϯάΛ΍ͬͨΓɻ࠷ۙͷ͓࢓ࣄ͸
   σʔλج൫ͷ੔ཧͱ͔ɻ
   ;0;0ςΫϊϩδʔζ
   ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ
   Ԙ࡚݈߂
   

   View Slide

3. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ͓͜ͱΘΓ
   3BJMTͷ࿩͸͠·ͤΜʂ
   

   View Slide

4. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ໨࣍
   αʔϏε঺հ
   σʔλج൫ͷ঺հ
   σʔλϑϩʔ
   Πϯϑϥ
   Ͳ͜Ͱ3VCZΛ͔͍ͭͬͯΔͷ͔
   σʔλϚʔτͷࠩ෼ߋ৽ͷϩδοΫ
   STQFDͰ42-ϑΝΠϧͷςετ
   

   View Slide

5. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓ Ҏ্ͷγϣοϓɺ Ҏ্
   ͷϒϥϯυͷऔΓѻ͍ʢ೥݄຤
   ࣌఺ʣ
   ˓ৗ࣌ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱ
   ຖ೔ฏۉ ఺Ҏ্ͷ৽ண঎඼Λܝࡌ
   ˓ଈ೔഑ૹαʔϏεΪϑτϥοϐϯ
   άαʔϏεπέ෷͍ͳͲ
   IUUQ[P[PKQ
   

   View Slide

6. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓ ສμ΢ϯϩʔυಥഁɺίʔσΟ
   ωʔτ౤ߘ૯਺͸ສ݅ʢ೥
   ݄຤࣌఺ʣ
   ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ
   ͕ར༻ՄೳͳશͯͷࠃʣͰμ΢ϯϩʔυ
   ͕Մೳ
   ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔ
   βʔʢ8&"3*45"ʣ΋஀ੜ
   IUUQTXFBSKQ
   

   View Slide

7. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓౰͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔ
   εʔπ
   ˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛε
   ϚʔτϑΥϯΧϝϥͰ౓ࡱӨ͢Δ͜
   ͱͰɺମܕσʔλΛܭଌ
   ˓ମܕσʔλ͸ɺॠ࣌ʹ%ϞσϧԽ͞
   Εɺ;0;0508/ΞϓϦʹอଘɻ
   IUUQ[P[PKQ[P[PTVJU
   

   View Slide

8. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔ
   λΛ΋ͱʹɺҰਓͻͱΓͷମܕʹ߹ͬ
   ͨʮ͋ͳͨαΠζʯͷΞΠςϜ
   ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔ
   γοΫΞΠςϜΛఏڙ
   ˓άϩʔόϧαΠτʮ;0;0DPNʯ
   Ͱւ֎ల։
   IUUQ[P[PKQQC
   

   View Slide

9. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   Copyright © ZOZO Technologies, Inc. All Rights Reserved.
   σʔλج൫ͷ঺հ
   ;0;0508/%#
   8&"3%#
   1#%#
   1PXFS#*
   4 #JH2VFSZ
   όονసૹ
   ($4
   "OBMZTU
   σʔλूܭ
   σʔλ׆༻
   

   View Slide

10. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ
    ;0;0508/%#
    8&"3%#
    1#%#
    1PXFS#*
    4 #JH2VFSZ
    όονసૹ
    ($4
    "OBMZTU
    σʔλूܭ
    σʔλ׆༻
    ࠓճ࿩͞ͳ͍෦෼
    

    View Slide

11. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ཁૉٕज़
    

    View Slide

12. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    &NCVMLͱ͸
    w&5- &YUSBDU5SBOTGFS-PBE
    πʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷσʔλΛόονసૹ
    wσʔλͷೖྗɾՃ޻ɾग़ྗΛ͢Δ෦෼ΛϓϥάΠϯͱͯ͠ఏڙ
    wઃఆϑΝΠϧ͸:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷγϟν͕ՄѪ͍
    

    View Slide

13. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    %JHEBHͱ͸
    wϫʔΫϑϩʔ؅ཧπʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷ&NCVMLδϣϒΛޮ཰తʹ؅ཧ
    wฒྻ࣮ߦ
    wϦτϥΠ
    wλεΫͷ࣮ߦঢ়گͷՄࢹԽ
    w࣮ߦϩάͷදࣔ
    wઃఆϑΝΠϧ͸΄΅:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷϞάϥ͕ՄѪ͍
    

    View Slide

14. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ
    ;0;0508/%#
    8&"3%#
    1#%#
    4 #JH2VFSZ
    &NCVMLʹΑΔ
    όονసૹ
    ($4
    σʔλूܭ
    ϫʔΫϑϩʔ؅ཧ
    

    View Slide

15. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ
    
    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH
    
    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#
    

    View Slide

16. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ
    
    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH
    
    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#
    δϣϒΩϡʔ
    λεΫͷ࣮ߦঢ়گ
    ࣮ߦϩά
    αʔόʔΛ૿΍ͯ͠
    ฒྻ౓Ξοϓ
    λεΫͷ࣮ߦ؀ڥ͸
    %PDLFSίϯςφ
    

    View Slide

17. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλج൫ͷ঺հ Πϯϑϥ
    
    %BUB$FOUFS
    %JSFDU$POOFDU 71$1FFSJOH
    %JHEBHNBTUFS
    "VSPSB
    %JHEBHXPSLFS
    "VUP4DBMJOH
    
    ɾɾɾ
    4
    0UIFS71$
    ($4
    #JH2VFSZ
    "-#
    ࣮ߦϩά
    

    View Slide

18. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    %JHEBHͷϫʔΫϑϩʔ঺հ
    +my_task_1:
    sh>: "echo this task runs first."
    +my_task_2:
    sh>: "echo this task runs next."
    ஞ࣮࣍ߦ
    +run_if_param_is_true:
    if>: ${param}
    _do:
    echo>: ${param} == true
    ৚݅෼ذ
    +repeat:
    for_each>:
    fruit: [apple, orange]
    _do:
    echo>: ${fruit}
    ϧʔϓ
    +repeat:
    for_each>:
    fruit: [apple, orange]
    _parallel: true
    _do:
    echo>: ${fruit}
    ฒྻ࣮ߦ
    

    View Slide

19. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Ͳ͜Ͱ3VCZ͔ͭͬͯΔͷ͔ʁ
    w%JHEBHͷ3VCZΦϖϨʔλʔ
    w3VCZεΫϦϓτͰϫʔΫϑϩʔΛಈతʹ࡞Δ͜ͱ͕Ͱ͖Δ
    wIUUQTEPDTEJHEBHJPPQFSBUPSTSCIUNM
    

    View Slide

20. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    ෳ਺ςʔϒϧͷಉظ
    wFNCVMLσΟϨΫτϦҎԼͷશϑΝΠϧʹରͯ͠FNCVMLΛىಈ
    w3VCZͰϑΝΠϧ໊ҰཡΛऔಘ͠ɺ%JHEBHͰฒྻ࣮ߦ
    timezone: UTC
    +set_tables:
    require: tasks.rb
    rb>: set_tables
    +transfer_tables:
    for_each>:
    table: ${tables}
    _parallel: true
    _do:
    sh>: embulk run embulk/${table}.yml
    def set_tables
    tables = Dir.glob('embulk/*.yml')
    Digdag.env.store(tables: tables)
    end
    TBNQMFEJH UBTLTSC
    

    View Slide

21. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ
    w΍Γ͍ͨ͜ͱ
    ूܭΫΤϦͷ࣮ߦ݁ՌΛಛఆͷςʔϒϧʹ೔࣍Ͱग़ྗͯ͠ੵΈ্͛ςʔϒϧʹ
    w❌ႈ౳Ͱ͸ͳ͍ྫ
    # INSERT INTO SELECTのBigQuery版
    query("集計クエリ", destination_table: 'table_name', write: 'append')
    w⭕ႈ౳ͳྫ
    table_yesterday = table('table_name_20181201')
    table_yesterday.copy('table_name_20181202')
    table_today = table('table_name_20181202')
    query("集計クエリ", destination_table: 'table_name_20181202', write: 'append')
    table_today.copy('table_name')
    UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF
    ᶃDPQZ ᶅDPQZ
    ᶄBQQFOE
    

    View Slide

22. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ
    wҙ֎ͱΠϨΪϡϥʔύλʔϯ͕ଟ͘ɺ৚݅෼ذ͕ଟ͍
    w%JHEBHͷ੍ޚߏจ͚ͩͰ͸ࣗ༝౓͕গͳ͍
    wॲཧΛ੾Γସ͑ΔͨΊʹ3VCZΛར༻
    wHFNJOTUBMMHPPHMFDMPVEͰ($1ͷ"1*͕࢖͑Δ
    UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF
    ᶃDPQZ ᶅDPQZ
    ᶄBQQFOE
    ৽نσʔλϚʔτͷ࣌ʹ͸
    ࡢ೔ͷςʔϒϧ͕ͳ͍
    5JNF1BSUJUJPOFEςʔϒϧͷ৔߹͸
    ௨ৗͷDPQZ͸࢖͑ͳ͍
    

    View Slide

23. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    wঢ়گ͋ΔσʔλϚʔτ͕ผͷσʔλϚʔτͷूܭ݁ՌΛར༻͍ͯ͠Δ
    wσʔλϚʔτͷूܭॲཧʹґଘؔ܎͕ൃੜʂ
    w΍Γ͍ͨ͜ͱґଘؔ܎Λຬͨͭͭ͠΋ɺฒྻ౓͸ՄೳͳݶΓ্͍͛ͨ
    EBUBNBSU
    EBUBNBSU
    EBUBNBSU
    4&-&$5ʙ'30.EBUBNBSU
    +0*/EBUBNBSUʙ
    EBUBNBSUͷूܭΑΓ΋ઌʹूܭ
    

    View Slide

24. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"(
    ʹ͢Δ
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i)
    %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ
    " #
    #EFQFOETPO"
    

    View Slide

25. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    σʔλϚʔτͷूܭͷґଘؔ܎ղܾ
    σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"(
    ʹ͢Δ
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i)
    %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ
    " #
    #EFQFOETPO"
    

    View Slide

26. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    STQFDͰ42-ϑΝΠϧͷνΣοΫ
    w42-ϑΝΠϧΛॻ͘ͷ͸σʔλΞφϦετ
    wຊ൪ʹग़͔ͯ͠Βόον͕མͪΔ͜ͱ͕͠͹͠͹
    wจࣈίʔυ͕4+*4
    w#JH2VFSZͷXFC6*ͱ"1*Ͱͷࠩҟ
    wσʔλϚʔτͷґଘؔ܎͕॥؀
    w$JSDMF$*Ͱશ42-ϑΝΠϧͷจ๏νΣοΫΛߦ͍ݕ஌
    w͋ͱɺ$JSDMF$*ͰΠϯςάϨʔγϣϯςετ΋ߦ͍ͬͯΔ
    

    View Slide

27. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    3VCZ࢖ͬͯΈͯΑ͔ͬͨ
    w࠷ॳ͸਺ߦͷͪΐͬͱͨ͠εΫϦϓτ
    wͩΜͩΜංେԽ͢ΔʹͭΕͯΦϒδΣΫτࢦ޲తͳઃܭʹ
    wΦϒδΣΫτࢦ޲ɾؔ਺ܕɾखଓ͖ܕͳͲͷελΠϧΛબ΂Δ
    wHFNΛ࢖͏͜ͱͰBXT΍($1ͷ"1*΋ୟ͚Δ
    

    View Slide

28. Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    Copyright © ZOZO Technologies, Inc. All Rights Reserved.
    8FSF)JSJOH
    w͍ͭ·Ͱ΋3BJMT΍͍͍ͬͯͷ͔ΩϟϦΞͰ೰ΜͰ͍Δํʂʂ
    wͦͷ3VCZܦݧɺฐࣾͷσʔλج൫ͳΒ׆͔ͤ·͢Α
    IUUQTUFDI[P[PDPNSFDSVJU
    

    View Slide