ZOZOTOWNのバッチデータ転送基盤紹介 / ZOZOTOWN's data transfer batch

Transcript

1. Copyright © ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͱ3VCZͰ࣮ݱ͢ΔϫʔΫϑϩʔ؅ཧ ;0;0508/ͷ
   όονσʔλసૹج൫঺հ גࣜձࣾ;0;0ςΫϊϩδʔζ
   ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ
   Ԙ࡚݈߂
2. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ࣗݾ঺հ ৽ଔͰ7"4*-:ʹೖࣾޙɺ."Λܦ ͯ;0;0ςΫϊϩδʔζ΁ɻੲ͸ 3BJMTͰXFC
   "1*Λ࡞ͬͨΓɺݕࡧ αʔόʔͷύϑΥʔϚϯεɾνϡʔ χϯάΛ΍ͬͨΓɻ࠷ۙͷ͓࢓ࣄ͸ σʔλج൫ͷ੔ཧͱ͔ɻ ;0;0ςΫϊϩδʔζ
   ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ Ԙ࡚
   ݈߂
3. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ͓͜ͱΘΓ 3BJMTͷ࿩͸͠·ͤΜʂ
4. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ໨࣍ αʔϏε঺հ
   σʔλج൫ͷ঺հ
   σʔλϑϩʔ
   Πϯϑϥ
   Ͳ͜Ͱ3VCZΛ͔͍ͭͬͯΔͷ͔
   σʔλϚʔτͷࠩ෼ߋ৽ͷϩδοΫ
   STQFDͰ42-ϑΝΠϧͷςετ
5. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓
    Ҏ্ͷγϣοϓɺ Ҏ্ ͷϒϥϯυͷऔΓѻ͍ʢ೥݄຤ ࣌఺ʣ
   ˓
   ৗ࣌ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱ ຖ೔ฏۉ ఺Ҏ্ͷ৽ண঎඼Λܝࡌ
   ˓
   ଈ೔഑ૹαʔϏε
   
   Ϊϑτϥοϐϯ άαʔϏε
   
   πέ෷͍
   ͳͲ
   IUUQ[P[PKQ
6. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓
    ສμ΢ϯϩʔυಥഁɺίʔσΟ ωʔτ౤ߘ૯਺͸ສ݅ʢ೥ ݄຤࣌఺ʣ
   ˓
   શੈքʢ"QQ
   4UPSF
   
   (PPHMF
   QMBZ ͕ར༻ՄೳͳશͯͷࠃʣͰμ΢ϯϩʔυ ͕Մೳ
   ˓
   ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔ βʔʢ8&"3*45"ʣ΋஀ੜ
   IUUQTXFBSKQ
7. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓
   ౰͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔ εʔπ
   ˓
   શମʹࢪ͞ΕͨυοτϚʔΧʔΛε ϚʔτϑΥϯΧϝϥͰ౓ࡱӨ͢Δ͜ ͱͰɺମܕσʔλΛܭଌ
   ˓
   ମܕσʔλ͸ɺॠ࣌ʹ%ϞσϧԽ͞ Εɺ;0;0508/ΞϓϦʹอଘɻ
   IUUQ[P[PKQ[P[PTVJU
8. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔ λΛ΋ͱʹɺҰਓͻͱΓͷମܕʹ߹ͬ ͨʮ͋ͳͨαΠζʯͷΞΠςϜ
   ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔ γοΫΞΠςϜΛఏڙ
   ˓
   άϩʔόϧαΠτʮ;0;0DPNʯ Ͱւ֎ల։
   IUUQ[P[PKQQC
9. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ ;0;0508/
   %# 8&"3
   %# 1#
   %# 1PXFS#* 4 #JH2VFSZ όονసૹ ($4 "OBMZTU σʔλूܭ σʔλ׆༻
10. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ ;0;0508/
    %# 8&"3
    %# 1#
    %# 1PXFS#* 4 #JH2VFSZ όονసૹ ($4 "OBMZTU σʔλूܭ σʔλ׆༻ ࠓճ࿩͞ͳ͍෦෼
11. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ཁૉٕज़
12. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. &NCVMLͱ͸ w&5- &YUSBDU
    5SBOTGFS
    -PBE πʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷσʔλΛόονసૹ
    wσʔλͷೖྗɾՃ޻ɾग़ྗΛ͢Δ෦෼ΛϓϥάΠϯͱͯ͠ఏڙ
    wઃఆϑΝΠϧ͸:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷγϟν͕ՄѪ͍
13. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͱ͸ wϫʔΫϑϩʔ؅ཧπʔϧ
    w5SFBTVSF%BUB੡ͷ044
    wେྔͷ&NCVMLδϣϒΛޮ཰తʹ؅ཧ
    wฒྻ࣮ߦ
    wϦτϥΠ
    wλεΫͷ࣮ߦঢ়گͷՄࢹԽ
    w࣮ߦϩάͷදࣔ
    wઃఆϑΝΠϧ͸΄΅:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯ؅ཧ͕༰қ
    wϩΰͷϞάϥ͕ՄѪ͍
14. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ ;0;0508/
    %# 8&"3
    %# 1#
    %# 4 #JH2VFSZ &NCVMLʹΑΔ
    όονసૹ ($4 σʔλूܭ ϫʔΫϑϩʔ؅ཧ
15. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$
    1FFSJOH %JHEBH
    NBTUFS "VSPSB %JHEBH
    XPSLFS
    "VUP4DBMJOH ɾɾɾ 4 0UIFS
    71$ ($4 #JH2VFSZ "-#
16. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$
    1FFSJOH %JHEBH
    NBTUFS "VSPSB %JHEBH
    XPSLFS
    "VUP4DBMJOH ɾɾɾ 4 0UIFS
    71$ ($4 #JH2VFSZ "-# δϣϒΩϡʔ
    λεΫͷ࣮ߦঢ়گ ࣮ߦϩά αʔόʔΛ૿΍ͯ͠
    ฒྻ౓Ξοϓ λεΫͷ࣮ߦ؀ڥ͸
    %PDLFSίϯςφ
17. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷ঺հ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$
    1FFSJOH %JHEBH
    NBTUFS "VSPSB %JHEBH
    XPSLFS
    "VUP4DBMJOH ɾɾɾ 4 0UIFS
    71$ ($4 #JH2VFSZ "-# ࣮ߦϩά
18. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͷϫʔΫϑϩʔ঺հ +my_task_1: sh>: "echo this task runs first." +my_task_2: sh>: "echo this task runs next." ஞ࣮࣍ߦ +run_if_param_is_true: if>: ${param} _do: echo>: ${param} == true ৚݅෼ذ +repeat: for_each>: fruit: [apple, orange] _do: echo>: ${fruit} ϧʔϓ +repeat: for_each>: fruit: [apple, orange] _parallel: true _do: echo>: ${fruit} ฒྻ࣮ߦ
19. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Ͳ͜Ͱ3VCZ͔ͭͬͯΔͷ͔ʁ w%JHEBHͷ3VCZΦϖϨʔλʔ
    w3VCZεΫϦϓτͰϫʔΫϑϩʔΛಈతʹ࡞Δ͜ͱ͕Ͱ͖Δ
    wIUUQTEPDTEJHEBHJPPQFSBUPSTSCIUNM
20. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. ෳ਺ςʔϒϧͷಉظ wFNCVMLσΟϨΫτϦҎԼͷશϑΝΠϧʹରͯ͠FNCVMLΛىಈ
    w3VCZͰϑΝΠϧ໊ҰཡΛऔಘ͠ɺ%JHEBHͰฒྻ࣮ߦ timezone: UTC +set_tables: require: tasks.rb rb>: set_tables +transfer_tables: for_each>: table: ${tables} _parallel: true _do: sh>: embulk run embulk/${table}.yml def set_tables tables = Dir.glob('embulk/*.yml') Digdag.env.store(tables: tables) end TBNQMFEJH UBTLTSC
21. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ w΍Γ͍ͨ͜ͱ
    
    
    ूܭΫΤϦͷ࣮ߦ݁ՌΛಛఆͷςʔϒϧʹ೔࣍Ͱग़ྗͯ͠ੵΈ্͛ςʔϒϧʹ
    w❌
    ႈ౳Ͱ͸ͳ͍ྫ
    # INSERT INTO SELECTのBigQuery版 query("集計クエリ", destination_table: 'table_name', write: 'append') w⭕
    ႈ౳ͳྫ
    table_yesterday = table('table_name_20181201') table_yesterday.copy('table_name_20181202') table_today = table('table_name_20181202') query("集計クエリ", destination_table: 'table_name_20181202', write: 'append') table_today.copy('table_name') UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF ᶃ
    DPQZ ᶅ
    DPQZ ᶄ
    BQQFOE
22. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷࠩ෼ߋ৽Λႈ౳ʹ wҙ֎ͱΠϨΪϡϥʔύλʔϯ͕ଟ͘ɺ৚݅෼ذ͕ଟ͍
    w%JHEBHͷ੍ޚߏจ͚ͩͰ͸ࣗ༝౓͕গͳ͍
    wॲཧΛ੾Γସ͑ΔͨΊʹ3VCZΛར༻
    wHFN
    JOTUBMM
    HPPHMFDMPVEͰ($1ͷ"1*͕࢖͑Δ UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF ᶃ
    DPQZ ᶅ
    DPQZ ᶄ
    BQQFOE ৽نσʔλϚʔτͷ࣌ʹ͸
    ࡢ೔ͷςʔϒϧ͕ͳ͍ 5JNF
    1BSUJUJPOFEςʔϒϧͷ৔߹͸
    ௨ৗͷDPQZ͸࢖͑ͳ͍
23. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔ܎ղܾ wঢ়گ
    ͋ΔσʔλϚʔτ͕ผͷσʔλϚʔτͷूܭ݁ՌΛར༻͍ͯ͠Δ
    w
    σʔλϚʔτͷूܭॲཧʹґଘؔ܎͕ൃੜʂ
    w΍Γ͍ͨ͜ͱ
    ґଘؔ܎Λຬͨͭͭ͠΋ɺฒྻ౓͸ՄೳͳݶΓ্͍͛ͨ EBUBNBSU EBUBNBSU EBUBNBSU 4&-&$5
    ʙ
    '30.
    EBUBNBSU
    
    
    +0*/
    EBUBNBSU
    ʙ EBUBNBSUͷूܭΑΓ΋ઌʹूܭ
24. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔ܎ղܾ σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"( ʹ͢Δ
    
    
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i) %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ " # #
    EFQFOET
    PO
    "
25. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔ܎ղܾ σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔ܎Λάϥϑ %"( ʹ͢Δ
    
    
    sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i) %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹ෼ׂ " # #
    EFQFOET
    PO
    "
26. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. STQFDͰ42-ϑΝΠϧͷνΣοΫ w42-ϑΝΠϧΛॻ͘ͷ͸σʔλΞφϦετ
    wຊ൪ʹग़͔ͯ͠Βόον͕མͪΔ͜ͱ͕͠͹͠͹
    wจࣈίʔυ͕4+*4
    w#JH2VFSZͷXFC6*ͱ"1*Ͱͷࠩҟ
    wσʔλϚʔτͷґଘؔ܎͕॥؀
    w$JSDMF$*Ͱશ42-ϑΝΠϧͷจ๏νΣοΫΛߦ͍ݕ஌
    w͋ͱɺ$JSDMF$*ͰΠϯςάϨʔγϣϯςετ΋ߦ͍ͬͯΔ
27. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. 3VCZ࢖ͬͯΈͯΑ͔ͬͨ w࠷ॳ͸਺ߦͷͪΐͬͱͨ͠εΫϦϓτ
    wͩΜͩΜංେԽ͢ΔʹͭΕͯΦϒδΣΫτࢦ޲తͳઃܭʹ
    wΦϒδΣΫτࢦ޲ɾؔ਺ܕɾखଓ͖ܕͳͲͷελΠϧΛબ΂Δ
    wHFNΛ࢖͏͜ͱͰBXT΍($1ͷ"1*΋ୟ͚Δ
28. Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright © ZOZO Technologies, Inc. All Rights Reserved. 8FSF
    )JSJOH w͍ͭ·Ͱ΋3BJMT΍͍͍ͬͯͷ͔ΩϟϦΞͰ೰ΜͰ͍Δํʂʂ
    wͦͷ3VCZܦݧɺฐࣾͷσʔλج൫ͳΒ׆͔ͤ·͢Α
    IUUQTUFDI[P[PDPNSFDSVJU