冪等性を考慮したデータ連携ジョブの設計

Transcript

1. ႈ౳ੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.

2. ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ੒ • σʔλ࿈ܞδϣϒͷߏ੒ • σʔλ࿈ܞͱႈ౳ੑ • ચ͍ସ͑ •

   ࠩ෼ߋ৽ • ·ͱΊ

3. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD

   2VFSZ #JH2VFSZ

4. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ

   4ZOD #JH2VFSZ

5. σʔλ࿈ܞδϣϒͷߏ੒ "QQMJDBUJPO
   %#T #JH2VFSZ %JHEBH &NCVML
   
   
   
   
   
   %JHEBHͱ
   
   
   
   
   
   
   
   
   
   &NCVMLͷར༻
    ར༻࣮੷ଟ਺
   

   ϫʔΫϑϩʔɺσʔλநग़ͷ࣮૷͕؆୯
    σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .
   64"
   ग़ுه
   
   .
   64"Ͱσʔλج൫؀ڥ੔උͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠
 IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN

6. Embulkͷ࣮૷ ೖྗσʔλʹ͍ͭͯͷઃఆ
    %#઀ଓ৘ใҎ֎
 ΄΅42-
   ग़ྗσʔλʹ͍ͭͯͷઃఆ

7. BigQuery্ͷσʔλ "QQMJDBUJPO
   %#T 4PVSDF
   ݩςʔϒϧͷ ίϐʔ .BSU
   සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠
 σʔλ

8. ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO
   %#T 4PVSDF .BSU NEPXOMPBEFS .-
   1SPEVDU #2ʹΞΫηεͯ͠ EBUB
   GSBNFܗࣜͰ
 ΠΠײ͡ʹσʔλΛ


   औಘ͢Δ
 1ZUIPO
   QBDLBHF

9. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO
   %#T 4PVSDF
   ݩςʔϒϧͷ ίϐʔ .BSU
   සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠


   σʔλ

10. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO
    %#T 4PVSDF
    ݩςʔϒϧͷ ίϐʔ .BSU
    සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠


    σʔλ ͜͜ͱ ͜͜Λ
    ָʹ͍ͨ͠

11. δϣϒͷࣦഊͱϦΧόϦ δϣϒ͸͍ΖΜͳཧ༝Ͱࣦഊ͢Δ
     σʔλߋ৽ͷ஗Ԇ
     σʔλ૿Ճ
     4BB4ଆͷෆ۩߹ͳͲͳͲ
    ϦΧόϦ͸δϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ
    

    ΦϖϨʔγϣϯϛεͷ๷ࢭ
     ్த·Ͱ௥Ճ͞ΕͨσʔλΛ࡟আͯ͠ʜ☠ ႈ౳ͳσʔλ࿈ܞδϣϒΛ࣮૷͍ͨ͠

12. σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO
    %#T ຖճσʔλΛ্ॻ͖͢Δ
    PS
    ೔෇ผςʔϒϧʹ͢Δ
     ΄ͱΜͲͷ৔߹͸͜ΕͰ͍͍
     ႈ౳ͳͷͰϦΧόϦ͸୯७ʹ࠶࣮ߦͰ0,
    

    ศརʂ

13. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO
    %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ
     σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ
     ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍
    

    δϣϒࣦഊ࣌ͷϦΧόϦ΋େม
     ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ

14. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO
    %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ
     σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ
     ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍
    

    δϣϒࣦഊ࣌ͷϦΧόϦ΋େม
     ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ

15. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ

16. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ

17. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩ෼σʔλΛநग़

    4UFQ

18. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ

19. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0,
    ඞཁ͕͋Ε͹ɺͲͷϑΣʔζ͔Β
    

    Ͱ΋΍Γ௚ͤΔ ϦΧόϦ

20. ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ͸࠶࣮ߦͰ0,
     ඞཁʹԠͯ͡೚ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ
     4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐ΂ͯͨΒҎԼͷࢿྉͰ΋ղઆ͞Ε͍ͯ·ͨ͠ʜ
     ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ
 ͓Ҿӽͨ͠͠࿩


    IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ

21. ஫ҙ఺ ͜ͷख๏ͷ஫ҙࣄ߲
     ͦ΋ͦ΋σʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈ෼ͷ
 ςʔϒϧΛอ͓࣋ͯ͘͠ͷ͸ݱ࣮తͰͳ͍
     աڈͷ͋Δ࣌఺ʹ໭Δʹ͸σʔλͷ࣋ͪํΛ޻෉͢Δ
 ඞཁ͕͋Δ
     FH
    CJUFNQPSBM
    EBUB
    NPEFMJOH
    


    ૢ࡞ཤྺ࣌఺ࢦఆΞΫηεͷ࣮ݱ
    
    #J5FNQPSBM
    %BUB
    .PEFM
    ͷ࣮ફ
 IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT

22. ·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ
     #JH2VFSZ
    XJUI
    %JHEBH
    
    &NCVML ͸Ғେ
     ႈ౳ͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λ঺հ
    

    ચ͍ସ͑
     ࠩ෼ߋ৽