冪等性を考慮したデータ連携ジョブの設計

Transcript

1. ႈ౳ੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.
2. ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ੒ • σʔλ࿈ܞδϣϒͷߏ੒ • σʔλ࿈ܞͱႈ౳ੑ • ચ͍ସ͑ • ࠩ෼ߋ৽ • ·ͱΊ
3. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD 2VFSZ #JH2VFSZ
4. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ 4ZOD #JH2VFSZ
5. σʔλ࿈ܞδϣϒͷߏ੒ "QQMJDBUJPO
   %#T #JH2VFSZ %JHEBH &NCVML
   
   
   
   
   
   %JHEBHͱ
   
   
   
   
   
   
   
   
   
   &NCVMLͷར༻
    ར༻࣮੷ଟ਺
    ϫʔΫϑϩʔɺσʔλநग़ͷ࣮૷͕؆୯
    σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .
   64"
   ग़ுه
   
   .
   64"Ͱσʔλج൫؀ڥ੔උͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠
 IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN
6. Embulkͷ࣮૷ ೖྗσʔλʹ͍ͭͯͷઃఆ
    %#઀ଓ৘ใҎ֎
 ΄΅42-
   ग़ྗσʔλʹ͍ͭͯͷઃఆ
7. BigQuery্ͷσʔλ "QQMJDBUJPO
   %#T 4PVSDF
   ݩςʔϒϧͷ ίϐʔ .BSU
   සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠
 σʔλ
8. ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO
   %#T 4PVSDF .BSU NEPXOMPBEFS .-
   1SPEVDU #2ʹΞΫηεͯ͠ EBUB
   GSBNFܗࣜͰ
 ΠΠײ͡ʹσʔλΛ
 औಘ͢Δ
 1ZUIPO
   QBDLBHF
9. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO
   %#T 4PVSDF
   ݩςʔϒϧͷ ίϐʔ .BSU
   සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠
 σʔλ
10. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO
    %#T 4PVSDF
    ݩςʔϒϧͷ ίϐʔ .BSU
    සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠
 σʔλ ͜͜ͱ ͜͜Λ
    ָʹ͍ͨ͠
11. δϣϒͷࣦഊͱϦΧόϦ δϣϒ͸͍ΖΜͳཧ༝Ͱࣦഊ͢Δ
     σʔλߋ৽ͷ஗Ԇ
     σʔλ૿Ճ
     4BB4ଆͷෆ۩߹ͳͲͳͲ
    ϦΧόϦ͸δϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ
     ΦϖϨʔγϣϯϛεͷ๷ࢭ
     ్த·Ͱ௥Ճ͞ΕͨσʔλΛ࡟আͯ͠ʜ☠ ႈ౳ͳσʔλ࿈ܞδϣϒΛ࣮૷͍ͨ͠
12. σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO
    %#T ຖճσʔλΛ্ॻ͖͢Δ
    PS
    ೔෇ผςʔϒϧʹ͢Δ
     ΄ͱΜͲͷ৔߹͸͜ΕͰ͍͍
     ႈ౳ͳͷͰϦΧόϦ͸୯७ʹ࠶࣮ߦͰ0,
     ศརʂ
13. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO
    %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ
     σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ
     ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍
     δϣϒࣦഊ࣌ͷϦΧόϦ΋େม
     ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ
14. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO
    %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ
     σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ
     ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍
     δϣϒࣦഊ࣌ͷϦΧόϦ΋େม
     ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ
15. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ
16. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ
17. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩ෼σʔλΛநग़ 4UFQ
18. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ
19. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO
    %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0,
    ඞཁ͕͋Ε͹ɺͲͷϑΣʔζ͔Β
    Ͱ΋΍Γ௚ͤΔ ϦΧόϦ
20. ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ͸࠶࣮ߦͰ0,
     ඞཁʹԠͯ͡೚ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ
     4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐ΂ͯͨΒҎԼͷࢿྉͰ΋ղઆ͞Ε͍ͯ·ͨ͠ʜ
     ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ
 ͓Ҿӽͨ͠͠࿩
 IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ
21. ஫ҙ఺ ͜ͷख๏ͷ஫ҙࣄ߲
     ͦ΋ͦ΋σʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈ෼ͷ
 ςʔϒϧΛอ͓࣋ͯ͘͠ͷ͸ݱ࣮తͰͳ͍
     աڈͷ͋Δ࣌఺ʹ໭Δʹ͸σʔλͷ࣋ͪํΛ޻෉͢Δ
 ඞཁ͕͋Δ
     FH
    CJUFNQPSBM
    EBUB
    NPEFMJOH
    
 ૢ࡞ཤྺ࣌఺ࢦఆΞΫηεͷ࣮ݱ
    
    #J5FNQPSBM
    %BUB
    .PEFM
    ͷ࣮ફ
 IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT
22. ·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ
     #JH2VFSZ
    XJUI
    %JHEBH
    
    &NCVML ͸Ғେ
     ႈ౳ͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λ঺հ
     ચ͍ସ͑
     ࠩ෼ߋ৽