冪等性を考慮したデータ連携ジョブの設計

8274bab3875ee7d5d7482c4aae4f6990?s=47 hsasakawa
April 12, 2019

 冪等性を考慮したデータ連携ジョブの設計

8274bab3875ee7d5d7482c4aae4f6990?s=128

hsasakawa

April 12, 2019
Tweet

Transcript

  1. ႈ౳ੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.

  2. ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ੒ • σʔλ࿈ܞδϣϒͷߏ੒ • σʔλ࿈ܞͱႈ౳ੑ • ચ͍ସ͑ •

    ࠩ෼ߋ৽ • ·ͱΊ
  3. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD

    2VFSZ #JH2VFSZ
  4. M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ

    4ZOD #JH2VFSZ
  5. σʔλ࿈ܞδϣϒͷߏ੒ "QQMJDBUJPO %#T #JH2VFSZ %JHEBH &NCVML %JHEBHͱ&NCVMLͷར༻  ར༻࣮੷ଟ਺ 

    ϫʔΫϑϩʔɺσʔλநग़ͷ࣮૷͕؆୯  σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .64"ग़ுه.64"Ͱσʔλج൫؀ڥ੔උͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠
 IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN
  6. Embulkͷ࣮૷ ೖྗσʔλʹ͍ͭͯͷઃఆ  %#઀ଓ৘ใҎ֎
 ΄΅42- ग़ྗσʔλʹ͍ͭͯͷઃఆ

  7. BigQuery্ͷσʔλ "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠
 σʔλ

  8. ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO %#T 4PVSDF .BSU NEPXOMPBEFS .-1SPEVDU #2ʹΞΫηεͯ͠ EBUBGSBNFܗࣜͰ
 ΠΠײ͡ʹσʔλΛ


    औಘ͢Δ
 1ZUIPOQBDLBHF
  9. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠


    σʔλ
  10. σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠


    σʔλ ͜͜ͱ ͜͜Λ ָʹ͍ͨ͠
  11. δϣϒͷࣦഊͱϦΧόϦ δϣϒ͸͍ΖΜͳཧ༝Ͱࣦഊ͢Δ  σʔλߋ৽ͷ஗Ԇ  σʔλ૿Ճ  4BB4ଆͷෆ۩߹ͳͲͳͲ ϦΧόϦ͸δϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ 

    ΦϖϨʔγϣϯϛεͷ๷ࢭ  ్த·Ͱ௥Ճ͞ΕͨσʔλΛ࡟আͯ͠ʜ☠ ႈ౳ͳσʔλ࿈ܞδϣϒΛ࣮૷͍ͨ͠
  12. σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO %#T ຖճσʔλΛ্ॻ͖͢ΔPS೔෇ผςʔϒϧʹ͢Δ  ΄ͱΜͲͷ৔߹͸͜ΕͰ͍͍  ႈ౳ͳͷͰϦΧόϦ͸୯७ʹ࠶࣮ߦͰ0, 

    ศརʂ
  13. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ  σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ  ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍ 

    δϣϒࣦഊ࣌ͷϦΧόϦ΋େม  ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ
  14. σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ  σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ  ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍ 

    δϣϒࣦഊ࣌ͷϦΧόϦ΋େม  ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ
  15. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ

  16. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ

  17. ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩ෼σʔλΛநग़

    4UFQ
  18. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ

  19. ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0, ඞཁ͕͋Ε͹ɺͲͷϑΣʔζ͔Β

    Ͱ΋΍Γ௚ͤΔ ϦΧόϦ
  20. ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ͸࠶࣮ߦͰ0,  ඞཁʹԠͯ͡೚ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ  4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐ΂ͯͨΒҎԼͷࢿྉͰ΋ղઆ͞Ε͍ͯ·ͨ͠ʜ  ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ
 ͓Ҿӽͨ͠͠࿩


    IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ
  21. ஫ҙ఺ ͜ͷख๏ͷ஫ҙࣄ߲  ͦ΋ͦ΋σʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈ෼ͷ
 ςʔϒϧΛอ͓࣋ͯ͘͠ͷ͸ݱ࣮తͰͳ͍  աڈͷ͋Δ࣌఺ʹ໭Δʹ͸σʔλͷ࣋ͪํΛ޻෉͢Δ
 ඞཁ͕͋Δ  FHCJUFNQPSBMEBUBNPEFMJOH


    ૢ࡞ཤྺ࣌఺ࢦఆΞΫηεͷ࣮ݱ#J5FNQPSBM%BUB.PEFMͷ࣮ફ
 IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT
  22. ·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ  #JH2VFSZXJUI %JHEBH &NCVML ͸Ғେ  ႈ౳ͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λ঺հ 

    ચ͍ସ͑  ࠩ෼ߋ৽