冪等性を考慮したデータ連携ジョブの設計

by hsasakawa

Slide 1

Slide 1 text

ႈ౳ੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.

Slide 2

Slide 2 text

ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ੒ • σʔλ࿈ܞδϣϒͷߏ੒ • σʔλ࿈ܞͱႈ౳ੑ • ચ͍ସ͑ • ࠩ෼ߋ৽ • ·ͱΊ

Slide 3

Slide 3 text

M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD 2VFSZ #JH2VFSZ

Slide 4

Slide 4 text

M3ͷσʔλج൫ͷߏ੒ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ 4ZOD #JH2VFSZ

Slide 5

Slide 5 text

σʔλ࿈ܞδϣϒͷߏ੒ "QQMJDBUJPO %#T #JH2VFSZ %JHEBH &NCVML %JHEBHͱ&NCVMLͷར༻ ར༻࣮੷ଟ਺ ϫʔΫϑϩʔɺσʔλநग़ͷ࣮૷͕؆୯ σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .64"ग़ுه.64"Ͱσʔλج൫؀ڥ੔උͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠  IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN

Slide 6

Slide 6 text

Embulkͷ࣮૷ ೖྗσʔλʹ͍ͭͯͷઃఆ %#઀ଓ৘ใҎ֎  ΄΅42- ग़ྗσʔλʹ͍ͭͯͷઃఆ

Slide 7

Slide 7 text

BigQuery্ͷσʔλ "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠  σʔλ

Slide 8

Slide 8 text

ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO %#T 4PVSDF .BSU NEPXOMPBEFS .-1SPEVDU #2ʹΞΫηεͯ͠ EBUBGSBNFܗࣜͰ  ΠΠײ͡ʹσʔλΛ  औಘ͢Δ  1ZUIPOQBDLBHF

Slide 9

Slide 9 text

σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠  σʔλ

Slide 10

Slide 10 text

σʔλ࿈ܞδϣϒͷߏ੒ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃ޻Λ ༧Ίద༻ͨ͠  σʔλ ͜͜ͱ ͜͜Λ ָʹ͍ͨ͠

Slide 11

Slide 11 text

δϣϒͷࣦഊͱϦΧόϦ δϣϒ͸͍ΖΜͳཧ༝Ͱࣦഊ͢Δ σʔλߋ৽ͷ஗Ԇ σʔλ૿Ճ 4BB4ଆͷෆ۩߹ͳͲͳͲ ϦΧόϦ͸δϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ ΦϖϨʔγϣϯϛεͷ๷ࢭ ్த·Ͱ௥Ճ͞ΕͨσʔλΛ࡟আͯ͠ʜ☠ ႈ౳ͳσʔλ࿈ܞδϣϒΛ࣮૷͍ͨ͠

Slide 12

Slide 12 text

σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO %#T ຖճσʔλΛ্ॻ͖͢ΔPS೔෇ผςʔϒϧʹ͢Δ ΄ͱΜͲͷ৔߹͸͜ΕͰ͍͍ ႈ౳ͳͷͰϦΧόϦ͸୯७ʹ࠶࣮ߦͰ0, ศརʂ

Slide 13

Slide 13 text

σʔλ࿈ܞύλʔϯ2: ࠩ෼ߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩ෼ͷΈΛ࿈ܞ σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ ୯७ʹ࣮૷͢Δͱႈ౳ʹͳΒͳ͍ δϣϒࣦഊ࣌ͷϦΧόϦ΋େม ೖͬͯ͠·ͬͨσʔλΛಛఆ࡟আͯ͠΍Γ௚͢ͳͲ

Slide 14

Slide 14 text

Slide 15

Slide 15 text

ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ

Slide 16

Slide 16 text

ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ

Slide 17

Slide 17 text

ႈ౳ͳࠩ෼ߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩ෼σʔλΛநग़ 4UFQ

Slide 18

Slide 18 text

ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ

Slide 19

Slide 19 text

ႈ౳ͳࠩ෼ߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞੒ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0, ඞཁ͕͋Ε͹ɺͲͷϑΣʔζ͔Β Ͱ΋΍Γ௚ͤΔ ϦΧόϦ

Slide 20

Slide 20 text

ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ͸࠶࣮ߦͰ0, ඞཁʹԠͯ͡೚ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ 4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐ΂ͯͨΒҎԼͷࢿྉͰ΋ղઆ͞Ε͍ͯ·ͨ͠ʜ ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ  ͓Ҿӽͨ͠͠࿩  IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ

Slide 21

Slide 21 text

஫ҙ఺ ͜ͷख๏ͷ஫ҙࣄ߲ ͦ΋ͦ΋σʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈ෼ͷ  ςʔϒϧΛอ͓࣋ͯ͘͠ͷ͸ݱ࣮తͰͳ͍ աڈͷ͋Δ࣌఺ʹ໭Δʹ͸σʔλͷ࣋ͪํΛ޻෉͢Δ  ඞཁ͕͋Δ FHCJUFNQPSBMEBUBNPEFMJOH  ૢ࡞ཤྺ࣌఺ࢦఆΞΫηεͷ࣮ݱ#J5FNQPSBM%BUB.PEFMͷ࣮ફ  IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT

Slide 22

Slide 22 text

·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ #JH2VFSZXJUI %JHEBH&NCVML ͸Ғେ ႈ౳ͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λ঺հ ચ͍ସ͑ ࠩ෼ߋ৽