Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
冪等性を考慮したデータ連携ジョブの設計
Search
hsasakawa
April 12, 2019
Technology
4
1.7k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
April 12, 2019
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
80
行動ログ処理基盤の構築
hsasakawa
0
3.2k
Data platform development on M3 USA
hsasakawa
0
980
Data Analysis Platform Development @M3, inc.
hsasakawa
0
3.3k
Other Decks in Technology
See All in Technology
複雑な構成要素を持つUIとの向き合い方 〜新・支出グラフでの実例〜 / B43 TECH TALK
nakamuuu
0
140
Postman v10リリース後を振り返る / Looking back at Postman v10 after release
yokawasa
1
160
非同期推論システムによるコスト削減と信頼性向上
koki_nishihara
0
240
Reducing Cross-Zone Egress at Spotify with Custom gRPC Load Balancing Recap
koh_naga
0
200
プロトタイピングによる不確実性の低減 / Reducing Uncertainty through Prototyping
ohbarye
5
380
どうするコスト最適化のトレードオフ
tetsuyaooooo
1
510
家族アルバム みてねにおけるGrafana活用術 / Grafana Meetup Japan Vol.1 LT
isaoshimizu
1
580
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
350
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
300
Meta Quest 3 で動く桜マシマシ WebXR アプリを IBM Cloud Code Engine と Babylon.js で作った話
1ftseabass
PRO
0
120
障害対応をちょっとずつよくしていくための 演習の作りかた
heleeen
0
140
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
100
Featured
See All Featured
Making Projects Easy
brettharned
108
5.5k
Atom: Resistance is Futile
akmur
259
25k
Building Flexible Design Systems
yeseniaperezcruz
319
37k
How to name files
jennybc
65
93k
Building Your Own Lightsaber
phodgson
99
5.7k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Become a Pro
speakerdeck
PRO
11
4.5k
Designing the Hi-DPI Web
ddemaree
276
33k
Testing 201, or: Great Expectations
jmmastey
28
6.3k
Bootstrapping a Software Product
garrettdimon
PRO
302
110k
Debugging Ruby Performance
tmm1
70
11k
Being A Developer After 40
akosma
57
580k
Transcript
ႈੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.
ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ • σʔλ࿈ܞδϣϒͷߏ • σʔλ࿈ܞͱႈੑ • ચ͍ସ͑ •
ࠩߋ৽ • ·ͱΊ
M3ͷσʔλج൫ͷߏ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλج൫ͷߏ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ
4ZOD #JH2VFSZ
σʔλ࿈ܞδϣϒͷߏ "QQMJDBUJPO %#T #JH2VFSZ %JHEBH &NCVML %JHEBHͱ&NCVMLͷར༻ ར༻࣮ଟ
ϫʔΫϑϩʔɺσʔλநग़ͷ࣮͕؆୯ σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .64"ग़ுه.64"Ͱσʔλج൫ڥඋͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠ IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN
Embulkͷ࣮ ೖྗσʔλʹ͍ͭͯͷઃఆ %#ଓใҎ֎ ΄΅42- ग़ྗσʔλʹ͍ͭͯͷઃఆ
BigQuery্ͷσʔλ "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠ σʔλ
ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO %#T 4PVSDF .BSU NEPXOMPBEFS .-1SPEVDU #2ʹΞΫηεͯ͠ EBUBGSBNFܗࣜͰ ΠΠײ͡ʹσʔλΛ
औಘ͢Δ 1ZUIPOQBDLBHF
σʔλ࿈ܞδϣϒͷߏ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠
σʔλ
σʔλ࿈ܞδϣϒͷߏ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠
σʔλ ͜͜ͱ ͜͜Λ ָʹ͍ͨ͠
δϣϒͷࣦഊͱϦΧόϦ δϣϒ͍ΖΜͳཧ༝Ͱࣦഊ͢Δ σʔλߋ৽ͷԆ σʔλ૿Ճ 4BB4ଆͷෆ۩߹ͳͲͳͲ ϦΧόϦδϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ
ΦϖϨʔγϣϯϛεͷࢭ ్த·ͰՃ͞ΕͨσʔλΛআͯ͠ʜ☠ ႈͳσʔλ࿈ܞδϣϒΛ࣮͍ͨ͠
σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO %#T ຖճσʔλΛ্ॻ͖͢ΔPSผςʔϒϧʹ͢Δ ΄ͱΜͲͷ߹͜ΕͰ͍͍ ႈͳͷͰϦΧόϦ୯७ʹ࠶࣮ߦͰ0,
ศརʂ
σʔλ࿈ܞύλʔϯ2: ࠩߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩͷΈΛ࿈ܞ σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ ୯७ʹ࣮͢ΔͱႈʹͳΒͳ͍
δϣϒࣦഊ࣌ͷϦΧόϦେม ೖͬͯ͠·ͬͨσʔλΛಛఆআͯ͠Γ͢ͳͲ
σʔλ࿈ܞύλʔϯ2: ࠩߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩͷΈΛ࿈ܞ σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ ୯७ʹ࣮͢ΔͱႈʹͳΒͳ͍
δϣϒࣦഊ࣌ͷϦΧόϦେม ೖͬͯ͠·ͬͨσʔλΛಛఆআͯ͠Γ͢ͳͲ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩσʔλΛநग़
4UFQ
ႈͳࠩߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞ 4UFQ
ႈͳࠩߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0, ඞཁ͕͋ΕɺͲͷϑΣʔζ͔Β
ͰΓͤΔ ϦΧόϦ
ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ࠶࣮ߦͰ0, ඞཁʹԠͯ͡ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ 4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐͯͨΒҎԼͷࢿྉͰղઆ͞Ε͍ͯ·ͨ͠ʜ ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ ͓Ҿӽͨ͠͠
IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ
ҙ ͜ͷख๏ͷҙࣄ߲ ͦͦσʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈͷ ςʔϒϧΛอ͓࣋ͯ͘͠ͷݱ࣮తͰͳ͍ աڈͷ͋Δ࣌ʹΔʹσʔλͷ࣋ͪํΛ͢Δ ඞཁ͕͋Δ FHCJUFNQPSBMEBUBNPEFMJOH
ૢ࡞ཤྺ࣌ࢦఆΞΫηεͷ࣮ݱ#J5FNQPSBM%BUB.PEFMͷ࣮ફ IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT
·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ #JH2VFSZXJUI %JHEBH &NCVML Ғେ ႈͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λհ
ચ͍ସ͑ ࠩߋ৽