Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
冪等性を考慮したデータ連携ジョブの設計
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
hsasakawa
April 12, 2019
Technology
6
2.1k
冪等性を考慮したデータ連携ジョブの設計
hsasakawa
April 12, 2019
Tweet
Share
More Decks by hsasakawa
See All by hsasakawa
Suffix Trees and Suffix Arrays
hsasakawa
0
130
行動ログ処理基盤の構築
hsasakawa
0
3.4k
Data platform development on M3 USA
hsasakawa
0
1.2k
Data Analysis Platform Development @M3, inc.
hsasakawa
0
3.5k
Other Decks in Technology
See All in Technology
【Oracle Cloud ウェビナー】[Oracle AI Database + AWS] Oracle Database@AWSで広がるクラウドの新たな選択肢とAI時代のデータ戦略
oracle4engineer
PRO
2
200
顧客の言葉を、そのまま信じない勇気
yamatai1212
1
370
旅先で iPad + Neovim で iOS 開発・執筆した話
zozotech
PRO
0
120
配列に見る bash と zsh の違い
kazzpapa3
3
180
30分でわかる「ネットワーク図の描き方入門」/infraengbooks56
corestate55
0
140
Cosmos World Foundation Model Platform for Physical AI
takmin
0
1k
マネージャー視点で考えるプロダクトエンジニアの評価 / Evaluating Product Engineers from a Manager's Perspective
hiro_torii
0
220
We Built for Predictability; The Workloads Didn’t Care
stahnma
0
150
登壇駆動学習のすすめ — CfPのネタの見つけ方と書くときに意識していること
bicstone
3
140
20260208_第66回 コンピュータビジョン勉強会
keiichiito1978
0
220
Red Hat OpenStack Services on OpenShift
tamemiya
0
150
Context Engineeringの取り組み
nutslove
0
430
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
51
51k
GraphQLの誤解/rethinking-graphql
sonatard
74
11k
Optimising Largest Contentful Paint
csswizardry
37
3.6k
jQuery: Nuts, Bolts and Bling
dougneiner
65
8.4k
Large-scale JavaScript Application Architecture
addyosmani
515
110k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
uxyall
0
430
Imperfection Machines: The Place of Print at Facebook
scottboms
269
14k
Reality Check: Gamification 10 Years Later
codingconduct
0
2k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
tmiket
1
110
技術選定の審美眼(2025年版) / Understanding the Spiral of Technologies 2025 edition
twada
PRO
117
110k
The innovator’s Mindset - Leading Through an Era of Exponential Change - McGill University 2025
jdejongh
PRO
1
99
The B2B funnel & how to create a winning content strategy
katarinadahlin
PRO
1
280
Transcript
ႈੑΛߟྀͨ͠ σʔλ࿈ܞδϣϒͷઃܭ Hirohito Sasakawa data engineer at M3, inc.
ΞδΣϯμ • ΤϜεϦʔͷσʔλج൫ͷߏ • σʔλ࿈ܞδϣϒͷߏ • σʔλ࿈ܞͱႈੑ • ચ͍ସ͑ •
ࠩߋ৽ • ·ͱΊ
M3ͷσʔλج൫ͷߏ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 4ZOD
2VFSZ #JH2VFSZ
M3ͷσʔλج൫ͷߏ 2VFSZ 2VFSZ 2VFSZ 1SPEVDUJPO … 3FQPSU … 4ZOD 2VFSZ
4ZOD #JH2VFSZ
σʔλ࿈ܞδϣϒͷߏ "QQMJDBUJPO %#T #JH2VFSZ %JHEBH &NCVML %JHEBHͱ&NCVMLͷར༻ ར༻࣮ଟ
ϫʔΫϑϩʔɺσʔλநग़ͷ࣮͕؆୯ σʔλιʔεʹԠͨ͡ϓϥάΠϯػߏ .64"ग़ுه.64"Ͱσʔλج൫ڥඋͱ,1*ϞχλϦϯ άπʔϧͷಋೖΛ࣮ࢪ͠·ͨ͠ IUUQTXXXNUFDICMPHFOUSZNVTBEFWFMPQNFOUEBUBQMBUGPSN
Embulkͷ࣮ ೖྗσʔλʹ͍ͭͯͷઃఆ %#ଓใҎ֎ ΄΅42- ग़ྗσʔλʹ͍ͭͯͷઃఆ
BigQuery্ͷσʔλ "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠ σʔλ
ػցֶशϓϩμΫτ͔Βͷσʔλऔಘ "QQMJDBUJPO %#T 4PVSDF .BSU NEPXOMPBEFS .-1SPEVDU #2ʹΞΫηεͯ͠ EBUBGSBNFܗࣜͰ ΠΠײ͡ʹσʔλΛ
औಘ͢Δ 1ZUIPOQBDLBHF
σʔλ࿈ܞδϣϒͷߏ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠
σʔλ
σʔλ࿈ܞδϣϒͷߏ (࠶ܝ) "QQMJDBUJPO %#T 4PVSDF ݩςʔϒϧͷ ίϐʔ .BSU සग़͢ΔՃΛ ༧Ίద༻ͨ͠
σʔλ ͜͜ͱ ͜͜Λ ָʹ͍ͨ͠
δϣϒͷࣦഊͱϦΧόϦ δϣϒ͍ΖΜͳཧ༝Ͱࣦഊ͢Δ σʔλߋ৽ͷԆ σʔλ૿Ճ 4BB4ଆͷෆ۩߹ͳͲͳͲ ϦΧόϦδϣϒͷ࠶࣮ߦ͚ͩͰࡁ·͍ͤͨ
ΦϖϨʔγϣϯϛεͷࢭ ్த·ͰՃ͞ΕͨσʔλΛআͯ͠ʜ☠ ႈͳσʔλ࿈ܞδϣϒΛ࣮͍ͨ͠
σʔλ࿈ܞύλʔϯ1: ચ͍ସ͑ "QQMJDBUJPO %#T ຖճσʔλΛ্ॻ͖͢ΔPSผςʔϒϧʹ͢Δ ΄ͱΜͲͷ߹͜ΕͰ͍͍ ႈͳͷͰϦΧόϦ୯७ʹ࠶࣮ߦͰ0,
ศརʂ
σʔλ࿈ܞύλʔϯ2: ࠩߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩͷΈΛ࿈ܞ σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ ୯७ʹ࣮͢ΔͱႈʹͳΒͳ͍
δϣϒࣦഊ࣌ͷϦΧόϦେม ೖͬͯ͠·ͬͨσʔλΛಛఆআͯ͠Γ͢ͳͲ
σʔλ࿈ܞύλʔϯ2: ࠩߋ৽ "QQMJDBUJPO %#T લճ࿈ܞͱͷࠩͷΈΛ࿈ܞ σʔλαΠζ͕େ͖͍ςʔϒϧͰඞཁʹͳΔ ୯७ʹ࣮͢ΔͱႈʹͳΒͳ͍
δϣϒࣦഊ࣌ͷϦΧόϦେม ೖͬͯ͠·ͬͨσʔλΛಛఆআͯ͠Γ͢ͳͲ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ
ႈͳࠩߋ৽ (1/2) "QQMJDBUJPO %#T લճͷςʔϒϧ લճͷςʔϒϧ Λίϐʔ 4UFQ લճ͔Βͷ ࠩσʔλΛநग़
4UFQ
ႈͳࠩߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞ 4UFQ
ႈͳࠩߋ৽ (2/2) "QQMJDBUJPO %#T લճͷςʔϒϧ ྆ऀΛ݁߹ͨ͠ ςʔϒϧΛ࡞ 4UFQ 4UFQ͔Β࠶࣮ߦͰ0, ඞཁ͕͋ΕɺͲͷϑΣʔζ͔Β
ͰΓͤΔ ϦΧόϦ
ࣦഊ࣌ͷϦΧόϦ ϦΧόϦ࠶࣮ߦͰ0, ඞཁʹԠͯ͡ҙͷ4UFQ͔Βͷ࠶࣮ߦ͕Մೳ 4UFQ·Ͱਖ਼ৗऴ͍ྃͯ͠ΔͷͰɺ4UFQ͔Β࠶࣮ߦͳͲ ͖ͬ͞ௐͯͨΒҎԼͷࢿྉͰղઆ͞Ε͍ͯ·ͨ͠ʜ ;0;0508/ͷ%8)Λ3FETIJGU͔Β#JH2VFSZʹ ͓Ҿӽͨ͠͠
IUUQTTQFBLFSEFDLDPNTIJP[BLJNPWJOH[P[PUPXOEXIGSPNSFETIJGUUPCJHRVFSZ
ҙ ͜ͷख๏ͷҙࣄ߲ ͦͦσʔλ͕ଟ͗ͯ͢ɺ͜ͷख๏ΛͱΔͷͰաڈͷ ςʔϒϧΛอ͓࣋ͯ͘͠ͷݱ࣮తͰͳ͍ աڈͷ͋Δ࣌ʹΔʹσʔλͷ࣋ͪํΛ͢Δ ඞཁ͕͋Δ FHCJUFNQPSBMEBUBNPEFMJOH
ૢ࡞ཤྺ࣌ࢦఆΞΫηεͷ࣮ݱ#J5FNQPSBM%BUB.PEFMͷ࣮ફ IUUQTTQFBLFSEFDLDPNGJNQMFNFOUJOHDPNNBOEIJTUPSZBOEUFNQPSBMBDDFTT
·ͱΊ ΤϜεϦʔͷσʔλج൫ͱ࿈ܞδϣϒʹ͍ͭͯফԽ #JH2VFSZXJUI %JHEBH &NCVML Ғେ ႈͳσʔλ࿈ܞδϣϒͷઃܭํ๏Λհ
ચ͍ସ͑ ࠩߋ৽