Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
changelogと戦う
Search
oprst
March 29, 2018
Programming
0
1.3k
changelogと戦う
#eureka_meetup
oprst
March 29, 2018
Tweet
Share
More Decks by oprst
See All by oprst
AI導入の理想と現実~コストと浸透〜
oprstchn
0
220
Other Decks in Programming
See All in Programming
商品比較サービス「マイベスト」における パーソナライズレコメンドの第一歩
ucchiii43
0
150
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
200
NPOでのDevinの活用
codeforeveryone
0
890
AIエージェントはこう育てる - GitHub Copilot Agentとチームの共進化サイクル
koboriakira
0
720
状態遷移図を書こう / Sequence Chart vs State Diagram
orgachem
PRO
2
200
A full stack side project webapp all in Kotlin (KotlinConf 2025)
dankim
0
140
TypeScriptでDXを上げろ! Hono編
yusukebe
3
730
効率的な開発手段として VRTを活用する
ishkawa
0
160
Agentic Coding: The Future of Software Development with Agents
mitsuhiko
0
120
LT 2025-06-30: プロダクトエンジニアの役割
yamamotok
0
850
『自分のデータだけ見せたい!』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
akitotsukahara
2
660
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
290
Featured
See All Featured
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
A designer walks into a library…
pauljervisheath
207
24k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
The World Runs on Bad Software
bkeepers
PRO
70
11k
We Have a Design System, Now What?
morganepeng
53
7.7k
What's in a price? How to price your products and services
michaelherold
246
12k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Adopting Sorbet at Scale
ufuk
77
9.5k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Optimizing for Happiness
mojombo
379
70k
Transcript
DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ
ࣗݾհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ
հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊॻ͍ͯ·͢ ࣗݾհ
CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e
n d a ࣄۀհ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍
ࣄۀհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t
r o d u c t i o n ൟళΦʔφʔͨͪͷҙݟΛͱʹɺ ຊʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏తͰઃཱͨ͠ձࣾͰ͢
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱͷຯํ ࡾํΑ͠
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ੳ༻ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ MPDBM
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ͢Δͷ͚ͩ͜͜ EJHEBHͰͷλεΫͷཧ
0 2 . ݱ ঢ় ͷ σ ʔ λ Π
ϯ ϑ ϥ MPDBM UBTLͷՃɺ13ϕʔεͰ 13 13 13
DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.
0 3 . c h a n g e l
o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ Γͳ͍σʔλͰChangelogͷΛ͍ͯͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷΛऔͬͯ͠·͏ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ σʔλଆͷཁͰ ຊ൪Λॏͨ͘͘͠ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ .Z42-ͷCJOMPH͓͏ ͜͜Λ׆༻͠Α͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͱʁ
0 3 . c h a n g e l
o g ͱ ઓ ͏ όΠφϦϩάʹɺςʔϒϧ࡞ૢ࡞ςʔϒϧσʔλͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕༻͞ΕΔ߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Ε͑ͦ͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ ௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRMڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
0 3 . c h a n g e l
o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰbinlogΛు͖ग़͢ TMBWFͷNZDPOG
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang CJOMPHΛKTPOʹͯ͠ు͖ग़͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͋ͱऔΓࠐΉ͚ͩʂ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱ͍͔ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ ݸਓใ
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHʹɺݸਓใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ QBSTFͨ͠CJOMPHɺ (#
0 3 . c h a n g e l
o g ͱ ઓ ͏ ࠓޙαʔϏε͕͢Δͬͯɺ σʔλྔ͕૿͑Δ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͳΔ͘ΠϯϑϥͷཧΛͤͣʹɺ େྔͷσʔλΛߴʹॲཧ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPX
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Cloud Dataflowͱ runnerΛ`DirectRunner `ʹ͢Εɺ localͰ࣮ߦͰ͖Δɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
0 3 . c h a n g e l
o g ͱ ઓ ͏ %BUBGMPXͰ͍ͬͯΔ͜ͱ ɾݸਓใͷআɺNBTL ɾి൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม SPXʹมߋ͕͋ͬͨߦ͕ͯ͢ೖͬͯ͘Δ
CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷΛ͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ͕͔ͬͨΒউ
खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ͍ͮΒ͍ʜ ·ͱΊ
Next Step
σʔλͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ੳΛ͢Δਓ͕Γͳ͍ʜ Ұॹʹੳ͢ΔਓืूதͰ͢ ืू
ࠓͳΒͬͪ͜Ε·͢ʂ ืू
͓ΘΓ