Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
changelogと戦う
Search
oprst
March 29, 2018
Programming
0
1.2k
changelogと戦う
#eureka_meetup
oprst
March 29, 2018
Tweet
Share
Other Decks in Programming
See All in Programming
TCAを用いたAmebaのリアーキテクチャ
dazy
0
210
Go 1.24でジェネリックになった型エイリアスの紹介
syumai
2
300
Domain-Driven Design (Tutorial)
hschwentner
13
22k
ソフトウェアエンジニアの成長
masuda220
PRO
12
2.1k
CDK開発におけるコーディング規約の運用
yamanashi_ren01
2
260
Serverless Rust: Your Low-Risk Entry Point to Rust in Production (and the benefits are huge)
lmammino
1
160
Kotlinの開発でも AIをいい感じに使いたい / Making the Most of AI in Kotlin Development
kohii00
5
1.4k
Rubyと自由とAIと
yotii23
6
1.8k
Amazon Bedrockマルチエージェントコラボレーションを諦めてLangGraphに入門してみた
akihisaikeda
1
120
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
920
2025.2.14_Developers Summit 2025_登壇資料
0101unite
0
210
ML.NETで始める機械学習
ymd65536
0
230
Featured
See All Featured
How to Think Like a Performance Engineer
csswizardry
22
1.4k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Become a Pro
speakerdeck
PRO
26
5.2k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
30
4.6k
Rails Girls Zürich Keynote
gr2m
94
13k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.3k
Statistics for Hackers
jakevdp
797
220k
Build The Right Thing And Hit Your Dates
maggiecrowley
34
2.5k
Transcript
DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ
ࣗݾհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ
հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊॻ͍ͯ·͢ ࣗݾհ
CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e
n d a ࣄۀհ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍
ࣄۀհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t
r o d u c t i o n ൟళΦʔφʔͨͪͷҙݟΛͱʹɺ ຊʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏తͰઃཱͨ͠ձࣾͰ͢
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱͷຯํ ࡾํΑ͠
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ੳ༻ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ MPDBM
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ͢Δͷ͚ͩ͜͜ EJHEBHͰͷλεΫͷཧ
0 2 . ݱ ঢ় ͷ σ ʔ λ Π
ϯ ϑ ϥ MPDBM UBTLͷՃɺ13ϕʔεͰ 13 13 13
DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.
0 3 . c h a n g e l
o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ Γͳ͍σʔλͰChangelogͷΛ͍ͯͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷΛऔͬͯ͠·͏ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ σʔλଆͷཁͰ ຊ൪Λॏͨ͘͘͠ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ .Z42-ͷCJOMPH͓͏ ͜͜Λ׆༻͠Α͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͱʁ
0 3 . c h a n g e l
o g ͱ ઓ ͏ όΠφϦϩάʹɺςʔϒϧ࡞ૢ࡞ςʔϒϧσʔλͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕༻͞ΕΔ߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Ε͑ͦ͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ ௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRMڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
0 3 . c h a n g e l
o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰbinlogΛు͖ग़͢ TMBWFͷNZDPOG
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang CJOMPHΛKTPOʹͯ͠ు͖ग़͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͋ͱऔΓࠐΉ͚ͩʂ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱ͍͔ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ ݸਓใ
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHʹɺݸਓใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ QBSTFͨ͠CJOMPHɺ (#
0 3 . c h a n g e l
o g ͱ ઓ ͏ ࠓޙαʔϏε͕͢Δͬͯɺ σʔλྔ͕૿͑Δ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͳΔ͘ΠϯϑϥͷཧΛͤͣʹɺ େྔͷσʔλΛߴʹॲཧ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPX
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Cloud Dataflowͱ runnerΛ`DirectRunner `ʹ͢Εɺ localͰ࣮ߦͰ͖Δɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
0 3 . c h a n g e l
o g ͱ ઓ ͏ %BUBGMPXͰ͍ͬͯΔ͜ͱ ɾݸਓใͷআɺNBTL ɾి൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม SPXʹมߋ͕͋ͬͨߦ͕ͯ͢ೖͬͯ͘Δ
CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷΛ͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ͕͔ͬͨΒউ
खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ͍ͮΒ͍ʜ ·ͱΊ
Next Step
σʔλͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ੳΛ͢Δਓ͕Γͳ͍ʜ Ұॹʹੳ͢ΔਓืूதͰ͢ ืू
ࠓͳΒͬͪ͜Ε·͢ʂ ืू
͓ΘΓ