Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
changelogと戦う
Search
oprst
March 29, 2018
Programming
0
1.2k
changelogと戦う
#eureka_meetup
oprst
March 29, 2018
Tweet
Share
Other Decks in Programming
See All in Programming
Djangoアプリケーション 運用のリアル 〜問題発生から可視化、最適化への道〜 #pyconshizu
kashewnuts
1
260
Boos Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
280
Datadog Workflow Automation で圧倒的価値提供
showwin
1
160
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
1k
Jasprが凄い話
hyshu
0
150
Amazon Q Developer Proで効率化するAPI開発入門
seike460
PRO
0
120
DRFを少しずつ オニオンアーキテクチャに寄せていく DjangoCongress JP 2025
nealle
2
260
5分で理解する SOLID 原則 #phpcon_nagoya
shogogg
1
300
機能が複雑化しても 頼りになる FactoryBotの話
tamikof
0
110
2025.2.14_Developers Summit 2025_登壇資料
0101unite
0
180
Jakarta EE meets AI
ivargrimstad
0
360
Rails 1.0 のコードで学ぶ find_by* と method_missing の仕組み / Learn how find_by_* and method_missing work in Rails 1.0 code
maimux2x
1
170
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
GraphQLの誤解/rethinking-graphql
sonatard
68
10k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
Music & Morning Musume
bryan
46
6.4k
RailsConf 2023
tenderlove
29
1k
Designing for humans not robots
tammielis
250
25k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
50k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
Transcript
DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ
ࣗݾհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ
հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊॻ͍ͯ·͢ ࣗݾհ
CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e
n d a ࣄۀհ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍
ࣄۀհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t
r o d u c t i o n ൟళΦʔφʔͨͪͷҙݟΛͱʹɺ ຊʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏తͰઃཱͨ͠ձࣾͰ͢
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱͷຯํ ࡾํΑ͠
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ੳ༻ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ MPDBM
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ͢Δͷ͚ͩ͜͜ EJHEBHͰͷλεΫͷཧ
0 2 . ݱ ঢ় ͷ σ ʔ λ Π
ϯ ϑ ϥ MPDBM UBTLͷՃɺ13ϕʔεͰ 13 13 13
DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.
0 3 . c h a n g e l
o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ Γͳ͍σʔλͰChangelogͷΛ͍ͯͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷΛऔͬͯ͠·͏ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ σʔλଆͷཁͰ ຊ൪Λॏͨ͘͘͠ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ .Z42-ͷCJOMPH͓͏ ͜͜Λ׆༻͠Α͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͱʁ
0 3 . c h a n g e l
o g ͱ ઓ ͏ όΠφϦϩάʹɺςʔϒϧ࡞ૢ࡞ςʔϒϧσʔλͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕༻͞ΕΔ߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Ε͑ͦ͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ ௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRMڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
0 3 . c h a n g e l
o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰbinlogΛు͖ग़͢ TMBWFͷNZDPOG
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang CJOMPHΛKTPOʹͯ͠ు͖ग़͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͋ͱऔΓࠐΉ͚ͩʂ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱ͍͔ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ ݸਓใ
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHʹɺݸਓใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ QBSTFͨ͠CJOMPHɺ (#
0 3 . c h a n g e l
o g ͱ ઓ ͏ ࠓޙαʔϏε͕͢Δͬͯɺ σʔλྔ͕૿͑Δ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͳΔ͘ΠϯϑϥͷཧΛͤͣʹɺ େྔͷσʔλΛߴʹॲཧ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPX
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Cloud Dataflowͱ runnerΛ`DirectRunner `ʹ͢Εɺ localͰ࣮ߦͰ͖Δɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
0 3 . c h a n g e l
o g ͱ ઓ ͏ %BUBGMPXͰ͍ͬͯΔ͜ͱ ɾݸਓใͷআɺNBTL ɾి൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม SPXʹมߋ͕͋ͬͨߦ͕ͯ͢ೖͬͯ͘Δ
CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷΛ͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ͕͔ͬͨΒউ
खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ͍ͮΒ͍ʜ ·ͱΊ
Next Step
σʔλͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ੳΛ͢Δਓ͕Γͳ͍ʜ Ұॹʹੳ͢ΔਓืूதͰ͢ ืू
ࠓͳΒͬͪ͜Ε·͢ʂ ืू
͓ΘΓ