Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
changelogと戦う
Search
oprst
March 29, 2018
Programming
0
1.3k
changelogと戦う
#eureka_meetup
oprst
March 29, 2018
Tweet
Share
More Decks by oprst
See All by oprst
AI導入の理想と現実~コストと浸透〜
oprstchn
0
96
Other Decks in Programming
See All in Programming
AIプログラマーDevinは PHPerの夢を見るか?
shinyasaita
1
170
Deep Dive into ~/.claude/projects
hiragram
9
1.6k
5つのアンチパターンから学ぶLT設計
narihara
1
120
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
430
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
460
NPOでのDevinの活用
codeforeveryone
0
420
Go1.25からのGOMAXPROCS
kuro_kurorrr
1
810
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
1k
エラーって何種類あるの?
kajitack
5
310
AIエージェントはこう育てる - GitHub Copilot Agentとチームの共進化サイクル
koboriakira
0
450
Select API from Kotlin Coroutine
jmatsu
1
190
ふつうの技術スタックでアート作品を作ってみる
akira888
0
150
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
42
7.5k
Into the Great Unknown - MozCon
thekraken
39
1.9k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
RailsConf 2023
tenderlove
30
1.1k
Gamification - CAS2011
davidbonilla
81
5.3k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
490
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
We Have a Design System, Now What?
morganepeng
53
7.7k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Being A Developer After 40
akosma
90
590k
Transcript
DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ
ࣗݾհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ
հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊॻ͍ͯ·͢ ࣗݾհ
CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e
n d a ࣄۀհ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍
ࣄۀհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t
r o d u c t i o n ൟళΦʔφʔͨͪͷҙݟΛͱʹɺ ຊʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏తͰઃཱͨ͠ձࣾͰ͢
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱͷຯํ ࡾํΑ͠
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ੳ༻ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ MPDBM
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ͢Δͷ͚ͩ͜͜ EJHEBHͰͷλεΫͷཧ
0 2 . ݱ ঢ় ͷ σ ʔ λ Π
ϯ ϑ ϥ MPDBM UBTLͷՃɺ13ϕʔεͰ 13 13 13
DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.
0 3 . c h a n g e l
o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ Γͳ͍σʔλͰChangelogͷΛ͍ͯͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷΛऔͬͯ͠·͏ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ σʔλଆͷཁͰ ຊ൪Λॏͨ͘͘͠ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ .Z42-ͷCJOMPH͓͏ ͜͜Λ׆༻͠Α͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͱʁ
0 3 . c h a n g e l
o g ͱ ઓ ͏ όΠφϦϩάʹɺςʔϒϧ࡞ૢ࡞ςʔϒϧσʔλͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕༻͞ΕΔ߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Ε͑ͦ͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ ௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRMڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
0 3 . c h a n g e l
o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰbinlogΛు͖ग़͢ TMBWFͷNZDPOG
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang CJOMPHΛKTPOʹͯ͠ు͖ग़͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͋ͱऔΓࠐΉ͚ͩʂ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱ͍͔ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ ݸਓใ
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHʹɺݸਓใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ QBSTFͨ͠CJOMPHɺ (#
0 3 . c h a n g e l
o g ͱ ઓ ͏ ࠓޙαʔϏε͕͢Δͬͯɺ σʔλྔ͕૿͑Δ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͳΔ͘ΠϯϑϥͷཧΛͤͣʹɺ େྔͷσʔλΛߴʹॲཧ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPX
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Cloud Dataflowͱ runnerΛ`DirectRunner `ʹ͢Εɺ localͰ࣮ߦͰ͖Δɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
0 3 . c h a n g e l
o g ͱ ઓ ͏ %BUBGMPXͰ͍ͬͯΔ͜ͱ ɾݸਓใͷআɺNBTL ɾి൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม SPXʹมߋ͕͋ͬͨߦ͕ͯ͢ೖͬͯ͘Δ
CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷΛ͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ͕͔ͬͨΒউ
खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ͍ͮΒ͍ʜ ·ͱΊ
Next Step
σʔλͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ੳΛ͢Δਓ͕Γͳ͍ʜ Ұॹʹੳ͢ΔਓืूதͰ͢ ืू
ࠓͳΒͬͪ͜Ε·͢ʂ ืू
͓ΘΓ