Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
changelogと戦う
Search
oprst
March 29, 2018
Programming
0
1.3k
changelogと戦う
#eureka_meetup
oprst
March 29, 2018
Tweet
Share
More Decks by oprst
See All by oprst
LLMを浸透させるための泥臭い話
oprstchn
1
160
AI導入の理想と現実~コストと浸透〜
oprstchn
1
530
Other Decks in Programming
See All in Programming
Navigating Dependency Injection with Metro
zacsweers
3
5.7k
AndroidXR向けにアプリを作るには
kotambourine
0
150
デザイナーが Androidエンジニアに 挑戦してみた
874wokiite
0
650
Model Pollution
hschwentner
1
170
Web技術を最大限活用してRAW画像を現像する / Developing RAW Images on the Web
ssssota
2
500
さようなら Date。 ようこそTemporal! 3年間先行利用して得られた知見の共有
8beeeaaat
3
1.5k
Android StudioのAIコーディングツール、 ぶっちゃけどうなん???
mkeeda
0
170
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
290
基礎から学ぶ大画面対応(Learning Large-Screen Support from the Ground Up)
tomoya0x00
0
6.7k
意外と簡単!?フロントエンドでパスキー認証を実現する WebAuthn
teamlab
PRO
2
820
iOSからFlutterへ20代エンジニアの転身とWidget Preview実践
nappannda
0
110
Playwrightはどのようにクロスブラウザをサポートしているのか
yotahada3
7
1.9k
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
4k
Navigating Team Friction
lara
189
15k
RailsConf 2023
tenderlove
30
1.2k
Making Projects Easy
brettharned
118
6.4k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
4 Signs Your Business is Dying
shpigford
185
22k
Designing for Performance
lara
610
69k
Docker and Python
trallard
46
3.6k
The World Runs on Bad Software
bkeepers
PRO
71
11k
Agile that works and the tools we love
rasmusluckow
330
21k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3k
Producing Creativity
orderedlist
PRO
347
40k
Transcript
DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ
ࣗݾհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ
հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊॻ͍ͯ·͢ ࣗݾհ
CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e
n d a ࣄۀհ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍
ࣄۀհ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t
r o d u c t i o n ൟళΦʔφʔͨͪͷҙݟΛͱʹɺ ຊʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏తͰઃཱͨ͠ձࣾͰ͢
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱͷຯํ ࡾํΑ͠
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ࣄ ۀ հ
ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ੳ༻ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ MPDBM
CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .
ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ͢Δͷ͚ͩ͜͜ EJHEBHͰͷλεΫͷཧ
0 2 . ݱ ঢ় ͷ σ ʔ λ Π
ϯ ϑ ϥ MPDBM UBTLͷՃɺ13ϕʔεͰ 13 13 13
DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.
0 3 . c h a n g e l
o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ Γͳ͍σʔλͰChangelogͷΛ͍ͯͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷΛऔͬͯ͠·͏ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ σʔλଆͷཁͰ ຊ൪Λॏͨ͘͘͠ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ .Z42-ͷCJOMPH͓͏ ͜͜Λ׆༻͠Α͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͱʁ
0 3 . c h a n g e l
o g ͱ ઓ ͏ όΠφϦϩάʹɺςʔϒϧ࡞ૢ࡞ςʔϒϧσʔλͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕༻͞ΕΔ߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͜Ε͑ͦ͏
0 3 . c h a n g e l
o g ͱ ઓ ͏ ௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRMڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
0 3 . c h a n g e l
o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰbinlogΛు͖ग़͢ TMBWFͷNZDPOG
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang CJOMPHΛKTPOʹͯ͠ు͖ग़͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͋ͱऔΓࠐΉ͚ͩʂ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͱ͍͔ͳ͍
0 3 . c h a n g e l
o g ͱ ઓ ͏ ݸਓใ
0 3 . c h a n g e l
o g ͱ ઓ ͏ CJOMPHʹɺݸਓใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ QBSTFͨ͠CJOMPHɺ (#
0 3 . c h a n g e l
o g ͱ ઓ ͏ ࠓޙαʔϏε͕͢Δͬͯɺ σʔλྔ͕૿͑Δ
0 3 . c h a n g e l
o g ͱ ઓ ͏ ͳΔ͘ΠϯϑϥͷཧΛͤͣʹɺ େྔͷσʔλΛߴʹॲཧ͍ͨ͠
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPX
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
0 3 . c h a n g e l
o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
0 3 . c h a n g e l
o g ͱ ઓ ͏ Cloud Dataflowͱ runnerΛ`DirectRunner `ʹ͢Εɺ localͰ࣮ߦͰ͖Δɻ
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
0 3 . c h a n g e l
o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
0 3 . c h a n g e l
o g ͱ ઓ ͏ %BUBGMPXͰ͍ͬͯΔ͜ͱ ɾݸਓใͷআɺNBTL ɾి൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม SPXʹมߋ͕͋ͬͨߦ͕ͯ͢ೖͬͯ͘Δ
CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷΛ͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ͕͔ͬͨΒউ
खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ͍ͮΒ͍ʜ ·ͱΊ
Next Step
σʔλͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ੳΛ͢Δਓ͕Γͳ͍ʜ Ұॹʹੳ͢ΔਓืूதͰ͢ ืू
ࠓͳΒͬͪ͜Ε·͢ʂ ืू
͓ΘΓ