Tweet
Tweet

Slide 1

Slide 1 text

DIBOHFMPHͱઓ͏ σʔλιϦϡʔγϣϯ෦ࢁޱ

Slide 2

Slide 2 text

ࣗݾ঺հ CONFIDENTIAL - Toreta Inc., All Right Reserved.

Slide 3

Slide 3 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ ঺ հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ΋԰ w ৬ྺɿ w *OOPWB4USVDUVSF d w %"56.456%*0 d w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊ΋ॻ͍ͯ·͢ ࣗݾ঺հ

Slide 4

Slide 4 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e n d a ࣄۀ঺հ ݱঢ়ͷσʔλΠϯϑϥ DIBOHFMPHͱͷઓ͍

Slide 5

Slide 5 text

ࣄۀ঺հ CONFIDENTIAL - Toreta Inc., All Right Reserved.

Slide 6

Slide 6 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t r o d u c t i o n ൟ੝ళΦʔφʔͨͪͷҙݟΛ΋ͱʹɺ ຊ౰ʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏໨తͰઃཱͨ͠ձࣾͰ͢

Slide 7

Slide 7 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 8

Slide 8 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 9

Slide 9 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 10

Slide 10 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 11

Slide 11 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 12

Slide 12 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 13

Slide 13 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱ৔ͷຯํ ࡾํΑ͠

Slide 14

Slide 14 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 15

Slide 15 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ

Slide 16

Slide 16 text

ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved.

Slide 17

Slide 17 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ෼ੳ༻؀ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ

Slide 18

Slide 18 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ؀ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ੒ MPDBM

Slide 19

Slide 19 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ࿥͢Δͷ͸͚ͩ͜͜ EJHEBHͰͷλεΫͷ؅ཧ

Slide 20

Slide 20 text

0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ MPDBM UBTLͷ௥Ճ͸ɺ13ϕʔεͰ 13 13 13

Slide 21

Slide 21 text

DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved.

Slide 22

Slide 22 text

0 3 . c h a n g e l o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ ଍Γͳ͍σʔλͰChangelogͷ࿩Λ͍ͯͨ͠

Slide 23

Slide 23 text

0 3 . c h a n g e l o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧ͸ɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ

Slide 24

Slide 24 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ

Slide 25

Slide 25 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷ޻਺Λऔͬͯ͠·͏ɻ

Slide 26

Slide 26 text

0 3 . c h a n g e l o g ͱ ઓ ͏ σʔλଆͷཁ๬Ͱ ຊ൪Λॏͨ͘͘͠ͳ͍

Slide 27

Slide 27 text

0 3 . c h a n g e l o g ͱ ઓ ͏ Ͳ͏͔ͨ͠

Slide 28

Slide 28 text

0 3 . c h a n g e l o g ͱ ઓ ͏ .Z42-ͷCJOMPH࢖͓͏ ͜͜Λ׆༻͠Α͏

Slide 29

Slide 29 text

0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHͱ͸ʁ

Slide 30

Slide 30 text

0 3 . c h a n g e l o g ͱ ઓ ͏ όΠφϦϩάʹ͸ɺςʔϒϧ࡞੒ૢ࡞΍ςʔϒϧσʔλ΁ͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕࢖༻͞ΕΔ৔߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ΋֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog

Slide 31

Slide 31 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͜Ε͸࢖͑ͦ͏

Slide 32

Slide 32 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ௐࠪ

Slide 33

Slide 33 text

0 3 . c h a n g e l o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRM؀ڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ

Slide 34

Slide 34 text

0 3 . c h a n g e l o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ

Slide 35

Slide 35 text

0 3 . c h a n g e l o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉ͸ׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰ΋binlogΛు͖ग़͢ TMBWFͷNZDPOG

Slide 36

Slide 36 text

0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠

Slide 37

Slide 37 text

0 3 . c h a n g e l o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang੡ CJOMPHΛKTPOʹͯ͠ు͖ग़͠

Slide 38

Slide 38 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͋ͱ͸औΓࠐΉ͚ͩʂ

Slide 39

Slide 39 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͱ͸͍͔ͳ͍

Slide 40

Slide 40 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ݸਓ৘ใ

Slide 41

Slide 41 text

0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHʹ͸ɺݸਓ৘ใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ

Slide 42

Slide 42 text

0 3 . c h a n g e l o g ͱ ઓ ͏ QBSTFͨ͠CJOMPH͸ɺ ೔໿(#

Slide 43

Slide 43 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ࠓޙαʔϏε͕੒௕͢Δ൐ͬͯɺ σʔλྔ͕૿͑Δ

Slide 44

Slide 44 text

0 3 . c h a n g e l o g ͱ ઓ ͏ ͳΔ΂͘Πϯϑϥͷ؅ཧΛͤͣʹɺ େྔͷσʔλΛߴ଎ʹॲཧ͍ͨ͠

Slide 45

Slide 45 text

0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE%BUBGMPX

Slide 46

Slide 46 text

0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ͸ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม׵֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε

Slide 47

Slide 47 text

0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ͸ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦ؀ڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ

Slide 48

Slide 48 text

0 3 . c h a n g e l o g ͱ ઓ ͏ Cloud Dataflowͱ͸ runnerΛ`DirectRunner `ʹ͢Ε͹ɺ localͰ࣮ߦͰ͖Δɻ

Slide 49

Slide 49 text

0 3 . c h a n g e l o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC

Slide 50

Slide 50 text

0 3 . c h a n g e l o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL

Slide 51

Slide 51 text

0 3 . c h a n g e l o g ͱ ઓ ͏ %BUBGMPXͰ΍͍ͬͯΔ͜ͱ ɾݸਓ৘ใͷ࡟আɺNBTL ɾి࿩൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม׵ SPXʹมߋ͕͋ͬͨߦ͢΂͕ͯೖͬͯ͘Δ

Slide 52

Slide 52 text

CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷ޻਺Λ࢖͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ଎౓͕஗͔ͬͨΒউ खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘౗ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ࢖͍ͮΒ͍ʜ ·ͱΊ

Slide 53

Slide 53 text

Next Step

Slide 54

Slide 54 text

σʔλ͸ͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ ෼ੳΛ͢Δਓ͕଍Γͳ͍ʜ Ұॹʹ෼ੳ͢ΔਓืूதͰ͢ ืू

Slide 55

Slide 55 text

ࠓͳΒͬͪ͜΋΍Ε·͢ʂ ืू

Slide 56

Slide 56 text

͓ΘΓ