changelogと戦う

#eureka_meetup

1. DIBOHFMPHͱઓ͏
   σʔλιϦϡʔγϣϯ෦
   ࢁޱ
2. ࣗݾ঺հ CONFIDENTIAL - Toreta Inc., All Right Reserved. 
3. CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ ঺ հ w ໊લɿࢁޱ
   কԝ .BTBDIJLB
   :BNBHVDIJ
   w UXJUUFSɿ!UZBLBDIJLB
   w ৬छɿσʔλؔ࿈ԿͰ΋԰
   w ৬ྺɿ
   w *OOPWB
   4USVDUVSF 
   d
   
   w %"56.
   456%*0 
   d
   
   w 5PSFUB ʙ
   
   ॳͷࣄۀձࣾ ຊ΋ॻ͍ͯ·͢ ࣗݾ঺հ
4. CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e n d a  ࣄۀ঺հ
    ݱঢ়ͷσʔλΠϯϑϥ
    DIBOHFMPHͱͷઓ͍
5. ࣄۀ঺հ CONFIDENTIAL - Toreta Inc., All Right Reserved. 
6. CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t r o d u c t i o n ൟ੝ళΦʔφʔͨͪͷҙݟΛ΋ͱʹɺ
   ຊ౰ʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ
   ͱ͍͏໨తͰઃཱͨ͠ձࣾͰ͢
7. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
8. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
9. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
10. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
11. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
12. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
13. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠
    wϑΣΞͳαʔϏε
    wҿ৯ళͷݱ৔ͷຯํ ࡾํΑ͠
14. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
15. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ࣄ ۀ ঺ հ
16. ݱঢ়ͷσʔλΠϯϑϥ CONFIDENTIAL - Toreta Inc., All Right Reserved. 
17. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ෼ੳ༻؀ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTT
    MPHܥ &YQPSU GCS #BUDI 'JMUFS
    
    
    
    &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
18. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ؀ڥߏங
    
    %FQMPZ EJHEBH4FSWFSͷߏ੒ MPDBM
19. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ
    ొ࿥͢Δͷ͸͚ͩ͜͜ EJHEBHͰͷλεΫͷ؅ཧ
20. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ MPDBM UBTLͷ௥Ճ͸ɺ13ϕʔεͰ 13 13 13
21. DIBOHFMPHͱઓ͏ CONFIDENTIAL - Toreta Inc., All Right Reserved. 
22. 0 3 . c h a n g e l o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ ଍Γͳ͍σʔλͰChangelogͷ࿩Λ͍ͯͨ͠
23. 0 3 . c h a n g e l o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ
    τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ
    ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧ͸ɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
24. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
25. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ
    αʔόʔαΠυͷ޻਺Λऔͬͯ͠·͏ɻ
26. 0 3 . c h a n g e l o g ͱ ઓ ͏ σʔλଆͷཁ๬Ͱ
    ຊ൪Λॏͨ͘͘͠ͳ͍
27. 0 3 . c h a n g e l o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
28. 0 3 . c h a n g e l o g ͱ ઓ ͏ .Z42-ͷCJOMPH࢖͓͏ ͜͜Λ׆༻͠Α͏
29. 0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHͱ͸ʁ
30. 0 3 . c h a n g e l o g ͱ ઓ ͏ όΠφϦϩάʹ͸ɺςʔϒϧ࡞੒ૢ࡞΍ςʔϒϧσʔλ΁ͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕࢖༻͞ΕΔ৔߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ΋֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
31. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͜Ε͸࢖͑ͦ͏
32. 0 3 . c h a n g e l o g ͱ ઓ ͏ ௐࠪ
33. 0 3 . c h a n g e l o g ͱ ઓ ͏ EPDLFSʹNBTUFS
    
    TMBWFͷ
    NZTRM؀ڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
34. 0 3 . c h a n g e l o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
35. 0 3 . c h a n g e l o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉ͸ׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰ΋binlogΛు͖ग़͢ TMBWFͷNZDPOG
36. 0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
37. 0 3 . c h a n g e l o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang੡ CJOMPHΛKTPOʹͯ͠ు͖ग़͠
38. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͋ͱ͸औΓࠐΉ͚ͩʂ
39. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͱ͸͍͔ͳ͍
40. 0 3 . c h a n g e l o g ͱ ઓ ͏ ݸਓ৘ใ
41. 0 3 . c h a n g e l o g ͱ ઓ ͏ CJOMPHʹ͸ɺݸਓ৘ใ͕ͷͬͨ··
    #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
42. 0 3 . c h a n g e l o g ͱ ઓ ͏ QBSTFͨ͠CJOMPH͸ɺ
    ೔໿(#
43. 0 3 . c h a n g e l o g ͱ ઓ ͏ ࠓޙαʔϏε͕੒௕͢Δ൐ͬͯɺ
    σʔλྔ͕૿͑Δ
44. 0 3 . c h a n g e l o g ͱ ઓ ͏ ͳΔ΂͘Πϯϑϥͷ؅ཧΛͤͣʹɺ
    େྔͷσʔλΛߴ଎ʹॲཧ͍ͨ͠
45. 0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE
    %BUBGMPX
46. 0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE
    %BUBGMPXͱ͸ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ
    σʔλΛม׵֦ͯ͠ॆ͢Δ
    ϑϧϚωʔδυαʔϏε
47. 0 3 . c h a n g e l o g ͱ ઓ ͏ $MPVE
    %BUBGMPXͱ͸ "QBDIF
    #FBNͰ
    ॻ͍ͨσʔλύΠϓϥΠϯͷ
    ࣮ߦ؀ڥ SVOOFS Λ
    ఏڙͯ͘͠ΕΔ
48. 0 3 . c h a n g e l o g ͱ ઓ ͏ Cloud Dataflowͱ͸ runnerΛ`DirectRunner `ʹ͢Ε͹ɺ localͰ࣮ߦͰ͖Δɻ
49. 0 3 . c h a n g e l o g ͱ ઓ ͏ "QBDIF
    #FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
50. 0 3 . c h a n g e l o g ͱ ઓ ͏ "QBDIF
    #FBNͷύΠϓϥΠϯ
    GPS
    %BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE
    5BTL
51. 0 3 . c h a n g e l o g ͱ ઓ ͏ %BUBGMPXͰ΍͍ͬͯΔ͜ͱ ɾݸਓ৘ใͷ࡟আɺNBTL
    ɾ
    ి࿩൪߸ɺࢯ໊ɺϝϞཝͳͲ
    ɾ#JH2VFSZͰऔΓճ͠ܗʹม׵ SPXʹมߋ͕͋ͬͨߦ͢΂͕ͯೖͬͯ͘Δ
52. CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷ޻਺Λ࢖͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ
    w%BUBqPXศརɻॲཧ଎౓͕஗͔ͬͨΒউ खʹBVUP4DBMJOHͯ͘͠ΕΔ
    w"QBDIF
    #FBNͷσόοΫ͕݁ߏ໘౗ष͍
    w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ࢖͍ͮΒ͍ʜ ·ͱΊ
53. Next Step
54. σʔλ͸ͲΜͲΜऔΕΔΑ͏ʹͳͬͯདྷ͕ͨ
    ෼ੳΛ͢Δਓ͕଍Γͳ͍ʜ
    Ұॹʹ෼ੳ͢ΔਓืूதͰ͢ ืू
55. ࠓͳΒͬͪ͜΋΍Ε·͢ʂ ืू
56. ͓ΘΓ