대용량 데이터베이스 동기화를 위한 최적의 CDC 시스템 구축기

$%$ %FCF[JVN ౵੉೐ۄੋҳઑ ੿೤ࢿѨࢎ ਍৔दߊࢤೠ੉ग৬೧Ѿҗ੿ *DFCFSHبੑ 2"

$IBOHF%BUB$BQUVSF ؘ੉ఠ߬੉झীࢲߊࢤೞח߸҃ࢎ೦ ࢗੑ ࣻ੿ ࢏ઁ ਸपदрਵ۽୶੸ೞҊӝ۾ೞחӝࣿ

$%$੄ઙܨ ௪ܻӝ߈ ۽Ӓӝ߈ ઱ӝ੸ਵ۽௪ܻܳप೯ೞৈ߸҃ࢎ೦୶୹ ౟ے੥࣌۽Ӓܳपदрਵ۽੍ਵݴ߸҃ࢎ೦୶୹

पदр߸҃࠙ӝ߈੉࢚஖ఐ૑ঌܿ ؘ੉ఠ߬੉झ݃੉Ӓۨ੉࣌ ৈ۞૑৉рীؘ੉ఠزӝച நदޖബച ؘ੉ఠ߬੉झਊبܻ࠙
ੌ߈੸ੋ$%$ࢎਊࢎ۹

਋ܻח t੹ࢎؘ੉ఠઑ૒ਵ۽ࢲ౵ಞചػৈ۞ؘ੉ఠ߬੉झղؘ੉ఠܳೠҔਵ۽਽૘u

$%$౵੉೐ۄੋ 8BSFIPVTF

Ӓ۞աj ޹хؘ੉ఠ ୡ؀ӏݽؘ੉ఠ ੿೤ࢿഛੋ ੸੤दझమ

о੢؀઺੸ੋ౟ے੥࣌۽Ӓӝ߈$%$য়೑ࣗझ ੌ߈੸ਵ۽஠೐஠ழ֏౟ӝ߈ਵ۽ز੘ .Z42- 1PTUHSF42- .POHP%#١׮নೠ%#.4૑ਗ पदрਵ۽য়೐ࣇӝ۾
੤োزद݃૑݄ਵ۽੍਷য়೐ࣇद੼ীࢲࠂҳоמ ૐ࠙झշࢫ *ODSFNFOUBM4OBQTIPU ӝמ૑ਗ %FCF[JVN

झշࢫ %.- 4PVSDF झշࢫ 5BSHFU
߄੉ցܻ۽Ӓ ܻబ࣌ઓ੤

ૐ࠙झշࢫ 4PVSDF 5BSHFU
߄੉ցܻ۽Ӓ

౟ے੥࣌۽Ӓ ࣻ૘ 5BSHFU
߈৔ 4PVSDF ߄੉ցܻ۽Ӓ ౠ੿ష೗ী   য়೐ࣇ (5*%T ੷੢

4PVSDF ౟ے੥࣌۽Ӓ */4&35*/50DVTUPNFST7"-6&4 t5POZ4UBSLu { "schema": { "type": "struct",
"fields": [ ... ], "optional": false, "name": "mysql-server-1.inventory.customers.Envelope" }, "payload": { "op": "c", "ts_ms": 1465491411815, "before": null, "after": { ... }, "source": { "db": "inventory", "table": "customers", ... } } } $IBOHF&WFOU

౟ے੥࣌۽Ӓ 5BSHFU ,BGLB { "schema": { "type": "struct", "fields": [
... ], "optional": false, "name": "mysql-server-1.inventory.customers.Envelope" }, "payload": { "op": "c", "ts_ms": 1465491411815, "before": null, "after": { ... }, "source": { "db": "inventory", "table": "customers", ... } } } $IBOHF&WFOU

%FCF[JVN౵੉೐ۄੋ ,BGLB 1PTUHSF42- .Z42- ,BGLB$POOFDU   4PVSDF$POOFDUPS %FCF[JVN   .Z42-
%FCF[JVN   1PTUHSF42- &4$POOFDUPS +%#$   $POOFDUPS &MBTUJD4FBSDI 8BSFIPVTF ,BGLB$POOFDU   4JOL$POOFDUPS

.Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH ౵੉೐ۄੋҳઑ .Z42- ,BGLB

ೞܖডরѐۨ௏٘୊ܻ োزప੉࠶ѐ োزࢎ੉ૉ5#

.Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH 'MJOL$%$ .Z42- ,BGLB

'MJOL$%$ 'MJOL $%$ %FCF[JVNӝ߈$%$য়೑ࣗझ ೒݂௼੄ౠ૚ ࠙࢑୊ܻ ୓௼ನੋ౟ ؘ੉ఠझ౟ܿ"1*١
൚ࣻ ૐ࠙झշࢫ੄߽۳ࣻ೯૑ਗ

ૐ࠙झշࢫ ߽۳ 5BSHFU
4PVSDF

୓௼ನੋ౟੷੢ 4PVSDF 5BSHFU
߄੉ցܻ۽Ӓ ࣻ૘ ߈৔ )%'4 ୓௼ನੋ౟ী   য়೐ࣇ (5*%T ੷੢

޹хؘ੉ఠ୊ܻ 'MJOL$%$ ,BGLB NBQ ೧यژח݃झఊ { "phone_number": "010-0000-0000", "email": "[email protected]",
... } { "phone_number": "-", "email": “[email protected]", ... }

'MJOL$%$بੑਵ۽঳਷੉੼ झշࢫࢿמ؀಩ೱ࢚ %FCF[JVN݅ਵ۽חࠛоמ೮؍ӝ؀ࢿמب׳ ߄੉ցܻ۽Ӓद੘য়೐ࣇ૒੽૑੿оמ TDBOTUBSUVQNPEF২࣌ਵ۽FBSMJFTU TQFDJ
fi D UJNFTUBNQ١द੘ਤ஖ਬোೞѱ૑੿ ੢গ୊ܻബਯࢿೱ࢚ ੢গद୓௼ನੋ౟ഝਊೞৈ੗زਵ۽੉੹࢚కࠂҳ ੤दب੹ۚҳ୓੸ਵ۽ࢸ੿ fi YFE - EFMBZ FYQPOFOUJBM - EFMBZ

'MJOL$%$ݽפఠ݂ ੟ѐࣻ߂୊ܻਯ ੟߹୓௼ನੋ౟ࣗਃदр ੟߹୊ܻਯ Ӓې೐

.Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH ,BGLB$POOFDU .Z42- ,BGLB

஠೐஠৬োزغחؘ੉ఠ୊ܻ೐ۨ੐ਕ௼ ೒۞Ӓੋইఃఫ୊۽ழ֏ఠܳ೒۞Ӓੋߑधਵ۽୶о ࠙࢑दझమਵ۽ਕழܳ୶оೞৈؘ੉ఠ୊ܻࢿמࣻಣ੸ഛ੢оמ ѐ߹ݫद૑ؘ੉ఠ߸ജ૑ਗ 4.5
3&45"1*ܳా೧औѱҙܻ߂+.9ܳాೠ਍৔૑಴ઁҕ ౠ૚

+%#$4JOL$POOFDUPS ஠೐஠੄ؘ੉ఠܳ+%#$٘ۄ੉ߡܳ૑ਗೞחҙ҅ഋؘ੉ఠ߬੉झ۽पदр੸੤

+%#$4JOL$POOFDUPS ஠೐஠੄ؘ੉ఠܳ+%#$٘ۄ੉ߡܳ૑ਗೞחҙ҅ഋؘ੉ఠ߬੉झ۽पदр੸੤ .Z42- 'MJOL$%$ ,BGLB$POOFDU .Z42- ,BGLB

+%#$4JOL$POOFDUPS +%#$4JOL $POOFDUPS .Z42- 'MJOL$%$ ,BGLB$POOFDU .Z42- ,BGLB ஠೐஠੄ؘ੉ఠܳ+%#$٘ۄ੉ߡܳ૑ਗೞחҙ҅ഋؘ੉ఠ߬੉झ۽पदр੸੤

+%#$4JOL$POOFDUPS ஠೐஠੄ؘ੉ఠܳ+%#$٘ۄ੉ߡܳ૑ਗೞחҙ҅ഋؘ੉ఠ߬੉झ۽पदр੸੤ +%#$4JOL $POOFDUPS .Z42- 'MJOL$%$ ,BGLB$POOFDU .Z42- ,BGLB

઱ਃࢸ੿ ݵ١ࢿ VQTFSUݽ٘ࢎਊೞৈэ਷1,ܳо૓ۨ௏٘ী؀೧ݵ١ࢿࠁ੢ ؘ੉ఠ઺ࠂ੸੤ߑ૑ ߽۳୊ܻ
UBTLTNBY২࣌ࢎਊೞৈ߽۳ࢗੑ ҃೷੸ਵ۽஠೐஠ష೗౵౭࣌ѐࣻ੄ীࢲ੿ب۽ࢸ੿ೞৈࢎਊ ߥ௼ੋࢲ౟ SFXSJUF#BUDI*OTFSUT CBUDITJ[F২࣌ࢎਊೞৈୡ׼୊ܻਯೱ࢚ ఋѶؘ੉ఠ߬੉झীऺ੉ח۽Ӓࢎ੉ૉ؀಩хࣗ

4.5 4JOHMF.FTTBHF5SBOTGPSN ѐ߹ݫद૑ী߸ജ۽૒ਸ੸ਊೡٸࢎਊغח߸ജӝמ ࣗझژחय௼ழ֏ఠীࢲؘ੉ఠܳ৻ࠗदझమਵ۽੹࣠ೞӝ੹߸ജ ࢎਊ઺ੋ4.5 &YUSBDU/FX3FDPSE4UBUFݫद૑ഋधਸழ֏ఠо੉೧ೡࣻ੓חױࣽഋधਵ۽߸ജ
5JNFTUBNQ$POWFSUFSदрҙ۲чਸழ֏ఠо੉೧ೡࣻ੓חчਵ۽߸ജ ӝఋ࢚ടীٮۄழझథೞѱѐߊೠ4.5

,BGLB$POOFDU੟ҙܻ ࢎղ,BGLB$POOFDU8FCয়೑ࣗझبੑਵ۽௿۞झఠীোز઺ੋ੟࢚కഛੋ߂ઑ੘   IUUQT HJUIVCDPNLBLBPLBGLB - DPOOFDU - XFC

,BGLB$POOFDUݽפఠ݂ য়೐ࣇழ޿पಁਯ ୡ׼SFBEਯ ୊ܻೡۨ௏٘ࣻ

.Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH ੿೤ࢿѨࢎ .Z42- ,BGLB

੿೤ࢿѨࢎ 4PVSDF 5BSHFU
زӝചػప੉࠶рী੿೤ࢿਸ઱ӝ੸ਵ۽Ѩࢎೞݴزӝചҗ੿ীࢲ   ߊࢤоמೠؘ੉ఠࠛੌ஖ఐ૑

4PVSDF 5BSHFU
ߓ஖୊ܻী੸೤ೠझ౵௼ࢎਊਵ۽؀ӏݽؘ੉ఠܳੌҚ۽ࡅܰѱ୊ܻ ࢎղҊࢿמೞن௿۞झఠ੗ਗഝਊ ؘ੉ఠ೐ۨ੐ਸాೠѨࢎ۽૒ੌਗച ੿೤ࢿѨࢎ

ؘ੉ఠ࠺Үߑߨ ੹୓ۨ௏٘࠺Ү ੌױਤ߸ച࠙ۨ௏٘݅࠺Ү ೒݂௼оݒੌ੄߸ച࠙1,ܻझ౟ܳೞنী੷੢ Ѿҗഛੋߑߨ
ੌױਤ੿೤ࢿѾҗؘܳ੉ఠ߬੉झী੷੢ ݒੌয়੹ঌܿਸా೧ഛੋ ੿೤ࢿѨࢎ 🟢 [VALIDATION] যઁ, য়ט ప੉࠶ ࣻ: 300 / 300 Ѩࢎ ؘ੉ఠ ੌ੗: 2024-09-30 🟢 success(੿ഛبо 100% ~ 99% ࢎ੉ੋ ҃਋) ప੉࠶ ࣻ: 299ѐ 🟡 warning(੿ഛبо 99% ~ 95% ࢎ੉ੋ ҃਋) ప੉࠶ ࣻ: 1ѐ - service_name - database_name.table_name 98.928% (2801980/2832336)

਍৔दߊࢤೠ੉ग৬೧Ѿҗ੿ ੉गప੉࠶झః݃߸҃ ੉गҗبೠ஠೐஠ష೗ࢤࢿ ੉गOVMMч੉ӝࠄчਵ۽߄Շয੸੤

੉ग ప੉࠶झః݃߸҃

੉गझః݃߸҃ਸपदрਵ۽߈৔ೞ૑ঋח੉ਬ ࢎ੹ীঌࣻহחࢲ࠺झઑ૒੄झః݃߸҃ਃ୒ ૑಴҅࢑۽૒җഐജ੉ӵ૕ࣻ੓חоמࢿ   FH૑಴҅࢑ীࢎਊೞחணۢ࢏ઁ ޹хؘ੉ఠо೧य݃झఊহ੉ࠁউઓ৻ࠗ۽աয়חࠁউ੉ग

ࠁউ ૑಴҅࢑۽૒җഐജ ੉गझః݃߸҃ਸपदрਵ۽߈৔ೞ૑ঋח੉ਬ

%%-੉߮౟ߊࢤ х૑߂ঌܿ ഛੋ ߈৔߂੤োز ੉ग%%-؀਽ࣽࢲ

%%-੉߮౟х૑߂ঌܿ %%-੉߮౟৬੉റ੉߮౟ٜझఈ ఋѶప੉࠶ী%%-߈৔੉റ੤োزӝળ %%-੉റ੉߮౟ٜࠗఠ׮द੹࣠೙ਃ ੉गਃҳࢎ೦

%%-੉߮౟ઁ৻ࢸ੿JODMVEFTDIFNBDIBOHFTGBMTFࢎਊ %%-੉റ੉߮౟ٜب੹࣠غযࢶউغӝীࠛ୽࠙ %%-੉߮౟ߊࢤद೒݂௼੟઺ױ ஠೐஠ীFYBDUMZ - PODF۽੹࣠
-> ஠೐஠۽ழ޿ө૑ࣻ೯غযঠ஠೐஠۽੹࣠غӝী੟઺ױࠛо   ੟ਸ઺ױೞݶ%%-૒੹੄੉߮౟ٜ੉஠೐஠۽੹࣠ؽਸࠁ੢ࠛо ੉गҊ۰೧ࠄࢎ೦

੉߮౟ߊࢤ ੉߮౟ఋੑഛੋ ੹࣠ %%-ߊࢤৈࠗ ୭ୡঌܿझఈ :FT /P ੉गۄ੉࠳۞ܻী࠙ӝ୶о ੉߮౟ߊࢤ ੉߮౟ఋੑഛੋ
੹࣠ ߸҃റ ߸҃੹

%%-੉߮౟ߊࢤद%%-੉߮౟৬੉റ੉߮౟ٜఋѶী߈৔9 ೒݂௼੟਷प೯ 3VOOJOH ࢚కਬ૑ ஠೐஠۽ழ޿੉ࣻ೯غয%%-૒੹੉߮౟ٜ੄੹࣠ࠁ੢ ੉गۄ੉࠳۞ܻী࠙ӝ୶о

੉गߊࢤঌܿ ؘ੉ఠ߬੉झ߂ప੉࠶੿ࠁ (5*%T (MPCBM5SBOTBDUJPO*EFOUJ fi FST ੤োزदࢎਊ
ߊࢤೠ%%-੉߮౟ Host: source-database-slv.domain DB & Table: source_database.source_table GTIDs: 03725867-eb09-11ed-8566-b4055d39b0bc:1-1129608071 Query: ALTER TABLE `source_database`.`source_table` ADD COLUMN `added_column` TINYINT NULL

߄੉ցܻ۽Ӓ CJOMPH חࣽࢲࠁ੢ਸਤ೧ష೗౵౭࣌੉rsੋష೗ࢎਊ ప੉࠶׼ೞա੄ష೗ ౵౭࣌੉ੋష೗੉ప੉࠶ࣻ݅ఀ೙ਃ ੉गҗبೠ஠೐஠ష೗ࢤࢿ

ѐ߹੸ੋప੉࠶ٜਸೞա੄஠೐஠ష೗ਵ۽೤஖חѪ਷ޖܻ ప੉࠶ౠࢿҗझః݃о೦࢚زੌೠࢥ٬ػప੉࠶ٜীୡ੼ ੉गҗبೠ஠೐஠ష೗ࢤࢿ

੉गࢥ٬ప੉࠶ٜ O ѐ੄ష೗ࢎਊ O TPVSDF@UBCMF@<O> UBSHFU@UBCMF@<O>

੉गࢥ٬ప੉࠶ٜ O ѐ੄ష೗ࢎਊ O TPVSDF@UBCMF@<O> UBSHFU@UBCMF@<O> ੗ ਗ խ
࠺

੉गױੌష೗ࢎਊ ױੌష೗ࢎਊ TPVSDF@UBCMF@<O> UBSHFU@UBCMF@<O>

੉गਃҳࢎ೦ ࢥ٬ػప੉࠶ٜ਷ೞա੄஠೐஠ష೗ҕਬ ஠೐஠ழ֏౟ীࢲݫद૑ܳࠁҊఋѶప੉࠶ࢶఖ

੉गழझథ4.5ҳഅ { "schema": {...}, "payload": { "source": { "db": "source_database",
// ਗୌ ؘ੉ఠ߬੉झݺ "table": "source_table" // ਗୌ ప੉࠶ݺ }, "op": "r", … } } ஠೐஠ݫद૑чನݘ਷٣࠺૑਑੄DIBOHFFWFOUನݘ DIBOHFFWFOUী੓חؘ੉ఠ߬੉झ৬ప੉࠶ݺਸࢎਊ

੉गழझథ4.5ҳഅ public abstract class RenameTopic<R extends ConnectRecord<R>> implements Transformation<R> {
@Override public R apply(R record) { final String[] parts = record.valueSchema().toString().split("\\."); final String db = parts[1]; // ؘ੉ఠ߬੉झݺ ୶୹ final String table = parts[2]; // ప੉࠶ݺ ୶୹ // ࢜۽਍ ష೗ݺ ࢤࢿ ߂ ࢎਊ final String newTopic = String.format(..., db, table, shardId); return record.newRecord(newTopic, ...); } } ݫद૑੄ؘ੉ఠ߬੉झ৬ప੉࠶ݺਸࢎਊೞৈష೗੉ܴਸ߸҃ ష೗੉ܴ߸҃੉ਬ ஠೐஠ழ֏౟ীࢲప੉࠶ݺਸࢸ੿ೞ૑ঋਵݶష೗ݺਸప੉࠶ݺਵ۽ೞৈ੸੤ ݫद૑ܳࠁҊఋѶప੉࠶ࢶఖ

੉ग୭Ӕ੄؀਽ ݫद૑੹࣠ܫ੉ࢎղҕਊ஠೐஠о੉٘ۄੋӝળୡҗ ஠೐஠ష೗ਸѐীࢲѐ۽ܻ࠙ ܻ࠙റѐష೗ݫद૑੹࣠ܫ ܻ࠙੹ష೗ݫद૑੹࣠ܫ ୭؀஖ӝળড˯ࣻળ

ప੉࠶ணٜۢ઺OVMM੉ೲਊغݶࢲӝࠄч੉OVMM੉ইצணۢীࢲߊࢤ ੉गOVMMч੉ӝࠄчਵ۽߄Շয੸੤ ఋѶప੉࠶ ࣗझప੉࠶ JE D EFGBVMU D
" # OVMM $ % JE D D " # $ %

ਗੋLBGLB - DPOOFDUۄ੉࠳۞ܻҙ۲ ஠೐஠ݫद૑ܳKTPOё୓۽߸ജद OVMMч؀नӝࠄчࢎਊ ߡ੹ীࢲ೧Ѿ ,"',"
ਗੋLBGLB - DPOOFDU - KECDۄ੉࠳۞ܻҙ۲ KTPOё୓ܳࣻ೯ؼ௪ܻ۽߸ജद OVMMч؀नӝࠄчࢎਊ ੉गਗੋ౵ঈ

੉गLBGLB - DPOOFDUۄ੉࠳۞ܻ۽૒ // ߡ੹ 3.4 ੉ೞ private JsonNode convertToJson(Schema
schema, Object value) { … if (schema.defaultValue() != null) return convertToJson(schema, schema.defaultValue()); } // ߡ੹ 3.5 ੉࢚ private JsonNode convertToJson(Schema schema, Object value) { … // ࢸ੿ ч ഛੋ ઑѤ ୶о if (schema.defaultValue() != null && config.replaceNullWithDefault()) return convertToJson(schema, schema.defaultValue()); }

੉गLBGLB - DPOOFDU - KECDۄ੉࠳۞ܻ۽૒ public Object get() { Object
val = values[field.index()]; // ӝࠄ ч੉ ੓ਵݶ ೦࢚ ӝࠄ ч ࢎਊ if (val == null && field.schema().defaultValue() != null) { val = field.schema().defaultValue(); } return val; }

LBGLB - DPOOFDUۄ੉࠳۞ܻߡ੹স -> SFQMBDFOVMMXJUIEFGBVMUGBMTFࢸ੿୶о
ӝࠄчઁডਸઁѢೞחழझథ4.5ҳഅ ੉ग؀਽

KTPOё୓ী؀਽غחۄ੉࠳۞ܻղࠗё୓ীࢲӝࠄчઁডઁѢ ੉गழझథ4.5 ઁѢറ ઁѢ੹ PSHBQBDIFLBGLBDPOOFDUEBUB4USVDU LFZ4USVDU\JE^ JE */5 DOVMM
*/5 PQUJPOBM EFGBVMU Dt"u 453*/( PQUJPOBM PSHBQBDIFLBGLBDPOOFDUEBUB4USVDU LFZ4USVDU\JE^ JE */5 DOVMM */5 PQUJPOBM Dt"u 453*/( PQUJPOBM

ؘ੉ఠۨ੉௼ೞ਋झӝࣿ য়࠳ં౟झషܻ૑ӝ߈$36%о૑ਗغחয়೑ࣗझప੉࠶ನݘ ߸ച࠙ী؀೧ૐ࠙୊ܻӝמ૑ਗ *DFCFSH

.Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH ౵੉೐ۄੋҳઑ .Z42- ,BGLB

૑಴҅࢑ীࢎਊ .Z42- 'MJOL$%$ ,BGLB$POOFDU *DFCFSH .Z42- ,BGLB ౵੉೐ۄੋҳઑ

ӝઓ૑಴҅࢑җ੿ .Z42- )%'4 &YUSBDU -PBE -PBE -PBE 5SBOTGPSN

)%'4 &YUSBDU -PBE -PBE -PBE 5SBOTGPSN .Z42-

)%'4 &YUSBDU -PBE -PBE -PBE 5SBOTGPSN ࢿ מ ઁ ೠ
.Z42- ࠗ ೞ

߈ࠂ ࠗೞ ߈ࠂغח.Z42-ప੉࠶ؒ೐ ؘ੉ఠ߬੉झࠗೞ۽झ౵௼ࢿמઁೠ &YUSBDUױ҅੄ޙઁ੼

য়࠳ં౟झషܻ૑۽ೞن౵ੌदझమࢎਊ ೒݂௼ীࢲই੉झߡӒ۽߄۽੸੤ *DFCFSHبੑ .Z42- 'MJOL$%$ ,BGLB ,BGLB$POOFDU *DFCFSH
.Z42-

*DFCFSHبੑറ૑಴҅࢑җ੿ )%'4 &YUSBDU -PBE -PBE -PBE 5SBOTGPSN *DFCFSH

߈ࠂغ঻؍.Z42-ప੉࠶ؒ೐ઁѢ ஠೐஠੄ઓࢿઁѢ -> ૐ࠙झշࢫױ҅ݫद૑੹࣠ܫઁড೧ࣗ ই੉झߡӒప੉࠶ࣗय -> ೙ਃࢿמ݅ఀ੗ਗೡ׼оמೞৈࣗयदрѐࢶ
ѐࢶػٜࠗ࠙

஠೐஠੄ઓࢿઁѢ   -> ࢎղҕਊ஠೐஠о੉٘ۄੋҊ۰೙ਃ9   ஠೐஠੄ઓࢿઁѢীٮۄ߸҃ػݫद૑ನݘ "4
- *4٣࠺૑਑੄DIBOHFFWFOU 50 - #&೒݂௼੄3PX%BUB ૐ࠙झշࢫױ҅ݫद૑੹࣠ܫઁড೧ࣗ // AS-IS: change event ನݘ ݫद૑ { "schema": {...}, "payload": { "before": {...}, "after": {...}, "source": {...}, "op": r, ... } } // TO-BE: RowData ನݘ ݫद૑ +I(25,hong,gildong,[email protected],Employee,1)

L L ஠೐஠੹࣠ӝળ NTHT ই੉झߡӒ੹࣠ӝળ NTHT ೒݂௼੟߽۳ࢿӝળݫद૑੹࣠ܫ

೒݂௼੟߽۳ࢿ ૐ࠙झշࢫױ҅ݫद૑੹࣠ܫ ૐ࠙झշࢫױ҅ݫद૑੹࣠ܫ ࢎղ஠೐஠੹࣠ܫ࢚ೠ

ই੉झߡӒప੉࠶ࣗय   -> ؘ੉ఠ߬੉झࠗೞܳҊ۰ೞ૑ঋইبغযӝ؀ࢿמ݅ఀझ౵௼੗ਗೡ׼оמ ѐࢶػࣗयदр

রۨ௏٘ప੉࠶झ౵௼ࣗयदр࠺Ү .Z42-ప੉࠶ࣗय *DFCFSHప੉࠶ࣗय झ౵௼   ੊झ௸ఠࢸ੿ ௏যࣻ ݫݽܻH ѐࣻ ௏যࣻ
ݫݽܻH ѐࣻ ଵҊࢎ೦ ࠗೞ۽ੋ೧ഈ੄ػ੊झ௸ఠࢸ੿ ஹಂ࣌ࣻ೯दр҃җ ಣӐࣻ೯दр ࠙ ࠙

ஹಂ࣌ DPNQBDUJPO झշࢫ݅ܐ FYQJSF@TOBQTIPU Ҋই౵ੌઁѢ EFMFUF@PSQIBO@ fi
MFT *DFCFSH୭੸ചӝמ੸ਊ

ই੉झߡӒח౵ੌ۽ؘ੉ఠٜਸҙܻ ೒݂௼ח೦࢚.FSHF - PO - 3FBEߑधਵ۽ই੉झߡӒীؘ੉ఠ੸੤ .FSHF
- PO - 3FBEؘ੉ఠ౵ੌҗ࢏ઁ౵ੌী࠙ܨೞৈؘ੉ఠ੷੢ ઑഥद੼ী೤୛ࠁৈષ ࢤࢿغח౵ੌٜ ؘ੉ఠ౵ੌ ࢏ઁ౵ੌ 'MJOL$%$

ؘ੉ఠ౵ੌٜҗ࢏ઁ౵ੌٜਸ೤୛࢜۽਍ؘ੉ఠ౵ੌਸࢤࢿ ஹಂ࣌ ஹಂ࣌੹ ஹಂ࣌റ
೤୛૓   ౵ੌ ӝઓ౵ੌٜਬ૑ OFX

೒݂௼חੌ੿઱ӝ ୓௼ನੋ౟ ݃׮࢜۽਍ؘ੉ఠ౵ੌҗ࢏ઁ౵ੌਸ҅ࣘࢤࢿ ౱ীࢲ੄ஹಂ࣌೙ਃࢿ

೒݂௼חੌ੿઱ӝ ୓௼ನੋ౟ ݃׮࢜۽਍ؘ੉ఠ౵ੌҗ࢏ઁ౵ੌਸ҅ࣘࢤࢿ ઑഥद೤୛ঠغח౵ੌࣻоदр੉൒ܳࣻ۾ૐо ౱ীࢲ੄ஹಂ࣌೙ਃࢿ

೒݂௼חੌ੿઱ӝ ୓௼ನੋ౟ ݃׮࢜۽਍ؘ੉ఠ౵ੌҗ࢏ઁ౵ੌਸ҅ࣘࢤࢿ ઑഥद೤୛ঠغח౵ੌࣻоदр੉൒ܳࣻ۾ૐо ઱ӝ੸ஹಂ࣌ਵ۽೤୛૓౵ੌٜਸࢤࢿ೧ࢲઑഥࢿמਬ૑ ౱ীࢲ੄ஹಂ࣌೙ਃࢿ

ஹಂ࣌হਸ҃਋ࣗयदр߸ചஏ੿ রۨ௏٘ప੉࠶ࣗय ੌಣӐ߸ച࠙.ѐ   ੌ઱ੌزউࣗयदрഛੋ झ౵௼੊झ௸ఠࢸ੿
௏যࣻ ݫݽܻH ѐࣻ

ࣗयदр߸ച ࣻ೯दр ࠙ ࠙ ࠙ ࠙ ࠙ ࠙ ࠙ ੌ
ੌ ੌ ੌ ੌ ஹಂ࣌૒റ ࠙ ࠙ ࠙ ࠙ ࠙

झշࢫ݅ܐ ই੉झߡӒחҗѢ࢚కઑഥ ఋ੐౟ۨ࠶ ૑ਗ -> җѢݽ࢚ٚక߂ҙ۲౵ੌٜଵઑ ౠ੿द੼੉੹੄җѢ࢚కܳաఋղח౵ੌٜ੄ଵઑઁѢ
  Ҋই౵ੌઁѢ ଵઑоࢎۄ૓౵ੌٜ࢏ઁ झշࢫ݅ܐ߂Ҋই౵ੌઁѢ

झշࢫ݅ܐܳా೧җѢ࢚కܳաఋղח౵ੌٜ੄ଵઑઁѢ Ҋই౵ੌઁѢܳా೧ଵઑоઁѢػ౵ੌٜ࢏ઁ झշࢫ݅ܐ߂Ҋই౵ੌઁѢ Ҋই౵ੌઁѢ झշࢫ݅ܐ
ଵઑઁѢ ౵ੌ࢏ઁ

޹хؘ੉ఠ ޹хؘ੉ఠח઱ӝ੸ਵ۽җѢ࢚క৬౵ੌٜ࢏ઁ ੌ߈ؘ੉ఠ ೞن࠶۾ࢎ੉ૉࠁ׮੘਷౵ੌٜ੉҅ࣘࢤࢿغݶ ೞن౵ੌदझమ*0ࢿמীࠗ੿੸੉ۄ࢏ઁ  
౱ীࢲ੄ࣻ೯ݾ੸

ࢥ٬ప੉࠶ਸೞա੄ই੉झߡӒప੉࠶۽ా೤ *DFCFSH 'MJOL$%$ 'MJOL$%$ ױੌই੉झߡӒప੉࠶ 'MJOL$%$

ࢥ٬ప੉࠶ࣻ݅ఀࣻ೯೧ঠೞחইې੘সٜ੉ߣਵ۽хࣗ ૑಴҅࢑ਸਤೠࣗय ই੉झߡӒ୭੸ചӝמ ஹಂ࣌ झշࢫ݅ܐ Ҋই౵ੌઁѢ ೞա੄ప੉࠶۽೤ச҃਋੄੉੼

పझ౟ܳਤೠࢸ੿ *DFCFSH 'MJOL$%$ 'MJOL$%$ 'MJOL$%$ ױੌই੉झߡӒప੉࠶

UBCMF@ UBCMF@ UBCMF@ EBUBJEFOUJUZ EBUBJEFOUJUZ ױੌই੉झߡӒప੉࠶
'MJOL$%$ 'MJOL$%$ 'MJOL$%$

UBCMF@ UBCMF@ UBCMF@ EBUBJEFOUJUZ EBUBJEFOUJUZ ױੌই੉झߡӒప੉࠶
'MJOL$%$ 'MJOL$%$ 'MJOL$%$ ߣ౵౭࣌

ࣗझప੉࠶݃׮ҳ࠙ػ҃۽ ౵౭࣌ ীؘ੉ఠ੸੤ ੸੤߂ઑഥز੘җ੿

ࣗझప੉࠶݃׮ҳ࠙ػ҃۽ ౵౭࣌ ীؘ੉ఠ੸੤ ప੉࠶ઑഥदп౵౭࣌ղীࢲؘ੉ఠ౵ੌҗ࢏ઁ౵ੌٜ೤஘റਬפৡ ই੉झߡӒ੄झப೒ې׬ ੸੤߂ઑഥز੘җ੿

ࣗझప੉࠶݃׮ҳ࠙ػ҃۽ ౵౭࣌ ীؘ੉ఠ੸੤ ప੉࠶ઑഥदп౵౭࣌ղীࢲؘ੉ఠ౵ੌҗ࢏ઁ౵ੌٜ೤஘റਬפৡ ই੉झߡӒ੄झப೒ې׬ ࣗझప੉࠶ٜ੉ޛܻ੸֤ܻ੸ਵ۽ҳ࠙غয੷੢߂ઑഥоמ
੸੤߂ઑഥز੘җ੿

୨೤রۨ௏٘ѐࢥ٬ػప੉࠶ झ౵௼ܳా೧ೞա੄ই੉झߡӒప੉࠶ࣗय ಣӐࣻ೯दрୡ ୌ݅ۨ௏٘ప੉࠶_ୡ పझ౟Ѿҗ *DFCFSHప੉࠶ࣗय
झ౵௼   ੊झ௸ఠࢸ੿ ௏যࣻ ݫݽܻH ѐࣻ ଵҊࢎ೦ ஹಂ࣌ࣻ೯റदр҃җ ಣӐࣻ೯दр ୡ

ৈ۞೒݂௼੟ীࢲೞա੄ই੉झߡӒప੉࠶ীழ޿ࣻ೯   -> زदࢿ੉ग۽р೵੸ழ޿पಁ ழ޿੤दبപࣻטܾࣻ੓ਵա पജ҃ীࢲחউ੿੸਍৔੉؊઺ਃ ੸ਊޅೠ੉ਬ //
ழ޿ਵ۽ ୭न ݫఋؘ੉ఠо ׳ۄઉ ী۞ ߊࢤ org.apache.iceberg.exceptions.CommitFailedException: Cannot commit:   Base metadata location ‘hdfs://.../.../XXX-A.metadata.json' is not same as the current table metadata location 'hdfs://.../.../XXX-B.metadata.json' for namespace.source_table

대용량 데이터베이스 동기화를 위한 최적의 CDC 시스템 구축기

대용량 데이터베이스 동기화를 위한 최적의 CDC 시스템 구축기

More Decks by kakao

Other Decks in Programming

Featured

Transcript