Slide 1

Slide 1 text

γεςϜϦϓϨʔεͷதͰ CDCΛऔΓೖΕͯΔ࿩ 2023-10-07 தࠃ஍ํDBษڧձ ೔ຊPostgreSQLϢʔβʔձ ߴڮ Ұٍ

Slide 2

Slide 2 text

ࣗݾ঺հ • ߴڮ Ұٍ (@ikkitang) • ελʔϑΣεςΟόϧגࣜձࣾ Tech Product Manager • TypeScriptͱGo • ೔ຊPostgreSQLϢʔβʔձ தࠃࢧ෦௕

Slide 3

Slide 3 text

͓͠ͳ͕͖ • CDC࠾༻ͷഎܠ • CDCͱ͸ʁ • Debeziumͷಛ௃

Slide 4

Slide 4 text

஫ҙࣄ߲ • εϥΠυ͸ެ։ࡁΈͰ͢ • ࣭໰ͱ͔͋Ε͹ #ChugokuDB Λ͝׆༻͍ͩ͘͞ʂ • ࿩ͷ࣠ͱͯ͠"γεςϜϦϓϨʔε"͕ग़ͯ͘ΔͷͰ ՝୊ײͷڞײͷҙਤΛࠐΊͯ ॴଐͷձࣾͷ࿩Λଟ͘͠·͕͢࠾༻໨త͡Όͳ͍Ͱ͢ • ※ CDC࠾༻ͷഎܠ͕݁ߏϘϦϡʔϜΛ͠Ίͯ͠·͍·ͨ͠

Slide 5

Slide 5 text

CDC࠾༻ͷഎܠ

Slide 6

Slide 6 text

CDC࠾༻ͷഎܠ ελʔϑΣεςΟόϧגࣜձࣾ • 2009೥7݄7೔૑ۀʢ15ظ໨ʣ • BtoB(C) ޲͚ͷϏδωεϞσϧΛల։ • ҿ৯ళ༷ͷECࢀೖɾӡ༻Λτʔλϧࢧ ԉ • ECαΠτ্Ͱ஫จɾܾࡁड෇ • ੡଄঎඼ͷडऔ • ੡଄঎඼ͷ͓ಧ͚

Slide 7

Slide 7 text

CDC࠾༻ͷഎܠ ελʔϑΣεςΟόϧגࣜձࣾͷϓϩμΫτͷ՝୊ײ • 3छྨͷΞΫλʔ • ੡଄ύʔτφʔ, ഑ૹύʔτφʔ, ٤৯ऀ༷ • 15ظͷϏδωεΛࢧ͑ͨෳࡶͳۀ຿ϑϩʔʢి࿩ʣ • ͦΕΒΛશ෦ࢧ͑Δ "தԝूݖͷΞϓϦέʔγϣϯ" • ʮγεςϜͷ౎߹Ͱ...͝ڠྗΛ͓ئ͍͍ͨ͠·͢ʯ

Slide 8

Slide 8 text

CDC࠾༻ͷഎܠ γεςϜϦϓϨʔεΛ΍Δ • γεςϜϦϑΝΫλϦϯάͰ͸ͳͯ͘ɺϦϓϨʔε. • ϦϑΝΫλϦϯά͸֎෦తͳৼΔ෣͍Λม͑ͳ͍ࣄ͕ఆٛ • ৼΔ෣͍΋ม͑ͳ͍ͱɺ՝୊ղܾʹͳΒͳ͍ͷͰม͍͑ͯ͘ • ٕज़తʹ͖Ε͍ʹͳ͚ͬͨͩͷچγεςϜ͸ͭ͘Βͳ͍

Slide 9

Slide 9 text

CDC࠾༻ͷഎܠ ϦϞσϦϯάΛ΍Δ • Ϣʔβʔͷߦಈ΍ಈػΛ෼ੳͯ͠ ௚ײతͳυϝΠϯʢ༻ޠɾUIʣͱͯ͠࠶ఆٛ͢Δ • ϦϞσϦϯάͨ͠ϞσϧΛѻ͏৽γεςϜΛఏڙ͢Δ

Slide 10

Slide 10 text

CDC࠾༻ͷഎܠ Ϟσϧ͕ίϯςΩετΛ·͙ͨ • ҰํͰɺҰਓͷΞΫλʔ͔Βݟͨ௚ײతͳυϝΠϯ͸ɺผͷΞΫλʔ͔ΒݟΔͱ௚ײత͡Ό ͳ͍ࣄ͕͋Δ • ʮΞΫλʔ͕มΘΔͱؔ৺͕มΘΔʯΛແࢹͯ͠৽γεςϜ಺Ͱѻ͍ͬͯ͘ͱ·ͨಉ͡໰୊ ͕ൃੜ͢Δ • ͏·͘ίϯςΩετΛ·͍ͨͰ͍͘࢓૊Έ͕ඞཁ => ΠϕϯτۦಈܕΞʔΩςΫνϟΛ࠾༻ • ྫɿ ੡଄ύʔτφʔ͕"঎඼Λొ࿥"ͨ࣌͠ɺͦͷ঎඼͸αϙʔτνʔϜͷ "৹ࠪ଴ͪ঎඼"ϦετʹฒͿ

Slide 11

Slide 11 text

CDC࠾༻ͷഎܠ·Ͱ΋͏গ͠Ͱ͢

Slide 12

Slide 12 text

CDC࠾༻ͷഎܠ ΠϕϯτۦಈܕΞʔΩςΫνϟ • ΞϓϦέʔγϣϯΛߏ੒͢Δίϯϙʔωϯτ͕Πϕϯτʹج͍ͮͯಈ࡞͢Δઃܭύλʔϯ • ಛఆͷΠϕϯτ͕ൃੜͨ࣌͠ʹద੾ʹΞΫγϣϯΛى͜͢ࣄΛ໨త • × ঎඼Λొ࿥ͯ͠ʂ • ̋ ঎඼͕ొ࿥͞Ε·ͨ͠ʂ • εέʔϥϏϦςΟ, ॊೈੑ͕ߴ·Δ • Apache Kafka ͱ͍͏ϛυϧ΢ΣΞͷγΣΞ͕ଟ͍

Slide 13

Slide 13 text

CDC࠾༻ͷഎܠ Apache Ka)aΛ༻͍ͨ ΠϕϯτۦಈܕΞʔΩςΫ νϟ • ੡଄ύʔτφʔ༻ͷΞϓϦέʔγϣϯ • SoEʢྑ͍UXΛఏڙ͍ͨ͠!ʣ • ࣾ಺޲͚؅ཧը໘ͷΞϓϦέʔγϣϯ • SoRʢ͍ͭ୭͕ͲΜͳࢦࣔͷجͰ࣮ߦ͔ͨ͠. ཤྺʹ ॏ͖Λஔ͘ʣ • ΞΫλʔ΍ΞϓϦέʔγϣϯͷੑ࣭ͷҧ͍Λka-aΛ෗ ഊ๷ࢭ૚ͱͯ͠औΓೖΕΔࣄͰંΓ߹͍͚ͭͯΔ

Slide 14

Slide 14 text

ҰํͰ..!

Slide 15

Slide 15 text

ແࢹͰ͖ͳ͍چγεςϜ • ݱঢ়΋Ϗδωε͸ಈ͍͍ͯͯͯɺಉ࣌ӡ༻͸ආ͚ΒΕͳ͍ • ઌఔઃܭͨ͠ka$a΁ͷϝοηʔδͷૹ৴ΛچγεςϜʹରͯ͠ खΛೖΕΔࣄ͸೉͍͠ • ͳΜͱ͔ɺچγεςϜͱ͏·͘࿈ܞΛऔΔࣄ͕ग़དྷͳ͍͔ʁ

Slide 16

Slide 16 text

CDCΛ࠾༻͢Δͧʂʂ

Slide 17

Slide 17 text

CDC ͱ͸

Slide 18

Slide 18 text

CDCͱ͸ • Change Data Capture • σʔλϕʔεͷมߋʢInsert,Update,DeleteʣΠϕϯτΛݕ஌͠ ผͷΞϓϦέʔγϣϯʹ఻ୡ͢ΔͨΊͷٕज़΍ख๏ • ࠾༻ࣄྫ • ৯΂ϩά͞Μɿ Elas5cSearch IndexͷϦΞϧλΠϜߋ৽ • Chatwork͞Μɿ CQRSϞσϧͷߋ৽(ಡΈࠐΈϞσϧͷߋ৽) • ΦϛΧϨ͞Μɿ σʔλϕʔεϦϑΝΫλϦϯά • Debezium ౕ͕ͬͯྑ͍Β͍͠

Slide 19

Slide 19 text

Debeziumͷಛ௃

Slide 20

Slide 20 text

Debezium ͷಛ௃ • Apache Ka*a Connect ʹ४ڌͯ͠࡞ΒΕͨ CDCͷπʔϧηοτ • Debeziumࣗମ͸ DBͷมߋΠϕϯτΛKa*aͷϝοηʔδͱͯ͠ ൃՐ͢Δ੹຿ • DB͝ͱʹConnectorΛ࢖ͬͯมߋΠϕϯτͷݕ஌Λ࣮ݱ͢Δ • MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Cloud Spanner, ...

Slide 21

Slide 21 text

࢖༻ྫ ͜ΜͳJSONΛ࡞Δ. { "name": "connector໊", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "ུ": "ུ", "database.hostname": "{mysql}", "database.port": "{port}", "database.user": "{user}", "database.password": "{password}", "database.include.list": "{schema}", "table.include.list": "{schema.table1,...}", "snapshot.mode": "initial" } } ͜Μͳσʔλ͕ಘΒΕΔ // Insert { "before": null, "after": { "id": 1, "name": "init name" } } // Update { "before": { "id": 1, "name": "before name" }, "after": { "id": 1, "name": "after name" } } // Delete { "before": { "id": 1, "name": "after name" }, "after": null }

Slide 22

Slide 22 text

ӡ༻ʹ͓͍ͯؾΛ͚ͭΔࣄ Debezium • ConnectorͰಡΈࠐΉςʔϒϧΛ௥Ճ͢Δ࣌͸ ৬ਓ͕ܳඞཁ ʢΫισΧίωΫλʔ໊Λ͚ͭͳ͍ʂ 1ഊʣ MySQL • ίωΫλఆٛ࣌ͷσϑΥϧτͷڍಈͱͯ͠ ςʔϒϧͷશϨίʔυΛಡΈऔΔʢεφοϓγϣοτʣ • ಡऔதʹಉ͡Ϩίʔυ͕ಡΊΔΑ͏ʹϩοΫऔΔ ʢάϩʔόϧϦʔυϩοΫ or ςʔϒϧϩοΫʣ • ΊͪΌͪ͘ΌͰ͔͍ςʔϒϧΛಡΉ࣌͸΍͹͍ʢ΍͹͍ 1ഊʣ PostgreSQL • ಡऔઐ༻γϦΞϥΠβϒϧ෼཭Ϩϕϧͷ τϥϯβΫγϣϯΛషΔ • ϩοΫ͸ແ͍

Slide 23

Slide 23 text

ӡ༻ʹ͓͍ͯؾΛ͚ͭΔࣄͷվળྫ • DebeziumͷϩοΫΛؾʹ͢Δ৔߹͸ɺ CDCઐ༻ͷReadReplicaΛཱͯΔ • Debeziumࣗମʹࢮ׆؂ࢹΛೖΕ͓ͯ͘ͷ͕ྑ͍ • αʔόʔ΍֤λεΫͷੜ͖ࢮʹ • ʮ๻͸ࡴ͞Ε·ͨ͠ɻࣗಈͰ͸ىಈग़དྷͳ͍ͷͰखಈͰىͯ͜͠ʯ ͬͯϩάʹग़ΔͷͰͦΕͷ؂ࢹ • ↑ ։ൃɾݕূ؀ڥͳͲͷ DBΛࢭΊΔ؀ڥͰൃੜ͢Δ • Subscrip6on Filter Ͱݕ஌ͯ͠ LambdaͰ Restart API Λࣗಈ࣮ߦ͢Δ

Slide 24

Slide 24 text

(10ϲ݄)ӡ༻ͯ͠ΈͯಘΒΕͨ஌ݟɾײ૝ • ΊͪΌͪ͘Ό଎͍ • APIʹϦΫΤετ౤͛ͯ Ϧϩʔυͨ͠Β൓ө͞ΕͯΔ • RESTͰͷઃఆָ͕ • IaaC • ެࣜυΩϡϝϯτͷ໢ཏੑ • ϋϚͬͯ΋ɺجຊެࣜDoc͕Ҿ༻ ͞ΕͯΔ • ϑϧϚωʔδυ͡Όͳ͍ • MSKͰka'a-connectαϙʔτͯ͠ Δ͚Ͳɺӡ༻ʹͷͤΔͷ͕ϦεΫ • ઃఆϛεͬͯΔͱͣͬͱϦτϥΠ ʢREST࢖͑ͳ͍ʣ • ϩʔΧϧ؀ڥ • DBͱͷ઀ଓ͕Ұఆࢭ·Δ࣌ʹ૬ ੑѱΊ

Slide 25

Slide 25 text

Ҏ্Ͱ͢ʂ