Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MirrroMaker2 소개
Search
Jungho Choi
April 15, 2022
Programming
0
580
MirrroMaker2 소개
2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.
Jungho Choi
April 15, 2022
Tweet
Share
Other Decks in Programming
See All in Programming
あなたの知らない「動画広告」の世界 - iOSDC Japan 2025
ukitaka
0
300
Reduxモダナイズ 〜コードのモダン化を通して、将来のライブラリ移行に備える〜
pvcresin
2
630
ネイティブ製ガントチャートUIを作って学ぶUICollectionViewLayoutの威力
jrsaruo
0
110
猫と暮らすネットワークカメラ生活🐈 ~Vision frameworkでペットを愛でよう~ / iOSDC Japan 2025
yutailang0119
0
210
実践AIチャットボットUI実装入門
syumai
7
2.3k
CI_CD「健康診断」のススメ。現場でのボトルネック特定から、健康診断を通じた組織的な改善手法
teamlab
PRO
0
150
PostgreSQLで手軽にDuckDBを使う!DuckDB&pg_duckdb入門/osk2025-duckdb
takahashiikki
1
230
CSC509 Lecture 02
javiergs
PRO
0
400
CSS Linter の現在地 2025年のベストプラクティスを探る
ryo_manba
10
3.1k
Local Peer-to-Peer APIはどのように使われていくのか?
hal_spidernight
2
410
iOSアプリの信頼性を向上させる取り組み/ios-app-improve-reliability
shino8rayu9
0
120
Чего вы не знали о строках в Python – Василий Рябов, PythoNN
sobolevn
0
150
Featured
See All Featured
Visualization
eitanlees
148
16k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Why Our Code Smells
bkeepers
PRO
339
57k
RailsConf 2023
tenderlove
30
1.2k
Practical Orchestrator
shlominoach
190
11k
Building an army of robots
kneath
306
46k
GraphQLの誤解/rethinking-graphql
sonatard
72
11k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.7k
Agile that works and the tools we love
rasmusluckow
330
21k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Transcript
.JSSPS.BLFS ࣗѐ
ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
.JSSPS.BLFS١ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2
internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
.JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config
consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
.JSSPS.BLFS ױ § § § §
.JSSPS.BLFS নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
.JSSPS.BLFS नӏ ష new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
.JSSPS.BLFS ౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
.JSSPS.BLFS ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10
Consumer Producer __consumer_offsets
.JSSPS.BLFS
.JSSPS.BLFS "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0
A.test-topic.part0 M M 2 A B
.JSSPS.BLFS ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector
§ topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs
ACLS A B MM2
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B
MM2
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across
clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats
A.heartbeats
.JSSPS.BLFS प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 §
properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
.JSSPS.BLFS "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic
A B
.JSSPS.BLFS ഝਊ
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer Producer A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B Producer Consumer test-topic A.test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset test-topic Data
center A Data center B A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data
center B A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B test-topic A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ MM2 Data center A Data center B
test-topic Producer Consumer A.test-topic test-topic Producer Consumer
VTFDBTF "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
.JSSPS.BLFS ݽפఠ݂
.JSSPS.BLFS ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count
§ byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
.JSSPS.BLFS
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property
print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
.JSSPS.BLFS 1SFGJYઁѢ MM2 A B test-topic A.test-topic test-topic
.JSSPS.BLFS 1SFGJYઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726)
§ prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry
.JSSPS.BLFS TDIFNBSFHJTUSZ § _schemas schema § Single Message Transforms
(SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry 1 2 3 4 5 6
хࢎפ