Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MirrroMaker2 소개
Search
Jungho Choi
April 15, 2022
Programming
0
540
MirrroMaker2 소개
2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.
Jungho Choi
April 15, 2022
Tweet
Share
Other Decks in Programming
See All in Programming
PRレビューのお供にDanger
stoticdev
1
240
Visual StudioのGitHub Copilotでいろいろやってみる
tomokusaba
1
220
Better Code Design in PHP
afilina
0
180
はじめての Go * WASM * OCR
sgash708
1
120
Go 1.24でジェネリックになった型エイリアスの紹介
syumai
2
300
CDKを使ったPagerDuty連携インフラのテンプレート化
shibuya_shogo
0
120
ナレッジイネイブリングにAIを活用してみる ゆるSRE勉強会 #9
nealle
0
170
DRFを少しずつ オニオンアーキテクチャに寄せていく DjangoCongress JP 2025
nealle
2
290
Drawing Heighway’s Dragon- Recursive Function Rewrite- From Imperative Style in Pascal 64 To Functional Style in Scala 3
philipschwarz
PRO
0
150
もう少しテストを書きたいんじゃ〜 #phpstudy
o0h
PRO
21
4.3k
Jasprが凄い話
hyshu
0
180
kintone開発を効率化するためにチームで試した施策とその結果を大放出!
oguemon
0
340
Featured
See All Featured
Gamification - CAS2011
davidbonilla
80
5.2k
Mobile First: as difficult as doing things right
swwweet
223
9.5k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
Code Review Best Practice
trishagee
67
18k
We Have a Design System, Now What?
morganepeng
51
7.4k
4 Signs Your Business is Dying
shpigford
183
22k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
580
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Into the Great Unknown - MozCon
thekraken
35
1.6k
A designer walks into a library…
pauljervisheath
205
24k
Transcript
.JSSPS.BLFS ࣗѐ
ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
.JSSPS.BLFS١ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2
internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
.JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config
consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
.JSSPS.BLFS ױ § § § §
.JSSPS.BLFS নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
.JSSPS.BLFS नӏ ష new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
.JSSPS.BLFS ౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
.JSSPS.BLFS ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10
Consumer Producer __consumer_offsets
.JSSPS.BLFS
.JSSPS.BLFS "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0
A.test-topic.part0 M M 2 A B
.JSSPS.BLFS ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector
§ topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs
ACLS A B MM2
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B
MM2
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across
clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats
A.heartbeats
.JSSPS.BLFS प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 §
properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
.JSSPS.BLFS "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic
A B
.JSSPS.BLFS ഝਊ
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer Producer A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B Producer Consumer test-topic A.test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset test-topic Data
center A Data center B A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data
center B A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B test-topic A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ MM2 Data center A Data center B
test-topic Producer Consumer A.test-topic test-topic Producer Consumer
VTFDBTF "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
.JSSPS.BLFS ݽפఠ݂
.JSSPS.BLFS ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count
§ byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
.JSSPS.BLFS
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property
print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
.JSSPS.BLFS 1SFGJYઁѢ MM2 A B test-topic A.test-topic test-topic
.JSSPS.BLFS 1SFGJYઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726)
§ prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry
.JSSPS.BLFS TDIFNBSFHJTUSZ § _schemas schema § Single Message Transforms
(SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry 1 2 3 4 5 6
хࢎפ