Slide 1
Slide 1 text
.JSSPS.BLFS ࣗѐ
Slide 2
Slide 2 text
ݾ ର
§ MirrorMaker
§ MirrorMaker2
§ MirrorMaker2
§
§
Slide 3
Slide 3 text
.JSSPS.BLFS١
< 2.4.0
MirrorMaker
KIP-382: MirrorMaker 2.0
KIP-597 : MirrorMaker2 internal topics Formatters
KIP-545 : support automated consumer offset sync
across clusters in MM 2.0
2.4.0
2.7.0
3.0.0
KIP-720 : Deprecate MirrorMaker v1
KIP-716: Allow configuring the location of
the offset-syncs topic with MirrorMaker2
Slide 4
Slide 4 text
.JSSPS.BLFS
Consumer Producer
./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \
--whitelist= "test-topic"
Slide 5
Slide 5 text
.JSSPS.BLFS ױ
§
§
§
§
Slide 6
Slide 6 text
.JSSPS.BLFS নߑೱ
test-topic test-topic
Mirroring
Mirroring
Consumer Producer
Slide 7
Slide 7 text
.JSSPS.BLFS नӏ ష
new-topic
./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \
--whitelist= "test-topic"
Consumer Producer
Slide 8
Slide 8 text
.JSSPS.BLFS ౭࣌
test-topic
partition 1
test-topic
partition 2
Consumer Producer
Slide 9
Slide 9 text
.JSSPS.BLFS ஶगݠ Ӓܛ
test-topic
Consumer
group-1
test-topic
offset : 10
Consumer Producer
__consumer_offsets
Slide 10
Slide 10 text
.JSSPS.BLFS
Slide 11
Slide 11 text
.JSSPS.BLFS "DUJWF"DUJWF
test-topic.part0
test-topic.part1
B.test-topic.part0
B.test-topic.part0
test-topic.part0
test-topic.part1
A.test-topic.part0
A.test-topic.part0
M
M
2
A B
Slide 12
Slide 12 text
.JSSPS.BLFS ҳࢿ
§ Kafka Connect
§ 3 Connector
§ MirrorSourceConnector
§ topic
§ topic ACLs
§ MirrorCheckpointConnector
§ consumer group offset
§ MirrorHeartbeatConnector
§ Heartbeat
Slide 13
Slide 13 text
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS
test-topic A.test-topic
mm2-offset-syncs.B.internal
Record
Offset syncs
Configs
ACLS
A B
MM2
Slide 14
Slide 14 text
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS
OffsetSync{topicPartition=test-topic-0,
upstreamOffset=224, downstreamOffset=65}
./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \
--formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter
§ KIP-597: MirrorMaker2 internal topics Formatters
Slide 15
Slide 15 text
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS
mm2-offset-syncs.B.internal
__consumer_offsets
A.checkpoints.internal
__consumer_offsets
Checkpoints
A B
MM2
Slide 16
Slide 16 text
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS
§ KIP-545: support automated consumer offset sync across clusters in MM 2.0
A->B.sync.group.offsets.enabled = true
§ A B .
§ A/B mm2-offset-syncs.B.internal
§ __consumer_offsets .
Slide 17
Slide 17 text
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS
heartbeats
A B
MM2
Checkpoints
MirrorSourceConnector
heartbeats
A.heartbeats
Slide 18
Slide 18 text
.JSSPS.BLFS प೯ ߑߨ
§ Dedicated MirrorMaker cluster
§ MirrorMaker2
§ properties
§ Running MirrorMaker in Connect cluster
§ Connect Cluster 3 Connector
Slide 19
Slide 19 text
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ
clusters = source, target
source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092
source->target.enabled = true
source->target.topics = .*
source->target.group = .*
source->target.sync.group.offsets.enabled = true
target->source.enabled = false
mm2.properties
./bin/connect-mirror-maker.sh mm2.properties
Slide 20
Slide 20 text
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ
{
"name": "mm2-msc",
"config": {
"name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"clusters": "source, target",
"source.cluster.alias": "source",
"source.cluster.bootstrap.servers": "source_kafka:9092",
"target.cluster.alias": "target",
"target.cluster.bootstrap.servers": "target_kafka:9092",
"topics": ".*",
"tasks.max": "5"
}
}
MirrorSourceConnector
POST http://connect_url/connectors
Slide 21
Slide 21 text
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ
{
"name": "mm2-cpc",
"config":
{
"name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"clusters": "source, target",
"source.cluster.alias": "source",
"source.cluster.bootstrap.servers": "source_kafka:9094",
"target.cluster.alias": "target",
"target.cluster.bootstrap.servers": "target_kafka:9094",
"groups": ".*",
"tasks.max": "5"
}
}
MirrorCheckpointConnector
POST http://connect_url/connectors
Slide 22
Slide 22 text
.JSSPS.BLFS "DUJWF 4UBOECZ
test-topic
M
M
2
A.test-topic
test-topic
A B
Slide 23
Slide 23 text
.JSSPS.BLFS ഝਊ
Slide 24
Slide 24 text
VTFDBTF -JWF.JHSBUJPO
MM2
Producer
Consumer
topic record & offset
test-topic
A.test-topic
Slide 25
Slide 25 text
VTFDBTF -JWF.JHSBUJPO
MM2
Producer
Consumer
topic record & offset
Consumer
test-topic
A.test-topic
Slide 26
Slide 26 text
VTFDBTF -JWF.JHSBUJPO
MM2
Producer
Consumer
topic record & offset
Consumer
Producer
A.test-topic
test-topic
Slide 27
Slide 27 text
VTFDBTF "DUJWF4UBOECZ
MM2
topic record & offset
Data center A Data center B
Producer
Consumer
test-topic
A.test-topic
Slide 28
Slide 28 text
VTFDBTF "DUJWF4UBOECZ
MM2
topic record & offset
test-topic
Data center A Data center B
A.test-topic
Producer
Consumer
Slide 29
Slide 29 text
VTFDBTF "DUJWF4UBOECZ
Producer
Consumer
test-topic
Data center A Data center B
A.test-topic
test-topic
Slide 30
Slide 30 text
VTFDBTF "DUJWF4UBOECZ
MM2
topic record & offset
Data center A Data center B
test-topic
A.test-topic
Producer
Consumer
Slide 31
Slide 31 text
VTFDBTF "DUJWF4UBOECZ
MM2
Data center A Data center B
test-topic
Producer
Consumer
A.test-topic
test-topic
Producer
Consumer
Slide 32
Slide 32 text
VTFDBTF "HHSFHBUJPO
MM2
Producer
Producer
A.test-topic
B.test-topic
Consumer
Slide 33
Slide 33 text
.JSSPS.BLFS ݽפఠ݂
Slide 34
Slide 34 text
.JSSPS.BLFS ݽפఠ݂
§ Throughput/latency per partition
§ kafka.connect.mirror:type=MirrorSourceConnector
§ record-count
§ byte-rate
§ record-age-ms
§ replication-latency-ms
§ offset checkpoint latency
§ kafka.connect.mirror:type=MirrorCheckpointConnector
§ checkpoint-latency-ms
Slide 35
Slide 35 text
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
Slide 36
Slide 36 text
.JSSPS.BLFS
Slide 37
Slide 37 text
.JSSPS.BLFS .JHSBUJPO
MM2
Slide 38
Slide 38 text
.JSSPS.BLFS .JHSBUJPO
MM2
Slide 39
Slide 39 text
.JSSPS.BLFS .JHSBUJPO
./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \
--topic mm2-offset.A.internal \
--property print.key=true --from-beginning
header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}]
body : {"offset":225}
§ source connector offset .
§ offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal)
§ MirrorSourceConnector mm2-offset.A.internal
Slide 40
Slide 40 text
.JSSPS.BLFS .JHSBUJPO
MM2
Consumer
mm2-offset.A.internal
Producer
mm2-offset.A.internal
Slide 41
Slide 41 text
.JSSPS.BLFS .JHSBUJPO
MM2
Consumer
mm2-offset.A.internal
Producer
mm2-offset.A.internal
offset
Slide 42
Slide 42 text
.JSSPS.BLFS 1SFGJYઁѢ
MM2
A B
test-topic
A.test-topic
test-topic
Slide 43
Slide 43 text
.JSSPS.BLFS 1SFGJYઁѢ
§ DefaultReplicationPolicy
§ prefix
§ IdentityReplicationPolicy (KAFKA-9726)
§ prefix
replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
§ replication.policy.class
§ DefaultReplicationPolicy ReplicationPolicy, Configurable 구현
§
Slide 44
Slide 44 text
.JSSPS.BLFS TDIFNBSFHJTUSZ
Data center A Data center B
Schema Registry
MM2
Schema Registry
Slide 45
Slide 45 text
.JSSPS.BLFS TDIFNBSFHJTUSZ
§ _schemas schema
§ Single Message Transforms (SMT)
§ SMT (source connector)
§ Schema Registry Transfer SMT
1. source schema ID
2. source schema registry schema
3. target schema registry schema
4. target
Slide 46
Slide 46 text
.JSSPS.BLFS TDIFNBSFHJTUSZ
Data center A Data center B
Schema Registry
MM2
Schema Registry
1
2
3 4
5
6
Slide 47
Slide 47 text
хࢎפ