MirrroMaker2 소개

2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.

Transcript

1. .JSSPS.BLFS
   ࣗѐ
2. ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
3. .JSSPS.BLFS
   ١੢ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2 internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
4. .JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
5. .JSSPS.BLFS ױ੼ § § § §
6. .JSSPS.BLFS
    নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
7. .JSSPS.BLFS
    नӏ ష೗ new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
8. .JSSPS.BLFS
    ౵౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
9. .JSSPS.BLFS
    ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10 Consumer Producer __consumer_offsets
10. .JSSPS.BLFS
11. .JSSPS.BLFS  "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0 A.test-topic.part0 M M 2 A B
12. .JSSPS.BLFS
     ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector § topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
13. .JSSPS.BLFS  .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs ACLS A B MM2
14. .JSSPS.BLFS  .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \ --topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
15. .JSSPS.BLFS  .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B MM2
16. .JSSPS.BLFS
     .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
17. .JSSPS.BLFS  .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats A.heartbeats
18. .JSSPS.BLFS
     प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 § properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
19. .JSSPS.BLFS
     EFEJDBUFE
    DMVTUFS
    "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092 target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
20. .JSSPS.BLFS
     $POOFDU
    $MVTUFS
    "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc", "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
21. .JSSPS.BLFS
     $POOFDU
    $MVTUFS
    "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc", "connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
22. .JSSPS.BLFS  "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic A B
23. .JSSPS.BLFS ഝਊ
24. VTF
    DBTF
      -JWF
    .JHSBUJPO MM2 Producer Consumer topic record & offset test-topic A.test-topic
25. VTF
    DBTF
      -JWF
    .JHSBUJPO MM2 Producer Consumer topic record & offset Consumer test-topic A.test-topic
26. VTF
    DBTF
      -JWF
    .JHSBUJPO MM2 Producer Consumer topic record & offset Consumer Producer A.test-topic test-topic
27. VTF
    DBTF
      "DUJWF4UBOECZ MM2 topic record & offset Data center A Data center B Producer Consumer test-topic A.test-topic
28. VTF
    DBTF
      "DUJWF4UBOECZ MM2 topic record & offset test-topic Data center A Data center B A.test-topic Producer Consumer
29. VTF
    DBTF
      "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data center B A.test-topic test-topic
30. VTF
    DBTF
      "DUJWF4UBOECZ MM2 topic record & offset Data center A Data center B test-topic A.test-topic Producer Consumer
31. VTF
    DBTF
      "DUJWF4UBOECZ MM2 Data center A Data center B test-topic Producer Consumer A.test-topic test-topic Producer Consumer
32. VTF
    DBTF
      "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
33. .JSSPS.BLFS ݽפఠ݂
34. .JSSPS.BLFS
     ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count § byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
35. .JSSPS.BLFS
     QSPNFUIFVT
    
    HSBGBOB
36. .JSSPS.BLFS ਍৔ ౲
37. .JSSPS.BLFS
     .JHSBUJPO MM2
38. .JSSPS.BLFS
     .JHSBUJPO MM2
39. .JSSPS.BLFS
     .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
40. .JSSPS.BLFS
     .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
41. .JSSPS.BLFS
     .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
42. .JSSPS.BLFS  1SFGJY
    ઁѢ MM2 A B test-topic A.test-topic test-topic
43. .JSSPS.BLFS  1SFGJY
    ઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726) § prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
44. .JSSPS.BLFS  TDIFNB
    SFHJTUSZ Data center A Data center B Schema Registry MM2 Schema Registry
45. .JSSPS.BLFS  TDIFNB
    SFHJTUSZ § _schemas schema § Single Message Transforms (SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
46. .JSSPS.BLFS  TDIFNB
    SFHJTUSZ Data center A Data center B Schema Registry MM2 Schema Registry 1 2 3 4 5 6
47. хࢎ೤פ׮