value is 3 Leader – Master Replica Handles Clients Read (Consume) and Write (Produce) Requests Keep tracks of Followers Acknowledges produces after making sure Followers have caught up Follower – Standby Replica Reads from Leader and stores a copy
Adds a DelayedProduce Operation to Produce Purgatory with new offset with timeout Leader User Produce, msg3 1 Log msg1 msg2 msg3 2 Purgatory msg3 3
to leaders with its log end offset Leader replies with the message Leader updates its record of follower offset Leader Log msg1 msg2 msg3 Follower msg3 Fetch, off3 1 2
have fetched the new message Completes DelayedProduce in the Produce Purgatory with offset smaller than the high watermark Replies client that produce is completed Leader Log msg1 msg2 msg3 Follower Follower Log msg1 msg2 msg3 Log msg1 msg2 msg3 HW HW 1 User Purgatory msg3 2 OK, msg3
partitions Pick a random BrokerID (N) Assign Leader of partition i to broker N + i Pick a random BrokerID (M) Assign Followers of partition to broker M + i, M + i + 1 Avoid same (leader, follower) pattern Partition Leader Follower 0 5 2, 3 1 0 3, 4 2 1 4, 5 3 2 5, 0 4 3 0, 1 5 4 1, 2 6 5 3, 4 7 0 4, 5
Responsible for fetching partitions from one broker Total Number of Fetchers (No. of brokers – 1) * num.replica.fetchers Broker2 Broker1 Fetch, p0 Fetch, p1 Fetch, p2 Fetch, p3 Fetch, p4 Fetch, p5
partitions Calculate hash of topic and modulo No. of Fetchers = K Assign partition i to fetcher K + i Partition Fetcher 0 1 1 2 2 0 3 1 4 2 5 0 6 1 7 2
for 2 partitions but Fetcher 0 is idle This breaks the load balancing This is the two assignments are round robin and the modulus are multiple to each other Broker Fetcher Number of Partitions 3 0 0 1 2 2 1 4 0 1 1 0 2 2 5 0 1 1 1 2 1
for 2 partitions but Fetcher 0 is idle This breaks the load balancing This is the two assignments are round robin and the modulus are multiple to each other Broker Fetcher Number of Partitions 3 0 0 1 2 2 1 4 0 1 1 0 2 2 5 0 1 1 1 2 1 We have 3 fetchers, we want to assign partitions to all of them. But it only assigns to 1 fetcher
Similar to multi-level load balancing preferably different hashing function Avoid setting number of fetchers to be factor of number of brokers Neither multiple of number of brokers