Slide 1

Slide 1 text

Ka#ka Huynh Quang Thao Trusting Social

Slide 2

Slide 2 text

Message Queue - Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. - Messages placed onto the queue are stored until the recipient retrieves them. (Wikipedia)

Slide 3

Slide 3 text

Overview Architecture

Slide 4

Slide 4 text

Ka#ka Architecture - General

Slide 5

Slide 5 text

Ka#ka Architecture - Partition

Slide 6

Slide 6 text

Ka#ka Architecture - replication

Slide 7

Slide 7 text

Ka#ka Architecture - Consumer Group

Slide 8

Slide 8 text

Ka#ka Architecture - overview

Slide 9

Slide 9 text

Ka#ka Architecture

Slide 10

Slide 10 text

Ka#ka at Grab https://engineering.grab.com/quotas-service

Slide 11

Slide 11 text

Ka#ka at Reddit https://redditblog.com/2017/05/24/view-counting-at-reddit/

Slide 12

Slide 12 text

Ka#ka Concepts

Slide 13

Slide 13 text

Zookeeper - It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed con#iguration service, synchronization service, and naming registry for large distributed systems. (Wikipedia)

Slide 14

Slide 14 text

Zookeeper - It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed con#iguration service, synchronization service, and naming registry for large distributed systems. (Wikipedia) - Offer high availability: Ka#ka, Spark, Solr, Hadoop.

Slide 15

Slide 15 text

Zookeeper - It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed con#iguration service, synchronization service, and naming registry for large distributed systems. (Wikipedia) - Offer high availability: Ka#ka, Spark, Solr, Hadoop. - Other similar services: etcd (used in Kubernetes), consult.

Slide 16

Slide 16 text

Zookeeper - It is essentially a centralized service for distributed systems to a hierarchical key-value store, which is used to provide a distributed con#iguration service, synchronization service, and naming registry for large distributed systems. (Wikipedia) - Offer high availability: Ka#ka, Spark, Solr, Hadoop. - Other similar services: etcd (used in Kubernetes), consult. - Ka#ka uses the Zookeeper to maintains broker con#igurations, consumer con#igurations (old version), available broker nodes, electing controller, service discovery, …

Slide 17

Slide 17 text

Broker & Controller - A Ka#ka cluster consists of one or more servers which are called brokers. - Producers are processes that publish the data into Ka#ka topics which stored in the brokers.

Slide 18

Slide 18 text

Broker & Controller - In a Ka#ka cluster, one of the brokers serves as the controller. - Elected using leader election on the Zookeeper. - Responsible for managing the states of partitions and replicas. - Electing partition leader.

Slide 19

Slide 19 text

Broker & Controller - In a Ka#ka cluster, one of the brokers serves as the controller. - Responsible for managing the states of partitions and replicas. - Electing partition leader. - Elected using leader election on the Zookeeper.

Slide 20

Slide 20 text

Topic-Partition-Replication - The topic is the place where data is published by producers and is pulled by consumers.

Slide 21

Slide 21 text

Topic-Partition-Replication - The topic is the place where data is published by producers and is pulled by consumers. - The topic is divided into several partitions. - Each partition is ordered and messages within a partition get an incrementing id, called offset. - Partitions allow us to parallelize the consuming work by splitting the data into multiple places.

Slide 22

Slide 22 text

Topic-Partition-Replication - The topic is the place where data is published by producers and is pulled by consumers. - The topic is divided into several partitions. - Each partition is ordered and messages within a partition get an incrementing id, called offset. - Partitions allow us to parallelize the consuming work by splitting the data into multiple places. - Every topic partition in Ka:ka is replicated n times, which n is the replication factor of the topic. - Each replication will be on a separate broker. - The replication factor must be less than or equal to the total brokers.

Slide 23

Slide 23 text

Topic-Partition-Replication - The topic is the place where data is published by producers and is pulled by consumers. - The topic is divided into several partitions. - Each partition is ordered and messages within a partition get an incrementing id, called offset. - Partitions allow us to parallelize the consuming work by splitting the data into multiple places. - Every topic partition in Ka:ka is replicated n times, which n is the replication factor of the topic. - Each replication will be on a separate broker. - The replication factor must be less than or equal to the total brokers. Question: Can the consumer read data from the replica?

Slide 24

Slide 24 text

Topic-Partition-Replication - Order is guaranteed only within the partition. - Data is kept only for a limited time. (default 2 weeks) - Once data is written to a partition, it can’t be changed. - Data is assigned randomly to a partition unless a key is provided.

Slide 25

Slide 25 text

Topic-Partition-Replication

Slide 26

Slide 26 text

Partition Leader/Follower/ISR - Ka#ka chooses one partition’s replica as the leader and all other partition’s leaders as followers. - All producers will write to this leader and all consumers will read from this leader. - A follower that is in-sync called an ISR. (in-sync replica) - Once the leader is down, Ka#ka will elect one follower in the "ISR set" to become the leader.

Slide 27

Slide 27 text

Partition Leader/Follower/ISR - Ka#ka chooses one partition’s replica as the leader and all other partition’s leaders as followers. - All producers will write to this leader and all consumers will read from this leader. - A follower that is in-sync called an ISR. (in-sync replica) - Once the leader is down, Ka#ka will elect one follower in the "ISR set" to become the leader. Question: How does the Ka#ka maintain the ISR set for each partition?

Slide 28

Slide 28 text

Producer and Consumer - Producer: a Ka#ka client that publishes messages to the Ka#ka cluster. - Consumer: a Ka#ka client that reads messages which is published by publishers at their own pace.

Slide 29

Slide 29 text

Producer - The producer has to specify the topic name and at least one broker to connect. - It it better to specify multiple brokers for the high availability. - Ka#ka will automatically take care of routing the data to the right brokers.

Slide 30

Slide 30 text

Producer - acknowledgement Producers can set the acknowledgment con#iguration for sending data. Acks = 0 - The message is pushed to the socket buffer. - The producer won't wait for the acknowledgment from the leader. Acks = 1 - The leader will append the message to its log and then returns the acknowledgment to the producer . - The producer will wait for the acknowledgment from the leader. Acks = all - The leader will append the message to its log. - The leader will wait for all acknowledgments from all in-sync replicas. - The producer will wait for the leader's acknowledgment. - No data loss.

Slide 31

Slide 31 text

Producer - acknowledgement Producers can set the acknowledgment con#iguration for sending data. Acks = 0 - The message is pushed to the socket buffer. - The producer won't wait for the acknowledgment from the leader. Acks = 1 - The leader will append the message to its log and then returns the acknowledgment to the producer . - The producer will wait for the acknowledgment from the leader. Acks = all - The leader will append the message to its log. - The leader will wait for all acknowledgments from all in-sync replicas. - The producer will wait for the leader's acknowledgment. - No data loss. Question: How does choosing the ACK value affect the in-sync replica set?

Slide 32

Slide 32 text

Producer - key message - Producers can choose to send a key with the message. - If a key is sent along with the message, it guarantees that all messages with the same key will always store in the same partition. - Some Ka#ka functionalities based on the key message: log compaction, cleaning up offsets ...

Slide 33

Slide 33 text

Consumer - The consumer has to specify the topic name, partition (optional) and at least one broker to connect. - It it better to specify multiple brokers for the high availability. - Ka#ka will automatically take care of pulling data from the right brokers. - Data is read in order in each partition.

Slide 34

Slide 34 text

Consumer - Offset types High Watermark: - The offset of messages that are fully replicated to all ISR-replicas. Log End Offset: - The latest offset of messages on the leader partition. - Consumer only reads up to the high watermark. Leader Partition 1 2 3 4 (HW) 5 6 7 (LEO) ISR-Replica 1 1 2 3 4 5 6 7 ISR-Replica 2 1 2 3 4 Out-of-sync 1 2 replica.lag.max.messages=4

Slide 35

Slide 35 text

Consumer - Offset types High Watermark: - The offset of messages that are fully replicated to all ISR-replicas. Log End Offset: - The latest offset of messages on the leader partition. - Consumer only reads up to the high watermark. Leader Partition 1 2 3 4 5 6 7 (HW + LEO) ISR-Replica 1 1 2 3 4 5 6 7 ISR-Replica 2 1 2 3 4 5 6 7 Out-of-sync 1 2 replica.lag.max.messages=4

Slide 36

Slide 36 text

Consumer Group - Messaging traditionally has two models: queuing and publish- subscribe. - queuing: allows us to divide up the processing of data over multiple consumer instances, which lets you scale your processing - Publish-subscribe: allow us broadcast data to multiple processes but has no way of scaling processing since every message goes to every subscriber. - The consumer group concept in Ka#ka generalizes these two concepts.

Slide 37

Slide 37 text

Consumer Group - Consumers can join a group by using the same group.id. - Ka#ka assigns partitions to the consumers in the same group. - Each partition is consumed by exactly one consumer in the group.

Slide 38

Slide 38 text

Consumer Group - Consumers can join a group by using the same group.id. - Ka#ka assigns partitions to the consumers in the same group. - Each partition is consumed by exactly one consumer in the group. - Each consumer within a group reads from exclusive partitions. - One consumer can consume multiple partitions. - It cannot have more consumers than partitions. Otherwise, some consumers will be inactive state.

Slide 39

Slide 39 text

Consumer Group - Consumers can join a group by using the same group.id. - Ka#ka assigns partitions to the consumers in the same group. - Each partition is consumed by exactly one consumer in the group. - Each consumer within a group reads from exclusive partitions. - One consumer can consume multiple partitions. - It cannot have more consumers than partitions. Otherwise, some consumers will be inactive state. Question: Do Ka#ka allow one topic have multiple consumer groups?

Slide 40

Slide 40 text

Consumer Group - Consumers can join a group by using the same group.id. - Ka#ka assigns partitions to the consumers in the same group. - Each partition is consumed by exactly one consumer in the group. - Each consumer within a group reads from exclusive partitions. - One consumer can consume multiple partitions. - It cannot have more consumers than partitions. Otherwise, some consumers will be inactive state. Question: Do Ka#ka allow one topic have multiple consumer groups? Answer: Yes - E.g: consumer groups for saving the data to database. Consumer group for transforming the data into other systems.

Slide 41

Slide 41 text

Consumer Group

Slide 42

Slide 42 text

Consumer Group

Slide 43

Slide 43 text

Consumer Group

Slide 44

Slide 44 text

Consumer Offset - Ka#ka stores the consumer offset in the topic named "_ _consumer_offsets" - [Group, Topic, Partition] -> [Offset, MetaData, TimeStamp] - Automatically create when a consumer using a group connects to the cluster. - This information stored in the Ka#ka cluster (old version: Zookeeper). - The consumer should commit the offset automatically or manually after reading the message.

Slide 45

Slide 45 text

Consumer Offset - Ka#ka stores the consumer offset in the topic named "_ _consumer_offsets" - [Group, Topic, Partition] -> [Offset, MetaData, TimeStamp] - Automatically create when a consumer using a group connects to the cluster. - This information stored in the Ka#ka cluster (old version: Zookeeper). - The consumer should commit the offset automatically or manually after reading the message. Question: What if the "_ _consumer_offsets" topic grows huge cause slowing down the search?

Slide 46

Slide 46 text

Consumer Group Coordinator - Group coordinator is the broker which receives heartbeats(or polling for message) from all consumers within the consumer group. - Every consumer group has only one group coordinator. - When a consumer want to join a consumer group, it sends a request to group coordinator.

Slide 47

Slide 47 text

Consumer Group Leader - The consumer group leader is one of the consumers in a consumer group. - When a consumer wants to join a consumer group, it sends a JoinGroup request to the group coordinator. - The #irst consumer to join the group becomes the group leader.

Slide 48

Slide 48 text

Group Leader and Coordinator relationship Rebalancing Event occurs when: - A new consumer joins the consumer group. - A consumer leaves the consumer group. - A consumer is "disconnected" to consumer group (group coordinator doesn't receive heartbeat event from this consumer)

Slide 49

Slide 49 text

Group Leader and Coordinator relationship Rebalancing Event occurs when: - A new consumer joins the consumer group. - A consumer leaves the consumer group. - A consumer is "disconnected" to consumer group (group coordinator doesn't receive heartbeat event from this consumer) - The group leader will make the assignment of each partition to each consumer. Then the group leader sends this assignment to the group coordinator. - The group coordinator will send back the assignment to all consumers. - Rebalancing event happens.

Slide 50

Slide 50 text

Group Leader and Coordinator relationship Rebalancing Event occurs when: - A new consumer joins the consumer group. - A consumer leaves the consumer group. - A consumer is "disconnected" to consumer group (group coordinator doesn't receive heartbeat event from this consumer) - The group leader will make the assignment of each partition to each consumer. Then the group leader sends this assignment to the group coordinator. - The group coordinator will send back the assignment to all consumers. - Rebalancing event happens. Question: Why doesn't the group coordinator (broker) take the job that assigns partitions to consumers, but the group leader (consumer) do that?

Slide 51

Slide 51 text

Group Leader and Coordinator Algorithm: 1.Members JoinGroup with their respective subscriptions. 2.The leader collects member subscriptions from its JoinGroup response and performs the group assignment. 3.All members (including the leader) send SyncGroup to #ind their assignment. 4.Once created, there are two cases which can trigger reassignment: a.Topic metadata changes that have no impact on subscriptions cause re-sync. The leader computes the new assignment and sends SyncGroup. b.Membership or subscription changes cause rejoin.

Slide 52

Slide 52 text

Class Diagram

Slide 53

Slide 53 text

Delivery Semantic

Slide 54

Slide 54 text

Delivery Semantic - At least once - At most once - Exactly once

Slide 55

Slide 55 text

Idempotent Producer

Slide 56

Slide 56 text

Idempotent Producer - Generate PID for each producer - PID and a sequence number are bundled together with the message

Slide 57

Slide 57 text

Isolation level for consumer Read Committed - non-transactional and COMMITTED transactional messages are visible. - Read until LSO: Last Stable Offset Read Uncommitted - All messages are visible

Slide 58

Slide 58 text

Exactly Once Semantic Part 1: Idempotent producer - guarantee messages are produced once and in order. Part 2: Producer Transaction support - Commit or fail a set of produced messages and consumer offsets across partitions Part 3: Consumer transaction support - Isolation level is read committed. - Filter out messages for aborted transactions, and wait for transactions to be committed before processing.

Slide 59

Slide 59 text

Exactly Once Semantic Part 1: Idempotent producer - guarantee messages are produced once and in order. Part 2: Producer Transaction support - Commit or fail a set of produced messages and consumer offsets across partitions Part 3: Consumer transaction support - Isolation level is read committed. - Filter out messages for aborted transactions, and wait for transactions to be committed before processing. Question: What if the producer is crashed, or the consumer is failed to commit its processed offset.

Slide 60

Slide 60 text

Ka#ka Security

Slide 61

Slide 61 text

Authentication TLS/SSL Integrating with PKI infrastructure SASL/GSSAPI Integrating with Kerberos infrastructure SASL/PLAIN Integrating with existing password server/database Custom SASL mechanism Integrating with existing authentication database Others SASL/SCAM, SASL/OathBearer(beta), SASL_SSL

Slide 62

Slide 62 text

Authentication using SASL/GSSAPI with Kerberos

Slide 63

Slide 63 text

Authentication using SASL/PLAIN

Slide 64

Slide 64 text

Authorization using Zookeeper ACL

Slide 65

Slide 65 text

The Ecosystem

Slide 66

Slide 66 text

Ka#ka Connect Ka#ka Connect is a framework for connecting Ka#ka with external systems e.g: databases, key-value stores, search indexes, and #ile systems.

Slide 67

Slide 67 text

Ka#ka Connect

Slide 68

Slide 68 text

Ka#ka Stream Ka#ka Streams is a library for building streaming applications that transform input Ka#ka topics into output Ka#ka topics.

Slide 69

Slide 69 text

Con#luent Schema Registry - Con#luent Schema Registry provides a RESTful interface for storing and retrieving Apache Avro® schemas. - It stores a versioned history of all schemas.

Slide 70

Slide 70 text

Monitoring Tools ⁃ Ka#ka consumer lag checking: https://github.com/linkedin/Burrow ⁃ Ka#ka toolbox (Topic UI, Schema UI, Connect UI): https://www.landoop.com/lenses-box/ ⁃ Ka#ka Manager: https://github.com/yahoo/ka#ka-manager ⁃ Zookeeper Con#iguration, Monitoring, Backup (Net#lix): https://github.com/soabase/exhibitor ⁃ Monitor the availability of Ka#ka clusters: https://github.com/linkedin/ka#ka-monitor ⁃ Ka#ka Tools: https://github.com/linkedin/ka#ka-tools ⁃ Cluster Management: https://github.com/apache/helix ⁃ Security Manager: https://github.com/simplesteph/ka#ka-security-manager ⁃ ....

Slide 71

Slide 71 text

Ka#ka vs RabbitMQ Advanced Message Queuing Protocol (RabbitMQ) Log based Message Protocol (Ka:ka) A single message is slow to process No problem Holds up the processing of subsequent messages in that partition. Message Order Cannot keep order in case of multiple consumers Can keep the order in the partition Consumers cannot keep up with producers when consumer die, we should carefully delete any queues whose consumer has shut down. Otherwise, they continue consume memory, affect other alive consumers. Just store messages on queue (hard drive) We choose Ka:ka when: - each message is fast to process and high message throughput - message order is important - Keyed messages - Replay old messages - Semantic Delivery

Slide 72

Slide 72 text

Client Implementation

Slide 73

Slide 73 text

Libraries There are 2 main Ka:ka client libraries: - https://github.com/Shopify/sarama - https://github.com/con#luentinc/con#luent-ka#ka-go

Slide 74

Slide 74 text

Sarama - Written in Golang. Open sourced by Shopify - Easier to read. Missed many features

Slide 75

Slide 75 text

Con#luent Ka#ka Go - Wrap again librdka#ka, which is written in C++. - Have many latest features from Ka#ka. - Still features behind of#icial Java Ka#ka client. - Con#luent: founded by Ka#ka original authors.

Slide 76

Slide 76 text

Discussion about the Ka#ka PR https://github.com/tsocial/ka#ka

Slide 77

Slide 77 text

References Designing Data-Intensive Applications Ka#ka The De#initive Guide Ka#ka Streams in Actions

Slide 78

Slide 78 text

References - http://ka#ka.apache.org/documentation.html - Consumer Heartbeats Proposal - https://cwiki.apache.org/con#luence/display/KAFKA/Offset+Management - Exactly Once Delivery Proposal - https://cwiki.apache.org/con#luence/display/KAFKA/Ka#ka+Replication - https://cwiki.apache.org/con#luence/display/KAFKA/Ka#ka+Client-side+Assignment+Proposal - https://github.com/edenhill/librdka#ka/issues/1308 - https://www.slideshare.net/jjkoshy/offset-management-in-ka#ka - https://softwareengineeringdaily.com/2016/10/07/ka#ka-streams-with-jay-kreps/ - Logs: What every software engineer should know about real-time data unifying - http://martin.kleppmann.com/2015/08/05/ka#ka-samza-unix-philosophy-distributed-data.html - http://martin.kleppmann.com/2018/01/18/event-types-in-ka#ka-topic.html - http://martin.kleppmann.com/2015/04/23/bottled-water-real-time-postgresql-ka#ka.html - https://www.con#luent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-ka#ka/ - https://www.con#luent.io/blog/how-choose-number-topics-partitions-ka#ka-cluster - https://engineering.linkedin.com/ka#ka/benchmarking-apache-ka#ka-2-million-writes-second-three- cheap-machines

Slide 79

Slide 79 text

Q&A