Apache Kafka - Speaker Deck

Slide 1

Slide 1 text

APACHE KAFKA EKO KURNIAWAN KHANNEDY

Slide 2

Slide 2 text

APACHE KAFKA EKO KURNIAWAN KHANNEDY ▸ Principal Software Development Engineer at Blibli.com ▸ Part of RnD Team at Blibli.com ▸ eko.k.khannedy@gdn-commerce.com

Slide 3

Slide 3 text

APACHE KAFKA AGENDA ▸ Kafka Intro ▸ Kafka Internals ▸ Installing Kafka ▸ Kafka Producer ▸ Kafka Consumer ▸ Kafka in blibli.com ▸ Demo ▸ Conclusion

Slide 4

Slide 4 text

KAFKA INTRO APACHE KAFKA

Slide 5

Slide 5 text

APACHE KAFKA BEFORE PUBLISH / SUBSCRIBE MESSAGING MEMBER ORDER RISK PAYMENT … ERP FINANCE …

Slide 6

Slide 6 text

APACHE KAFKA PUBLISH / SUBSCRIBE MESSAGING MEMBER ORDER RISK PAYMENT … ERP FINANCE … MESSAGING SYSTEM / MESSAGE BROKER

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

APACHE KAFKA WHAT IS KAFKA ▸ Apache Kafka is a publish/subscribe messaging system, or more recently a “distributing streaming platform” ▸ Opensource project under Apache Software Foundation.

Slide 9

Slide 9 text

APACHE KAFKA KAFKA HISTORY ▸ Kafka was born to solve the data pipeline problem in LinkedIn. ▸ The development team at LinkedIn was led by Jay Kreps, now CEO of Conﬂuent. ▸ Kafka was released as an Open Source project on Github in late 2010, and join Apache Software Foundation in 2011.

Slide 10

Slide 10 text

KAFKA INTERNALS APACHE KAFKA

Slide 11

Slide 11 text

APACHE KAFKA BROKER TOPIC A  PARTITION 0 TOPIC A  PARTITION 1 KAFKA BROKER

Slide 12

Slide 12 text

APACHE KAFKA CLUSTER TOPIC A  PARTITION 0 TOPIC A  PARTITION 1 (LEADER) KAFKA BROKER 1 TOPIC A  PARTITION 0 TOPIC A  PARTITION 1 (LEADER) KAFKA BROKER 2

Slide 13

Slide 13 text

APACHE KAFKA TOPICS ▸ Messages in Kafka are categorized into Topics. ▸ The closest analogy for topic is a database table, or a folder in ﬁlesystem.

Slide 14

Slide 14 text

APACHE KAFKA PARTITIONS

Slide 15

Slide 15 text

APACHE KAFKA REPLICATION FACTOR TOPIC A  PARTITION 0 TOPIC A  PARTITION 1 KAFKA BROKER 1 TOPIC A  PARTITION 0 KAFKA BROKER 2 TOPIC A  PARTITION 1 KAFKA BROKER 3 TOPIC A  PARTITION 0 TOPIC A  PARTITION 1 KAFKA BROKER 4

Slide 16

Slide 16 text

APACHE KAFKA CONSUMER GROUP

Slide 17

Slide 17 text

APACHE KAFKA CONSUMER GROUP (2)

Slide 18

Slide 18 text

APACHE KAFKA RETENTION POLICY ▸ A key feature of Apache Kafka is that of retention, or the durable storage of messages for some period of time. ▸ We can set retention policy per topics by time or by size.

Slide 19

Slide 19 text

APACHE KAFKA MIRROR MAKER

Slide 20

Slide 20 text

INSTALLING KAFKA APACHE KAFKA

Slide 21

Slide 21 text

APACHE KAFKA JAVA ▸ Kafka using Java 8.

Slide 22

Slide 22 text

APACHE KAFKA ZOOKEEPER KAFKA BROKER PRODUCER CONSUMER ZOOKEEPER Metadata

Slide 23

Slide 23 text

APACHE KAFKA KAFKA BROKER # Minimum Broker Configuration broker.id=0 # must unique in cluster zookeeper.connect=localhost:2181 log.dirs=data/kafka-logs

Slide 24

Slide 24 text

APACHE KAFKA CREATE / UPDATE TOPIC kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic topic_name kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic_name --partitions 2 --replication-factor 2

Slide 25

Slide 25 text

KAFKA PRODUCER APACHE KAFKA

Slide 26

Slide 26 text

APACHE KAFKA PRODUCER RECORD PRODUCER RECORD TOPIC PARTITION KEY VALUE

Slide 27

Slide 27 text

APACHE KAFKA SERIALIZER PRODUCER RECORD TOPIC PARTITION KEY VALUE SERIALIZER

Slide 28

Slide 28 text

APACHE KAFKA PARTITIONER PRODUCER RECORD TOPIC PARTITION KEY VALUE SERIALIZER PARTITIONER Send to Broker

Slide 29

Slide 29 text

APACHE KAFKA KAFKA PRODUCER Properties props = new Properties(); props.put("bootstrap.servers", "broker1:9092,broker2:9092"); props.put("key.serializer", “org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); producer = new KafkaProducer(kafkaProps);

Slide 30

Slide 30 text

APACHE KAFKA SEND MESSAGE record = new ProducerRecord<>(topicName, key, value); producer.send(record);

Slide 31

Slide 31 text

KAFKA CONSUMER APACHE KAFKA

Slide 32

Slide 32 text

APACHE KAFKA CONSUMER GROUP

Slide 33

Slide 33 text

APACHE KAFKA PARTITION REBALANCE

Slide 34

Slide 34 text

APACHE KAFKA PARTITION REBALANCE

Slide 35

Slide 35 text

APACHE KAFKA PARTITION REBALANCE

Slide 36

Slide 36 text

APACHE KAFKA CONSUMER RECORD & DESERIALIZER CONSUMER RECORD TOPIC PARTITION KEY VALUE DESERIALIZER From Broker

Slide 37

Slide 37 text

APACHE KAFKA KAFKA CONSUMER Properties props = new Properties(); props.put("bootstrap.servers", "broker1:9092,broker2:9092"); props.put("group.id", "GroupName"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumer = new KafkaConsumer(props);

Slide 38

Slide 38 text

APACHE KAFKA GET MESSAGES consumer.subscribe(Collections.singletonList("topicName")); Long timeout = 1000L; ConsumerRecords records = consumer.poll(timeout);

Slide 39

Slide 39 text

KAFKA IN BLIBLI APACHE KAFKA

Slide 40

Slide 40 text

APACHE KAFKA API GATEWAY EVENT API GATEWAY MEMBER API GATEWAY COMMON API GATEWAY … KAFKA ANALYTICS … …

Slide 41

Slide 41 text

APACHE KAFKA CURRENT PRODUCT (CODENAME X) X MEMBER X CART X AUTH X WISHLIST API GATEWAY X YYYY X XXX X ORDER X PRODUCT

Slide 42

Slide 42 text

APACHE KAFKA NEW PRODUCT (CODENAME VERONICA) VERONICA MEMBER VERONICA CORE VERONICA MERCHANT KAFKA VERONICA NOTIFICATION API GATEWAY

Slide 43

Slide 43 text

DEMO

Slide 44

Slide 44 text

CONCLUSION APACHE KAFKA

Slide 45

Slide 45 text

APACHE KAFKA WHY KAFKA? ▸ Multiple Consumer ▸ Flexible Scalability ▸ Flexible Durability ▸ High Performance ▸ Multi-Datacenter

Slide 46

Slide 46 text

WE ARE HIRING! eko.k.khannedy@gdn-commerce.com APACHE KAFKA

Slide 47

Slide 47 text

APACHE KAFKA REFERENCES ▸ http://kafka.apache.org/ ▸ https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million- writes-second-three-cheap-machines ▸ https://engineering.linkedin.com/kafka/running-kafka-scale