Apache Kafka

JVM Meetup #5 - Apache Kafka at Blibli.com

Transcript

1. APACHE KAFKA EKO KURNIAWAN KHANNEDY
2. APACHE KAFKA EKO KURNIAWAN KHANNEDY ▸ Principal Software Development Engineer at Blibli.com ▸ Part of RnD Team at Blibli.com ▸ eko.k.khannedy@gdn-commerce.com
3. APACHE KAFKA AGENDA ▸ Kafka Intro ▸ Kafka Internals ▸ Installing Kafka ▸ Kafka Producer ▸ Kafka Consumer ▸ Kafka in blibli.com ▸ Demo ▸ Conclusion
4. KAFKA INTRO APACHE KAFKA
5. APACHE KAFKA BEFORE PUBLISH / SUBSCRIBE MESSAGING MEMBER ORDER RISK PAYMENT … ERP FINANCE …
6. APACHE KAFKA PUBLISH / SUBSCRIBE MESSAGING MEMBER ORDER RISK PAYMENT … ERP FINANCE … MESSAGING SYSTEM / MESSAGE BROKER
8. APACHE KAFKA WHAT IS KAFKA ▸ Apache Kafka is a publish/subscribe messaging system, or more recently a “distributing streaming platform” ▸ Opensource project under Apache Software Foundation.
9. APACHE KAFKA KAFKA HISTORY ▸ Kafka was born to solve the data pipeline problem in LinkedIn. ▸ The development team at LinkedIn was led by Jay Kreps, now CEO of Confluent. ▸ Kafka was released as an Open Source project on Github in late 2010, and join Apache Software Foundation in 2011.
10. KAFKA INTERNALS APACHE KAFKA
11. APACHE KAFKA BROKER TOPIC A
 PARTITION 0 TOPIC A
 PARTITION 1 KAFKA BROKER
12. APACHE KAFKA CLUSTER TOPIC A
 PARTITION 0 TOPIC A
 PARTITION 1 (LEADER) KAFKA BROKER 1 TOPIC A
 PARTITION 0 TOPIC A
 PARTITION 1 (LEADER) KAFKA BROKER 2
13. APACHE KAFKA TOPICS ▸ Messages in Kafka are categorized into Topics. ▸ The closest analogy for topic is a database table, or a folder in filesystem.
14. APACHE KAFKA PARTITIONS
15. APACHE KAFKA REPLICATION FACTOR TOPIC A
 PARTITION 0 TOPIC A
 PARTITION 1 KAFKA BROKER 1 TOPIC A
 PARTITION 0 KAFKA BROKER 2 TOPIC A
 PARTITION 1 KAFKA BROKER 3 TOPIC A
 PARTITION 0 TOPIC A
 PARTITION 1 KAFKA BROKER 4
16. APACHE KAFKA CONSUMER GROUP
17. APACHE KAFKA CONSUMER GROUP (2)
18. APACHE KAFKA RETENTION POLICY ▸ A key feature of Apache Kafka is that of retention, or the durable storage of messages for some period of time. ▸ We can set retention policy per topics by time or by size.
19. APACHE KAFKA MIRROR MAKER
20. INSTALLING KAFKA APACHE KAFKA
21. APACHE KAFKA JAVA ▸ Kafka using Java 8.
22. APACHE KAFKA ZOOKEEPER KAFKA BROKER PRODUCER CONSUMER ZOOKEEPER Metadata
23. APACHE KAFKA KAFKA BROKER # Minimum Broker Configuration broker.id=0 # must unique in cluster zookeeper.connect=localhost:2181 log.dirs=data/kafka-logs
24. APACHE KAFKA CREATE / UPDATE TOPIC kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic topic_name kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic_name --partitions 2 --replication-factor 2
25. KAFKA PRODUCER APACHE KAFKA
26. APACHE KAFKA PRODUCER RECORD PRODUCER RECORD TOPIC PARTITION KEY VALUE
27. APACHE KAFKA SERIALIZER PRODUCER RECORD TOPIC PARTITION KEY VALUE SERIALIZER
28. APACHE KAFKA PARTITIONER PRODUCER RECORD TOPIC PARTITION KEY VALUE SERIALIZER PARTITIONER Send to Broker
29. APACHE KAFKA KAFKA PRODUCER Properties props = new Properties(); props.put("bootstrap.servers", "broker1:9092,broker2:9092"); props.put("key.serializer", “org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); producer = new KafkaProducer<String, String>(kafkaProps);
30. APACHE KAFKA SEND MESSAGE record = new ProducerRecord<>(topicName, key, value); producer.send(record);
31. KAFKA CONSUMER APACHE KAFKA
32. APACHE KAFKA CONSUMER GROUP
33. APACHE KAFKA PARTITION REBALANCE
34. APACHE KAFKA PARTITION REBALANCE
35. APACHE KAFKA PARTITION REBALANCE
36. APACHE KAFKA CONSUMER RECORD & DESERIALIZER CONSUMER RECORD TOPIC PARTITION KEY VALUE DESERIALIZER From Broker
37. APACHE KAFKA KAFKA CONSUMER Properties props = new Properties(); props.put("bootstrap.servers", "broker1:9092,broker2:9092"); props.put("group.id", "GroupName"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); consumer = new KafkaConsumer<String, String>(props);
38. APACHE KAFKA GET MESSAGES consumer.subscribe(Collections.singletonList("topicName")); Long timeout = 1000L; ConsumerRecords<String, String> records = consumer.poll(timeout);
39. KAFKA IN BLIBLI APACHE KAFKA
40. APACHE KAFKA API GATEWAY EVENT API GATEWAY MEMBER API GATEWAY COMMON API GATEWAY … KAFKA ANALYTICS … …
41. APACHE KAFKA CURRENT PRODUCT (CODENAME X) X MEMBER X CART X AUTH X WISHLIST API GATEWAY X YYYY X XXX X ORDER X PRODUCT
42. APACHE KAFKA NEW PRODUCT (CODENAME VERONICA) VERONICA MEMBER VERONICA CORE VERONICA MERCHANT KAFKA VERONICA NOTIFICATION API GATEWAY
43. DEMO
44. CONCLUSION APACHE KAFKA
45. APACHE KAFKA WHY KAFKA? ▸ Multiple Consumer ▸ Flexible Scalability ▸ Flexible Durability ▸ High Performance ▸ Multi-Datacenter
46. WE ARE HIRING! eko.k.khannedy@gdn-commerce.com APACHE KAFKA
47. APACHE KAFKA REFERENCES ▸ http://kafka.apache.org/ ▸ https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million- writes-second-three-cheap-machines ▸ https://engineering.linkedin.com/kafka/running-kafka-scale