Multi-Datacenter Kafka at Blibli.com

Multi-Datacenter Kafka at Blibli.com

Transcript

1. MULTI-DATACENTER KAFKA
2. MULTI-DATACENTER KAFKA EKO KURNIAWAN KHANNEDY ▸ Senior Principal R&D Engineer at Blibli.com ▸ JVM User Group Indonesia Committee ▸ Blogger & Vlogger
3. BLIBLI.COM IS ONE OF THE BIGGEST B2C ECOMMERCE IN INDONESIA
4. MULTI-DATACENTER KAFKA BLIPAY (EWALLET, PAYMENT) BLIBLI.COM (ECOMMERCE) BLIBLI.COM SYSTEM
5. MULTI-DATACENTER KAFKA MESSAGE BROKER IN BLIBLI.COM ▸ Use RABBITMQ from the beginning ▸ Start using KAFKA in 2015 ▸ Deprecated RABBITMQ in 2017 because scalability issue ▸ Migrating from RABBITMQ to KAFKA until now
6. MULTI-DATACENTER KAFKA MESSAGE BROKER IN BLIPAY ▸ Start develop Blipay system in 2017 ▸ Using Event-Driven Architecture (No HTTP Call beetwen services, All communication using Message Broker) ▸ KAFKA is the only Message Broker in Blipay ▸ Based on regulation in Indonesia, Blipay must using multi data-center for disaster recovery
7. AND THIS IS OUR JOURNEY USING KAFKA
8. MULTI-DATACENTER KAFKA BLIPAY ARCHITECTURE BLIPAY-API BLIPAY-MEMBER BLIPAY- MERCHANT BLIPAY- TRANSACTION BLIPAY-PAYMENT BLIPAY- NOTIFICATION BLIPAY- PROMOTION APACHE KAFKA MONGODB MONGODB POSTGRE MONGODB MONGODB REDIS POSTGRE
9. MULTI-DATACENTER KAFKA ALL COMMUNICATION USING KAFKA BLIPAY-API BLIPAY-MEMBER BLIPAY- MERCHANT BLIPAY- TRANSACTION BLIPAY-PAYMENT BLIPAY- NOTIFICATION BLIPAY- PROMOTION APACHE KAFKA 1. HTTP Request 2. HTTP Request 4. SaveBalanceEvent 3. Some Events 4. SaveTransactionEvent
10. WE ARE HAPPY USING EVENT-DRIVEN ARCHITECTURE
11. UNTIL WE NEED TO DEPLOY TO MULTI-DATACENTER
12. MULTI-DATACENTER KAFKA SOLUTION NO 1 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER MIRROR MAKER MIRROR MAKER DATABASES DATABASES
13. MULTI-DATACENTER KAFKA PROBLEM WITH SOLUTION NO 1 ▸ INFINITE LOOP ▸ Mirror Maker in datacenter 1 send message to datacenter 2 ▸ Mirror Maker in datacenter 2 receive the message, and send back to datacenter 1
14. MULTI-DATACENTER KAFKA SOLUTION NO 2 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER MIRROR MAKER DATABASES DATABASES
15. MULTI-DATACENTER KAFKA PROBLEM WITH SOLUTION NO 2 ▸ All services in all datacenter will consume the message. ▸ For blipay-notification, we receive duplicate sms & email
16. MULTI-DATACENTER KAFKA SOLUTION NO 3 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER DATABASES DATABASES Database Mirroring
17. MULTI-DATACENTER KAFKA PROBLEM WITH SOLUTION NO 3 ▸ Not all services using database ▸ Some services need to react based on event, like blipay-notification.
18. MULTI-DATACENTER KAFKA SOLUTION NO 4 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER DATABASES DATABASES Database Mirroring MIRROR MAKER
19. MULTI-DATACENTER KAFKA PROBLEM WITH SOLUTION NO 4 ▸ All services in all datacenter will consume the message. ▸ And we still get duplicate sms & email ▸ And now we get error DUPLICATE PRIMARY KEY, because database sync and service receive message from kafka
20. MULTI-DATACENTER KAFKA SOLUTION NO 5 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER DATABASES DATABASES Database Mirroring MIRROR MAKER Block 
 Network
21. MULTI-DATACENTER KAFKA RESULT WITH SOLUTION NO 5 ▸ No duplicate primary key ▸ No duplicate sms & email ▸ And we are happy :D
22. IT'S TIME FOR SIMULATION
23. MULTI-DATACENTER KAFKA DISASTER RECOVERY SIMULATION ▸ Shutdown datacenter 1 ▸ Switch all traffic to datacenter 2 ▸ Unblock microservices network to kafka
24. MULTI-DATACENTER KAFKA AND THE RESULT ARE ... ▸ All microservice start consuming all kafka message from beginning ▸ Duplicate Primary Key Error EVERYWHERE! ▸ We receive SMS & Email again from beginning
25. WHY THIS HAPPEN?
26. MULTI-DATACENTER KAFKA ANATOMY OF KAFKA TOPIC PARTITION 1 PARTITION 2 PARTITION 3 PARTITION 4
27. MULTI-DATACENTER KAFKA ANATOMY OF KAFKA PARTITION
28. KAFKA MIRROR MAKER ONLY SEND MESSAGE TO OTHER KAFKA CLUSTER, BUT NOT THE OFFSET
29. MULTI-DATACENTER KAFKA SOLUTION NO 6 MICROSERVICES APACHE KAFKA CLUSTER MICROSERVICES APACHE KAFKA CLUSTER DATABASES DATABASES Database Mirroring MIRROR MAKER Block 
 Network OFFSET
 SYNC
30. LET'S DO SIMULATION! AGAIN!
31. MULTI-DATACENTER KAFKA AND THE RESULT ARE ... ▸ No duplicate primary key ▸ No duplicate sms & email ▸ And we are happy :D
32. BUT THERE IS ONE MORE THING
33. EVERY MONTH WE DEPLOY NEW FEATURES
34. MULTI-DATACENTER KAFKA AND THIS IS WHAT HAPPENS WHEN WE DEPLOY NEW FEATURES ▸ We start receiving sms & email about blipay from beginning, 
 but not all sms & email. ▸ Some service start reconsuming kafka messages from begining, 
 but again, not all messages. ▸ Now we are getting confused
35. LONG STORY SHORT! IT'S BECAUSE KAFKA OFFSET RETENTION
36. DEFAULT IS 1 DAY
37. SO IF IN 1 DAY, THERE IS NO ACTIVITY IN THE OFFSET, IT WILL RESET TO ZERO
38. SO, WE INCREASE OFFSET RETENTION MORE THAN TOPIC RETENTION
39. HTTPS://WWW.BLIBLI.COM/PAGE/KARIR/