Efficient Integrating Data from Multiple Data Providers

By Yumei Chen @LINE TECHPULSE 2019 https://techpulse.line.me/

Transcript

2. Efficient Integrating Data
 from Multiple Data Providers > Yu-Mei Chen / LINE TAIWAN
3. Only Title / 80pt / Arial / Bold
4. Partner A LINE SPOT Users Partner B … Partner C
5. Practices From Cooperating With Partners > Processing Data Manually
 Always Preparing the manual way to update the data,
 such as updating data by import the .csv files. > Open-Closed Principle
 There are no permanent enemies, and no permanent friends. > Generalization & Customization
 Fighting for general data structures,
 but still keep a flexible layer for customized.
6. Its All About “Maintainability” Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Simplicity Evolvability Operability
7. Its All About “Maintainability” Simplicity Evolvability Operability Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Processing Data Manually Open-Closed Principle Generalization & Customization
8. Our Solution: Event Sourcing A Cloud Design Pattern for Backend Architecture
9. Event Sourcing With Kafka Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
10. Processing Data Manually Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
11. Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
12. Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch Generalization & Customization
13. Summary > Challenge in LINE SPOT: Maintainability > Solution: Event Sourcing
14. Thank You