Efficient Integrating Data  from Multiple Data Providers

by LINE Developers Taiwan

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Efficient Integrating Data  from Multiple Data Providers > Yu-Mei Chen / LINE TAIWAN

Slide 3

Slide 3 text

Only Title / 80pt / Arial / Bold

Slide 4

Slide 4 text

Partner A LINE SPOT Users Partner B … Partner C

Slide 5

Slide 5 text

Practices From Cooperating With Partners > Processing Data Manually  Always Preparing the manual way to update the data,  such as updating data by import the .csv files. > Open-Closed Principle  There are no permanent enemies, and no permanent friends. > Generalization & Customization  Fighting for general data structures,  but still keep a flexible layer for customized.

Slide 6

Slide 6 text

Its All About “Maintainability” Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Simplicity Evolvability Operability

Slide 7

Slide 7 text

Its All About “Maintainability” Simplicity Evolvability Operability Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Processing Data Manually Open-Closed Principle Generalization & Customization

Slide 8

Slide 8 text

Our Solution: Event Sourcing A Cloud Design Pattern for Backend Architecture

Slide 9

Slide 9 text

Event Sourcing With Kafka Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize  View for  Store Elasticsearch

Slide 10

Slide 10 text

Processing Data Manually Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize  View for  Store Elasticsearch

Slide 11

Slide 11 text

Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize  View for  Store Elasticsearch

Slide 12

Slide 12 text

Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize  View for  Store Elasticsearch Generalization & Customization