Efficient integrating data from multiple data providers

Slide 1

Slide 1 text

2019 DevDay Efficient Integrating Data From Multiple Data Providers > Yu-Mei Chen > LINE Taiwan Backend Software Engineer

Slide 2

Slide 2 text

Subtitle

Slide 3

Slide 3 text

Partner A LINE SPOT Users Partner B Partner C … $

Slide 4

Slide 4 text

Practices From Cooperating with Partners > Open-Closed Principle  There are no permanent enemies, and no permanent friends. > Generalization & Customization  Fighting for general data structures,  but still keep a flexible layer for customized. > Processing Data Manually  Always Preparing the manual way to update the data,  such as updating data by import the .csv files.

Slide 5

Slide 5 text

Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Its All About “Maintainability” Operability Evolvability Simplicity

Slide 6

Slide 6 text

Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Its All About “Maintainability” Processing Data Manually Operability Generalization & Customization Evolvability Simplicity Open-Closed Principle

Slide 7

Slide 7 text

Our Solution: Event Sourcing A Cloud Design Pattern for Backend Architecture

Slide 8

Slide 8 text

Event Sourcing With Kafka Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store  Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize  View for  Store

Slide 9

Slide 9 text

Processing Data Manually Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store  Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize  View for  Store

Slide 10

Slide 10 text

Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store  Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize  View for  Store

Slide 11

Slide 11 text

Generalization & Customization Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store  Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize  View for  Store

Slide 12

Slide 12 text

Summary > Challenge in LINE SPOT: Maintainability  > Solution: Event Sourcing Thank You for Listening And welcome for further discussion: yumei.chen@linecorp.com