Efficient integrating data from multiple data providers

Efficient integrating data from multiple data providers

Yu-Mei Chen
LINE Taiwan Backend Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S2-4

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 21, 2019
Tweet

Transcript

  1. 2019 DevDay Efficient Integrating Data From Multiple Data Providers >

    Yu-Mei Chen > LINE Taiwan Backend Software Engineer
  2. Subtitle

  3. Partner A LINE SPOT Users Partner B Partner C …

    $
  4. Practices From Cooperating with Partners > Open-Closed Principle
 There are

    no permanent enemies, and no permanent friends. > Generalization & Customization
 Fighting for general data structures,
 but still keep a flexible layer for customized. > Processing Data Manually
 Always Preparing the manual way to update the data,
 such as updating data by import the .csv files.
  5. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn

    Its All About “Maintainability” Operability Evolvability Simplicity
  6. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn

    Its All About “Maintainability” Processing Data Manually Operability Generalization & Customization Evolvability Simplicity Open-Closed Principle
  7. Our Solution: Event Sourcing A Cloud Design Pattern for Backend

    Architecture
  8. Event Sourcing With Kafka Apache Kafka Partner A Stream Store

    Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  9. Processing Data Manually Apache Kafka Partner A Stream Store Service

    Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  10. Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon

    Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  11. Generalization & Customization Apache Kafka Partner A Stream Store Service

    Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  12. Summary > Challenge in LINE SPOT: Maintainability
 > Solution: Event

    Sourcing Thank You for Listening And welcome for further discussion: yumei.chen@linecorp.com