Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Efficient integrating data from multiple data providers

Efficient integrating data from multiple data providers

Yu-Mei Chen
LINE Taiwan Backend Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S2-4

LINE DevDay 2019

November 21, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay Efficient Integrating Data From Multiple Data Providers >

    Yu-Mei Chen > LINE Taiwan Backend Software Engineer
  2. Practices From Cooperating with Partners > Open-Closed Principle
 There are

    no permanent enemies, and no permanent friends. > Generalization & Customization
 Fighting for general data structures,
 but still keep a flexible layer for customized. > Processing Data Manually
 Always Preparing the manual way to update the data,
 such as updating data by import the .csv files.
  3. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn

    Its All About “Maintainability” Operability Evolvability Simplicity
  4. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn

    Its All About “Maintainability” Processing Data Manually Operability Generalization & Customization Evolvability Simplicity Open-Closed Principle
  5. Event Sourcing With Kafka Apache Kafka Partner A Stream Store

    Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  6. Processing Data Manually Apache Kafka Partner A Stream Store Service

    Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  7. Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon

    Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  8. Generalization & Customization Apache Kafka Partner A Stream Store Service

    Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
  9. Summary > Challenge in LINE SPOT: Maintainability
 > Solution: Event

    Sourcing Thank You for Listening And welcome for further discussion: [email protected]