Pro Yearly is on sale from $80 to $50! »

Efficient Integrating Data
 from Multiple Data Providers

Efficient Integrating Data
 from Multiple Data Providers

By Yumei Chen @LINE TECHPULSE 2019 https://techpulse.line.me/

2102a6b8760bd6f57f672805723dd83a?s=128

line_developers_tw

December 04, 2019
Tweet

Transcript

  1. None
  2. Efficient Integrating Data
 from Multiple Data Providers > Yu-Mei Chen

    / LINE TAIWAN
  3. Only Title / 80pt / Arial / Bold

  4. Partner A LINE SPOT Users Partner B … Partner C

  5. Practices From Cooperating With Partners > Processing Data Manually
 Always

    Preparing the manual way to update the data,
 such as updating data by import the .csv files. > Open-Closed Principle
 There are no permanent enemies, and no permanent friends. > Generalization & Customization
 Fighting for general data structures,
 but still keep a flexible layer for customized.
  6. Its All About “Maintainability” Reference: Chapter 1 of “Designing Data-Intensive

    Applications”, Author: Martin Kleppmenn Simplicity Evolvability Operability
  7. Its All About “Maintainability” Simplicity Evolvability Operability Reference: Chapter 1

    of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Processing Data Manually Open-Closed Principle Generalization & Customization
  8. Our Solution: Event Sourcing A Cloud Design Pattern for Backend

    Architecture
  9. Event Sourcing With Kafka Apache Kafka Partner A Stream Store

    Service Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
  10. Processing Data Manually Apache Kafka Partner A Stream Store Service

    Coupon Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
  11. Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon

    Service Voucher Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch
  12. Apache Kafka Partner A Stream Store Service Coupon Service Voucher

    Service Store Locator Kafka Connector Kafka Connector Kafka Connector Coupon DB Mysql Voucher DB Mysql POI DB Mysql Partner B Stream Partner C Stream … Stream Partner Dispatcher Materialize
 View for
 Store Elasticsearch Generalization & Customization
  13. Summary > Challenge in LINE SPOT: Maintainability > Solution: Event

    Sourcing
  14. Thank You