Efficient integrating data from multiple data providers

Yu-Mei Chen
LINE Taiwan Backend Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S2-4

Transcript

1. 2019 DevDay
   Efficient Integrating Data From
   Multiple Data Providers
   > Yu-Mei Chen
   > LINE Taiwan Backend Software Engineer
   

   View Slide

2. Subtitle
   

   View Slide

3. Partner A
   LINE SPOT
   Users
   Partner B
   Partner C
   …
   $
   

   View Slide

4. Practices From Cooperating with Partners
   > Open-Closed Principle

   There are no permanent enemies, and no permanent friends.
   > Generalization & Customization

   Fighting for general data structures,

   but still keep a flexible layer for customized.
   > Processing Data Manually

   Always Preparing the manual way to update the data,

   such as updating data by import the .csv files.
   

   View Slide

5. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn
   Its All About “Maintainability”
   Operability Evolvability
   Simplicity
   

   View Slide

6. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn
   Its All About “Maintainability”
   Processing Data Manually
   Operability
   Generalization & Customization
   Evolvability
   Simplicity
   Open-Closed Principle
   

   View Slide

7. Our Solution: Event Sourcing
   A Cloud Design Pattern for Backend Architecture
   

   View Slide

8. Event Sourcing With Kafka
   Apache Kafka
   Partner A
   Stream
   Store
   Service
   Coupon
   Service
   Voucher
   Service
   Store

   Locator
   Kafka
   Connector
   Kafka
   Connector
   Kafka
   Connector
   Partner B
   Stream
   Partner C
   Stream
   … Stream
   Partner
   Dispatcher
   Mysql
   POI DB
   Mysql
   Coupon DB
   Mysql
   Voucher DB
   Elasticsearch
   Materialize

   View for

   Store
   

   View Slide

9. Processing Data Manually
   Apache Kafka
   Partner A
   Stream
   Store
   Service
   Coupon
   Service
   Voucher
   Service
   Store

   Locator
   Kafka
   Connector
   Kafka
   Connector
   Kafka
   Connector
   Partner B
   Stream
   Partner C
   Stream
   … Stream
   Partner
   Dispatcher
   Mysql
   POI DB
   Mysql
   Coupon DB
   Mysql
   Voucher DB
   Elasticsearch
   Materialize

   View for

   Store
   

   View Slide

10. Open-Closed Principle
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store
    

    View Slide

11. Generalization & Customization
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store
    

    View Slide

12. Summary
    > Challenge in LINE SPOT: Maintainability

    > Solution: Event Sourcing
    Thank You for Listening
    And welcome for further discussion:
    yumei.chen@linecorp.com
    

    View Slide