$30 off During Our Annual Pro Sale. View Details »

Efficient integrating data from multiple data providers

Efficient integrating data from multiple data providers

Yu-Mei Chen
LINE Taiwan Backend Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S2-4

LINE DevDay 2019

November 21, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay
    Efficient Integrating Data From
    Multiple Data Providers
    > Yu-Mei Chen
    > LINE Taiwan Backend Software Engineer

    View Slide

  2. Subtitle

    View Slide

  3. Partner A
    LINE SPOT
    Users
    Partner B
    Partner C

    $

    View Slide

  4. Practices From Cooperating with Partners
    > Open-Closed Principle

    There are no permanent enemies, and no permanent friends.
    > Generalization & Customization

    Fighting for general data structures,

    but still keep a flexible layer for customized.
    > Processing Data Manually

    Always Preparing the manual way to update the data,

    such as updating data by import the .csv files.

    View Slide

  5. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn
    Its All About “Maintainability”
    Operability Evolvability
    Simplicity

    View Slide

  6. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn
    Its All About “Maintainability”
    Processing Data Manually
    Operability
    Generalization & Customization
    Evolvability
    Simplicity
    Open-Closed Principle

    View Slide

  7. Our Solution: Event Sourcing
    A Cloud Design Pattern for Backend Architecture

    View Slide

  8. Event Sourcing With Kafka
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store

    View Slide

  9. Processing Data Manually
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store

    View Slide

  10. Open-Closed Principle
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store

    View Slide

  11. Generalization & Customization
    Apache Kafka
    Partner A
    Stream
    Store
    Service
    Coupon
    Service
    Voucher
    Service
    Store

    Locator
    Kafka
    Connector
    Kafka
    Connector
    Kafka
    Connector
    Partner B
    Stream
    Partner C
    Stream
    … Stream
    Partner
    Dispatcher
    Mysql
    POI DB
    Mysql
    Coupon DB
    Mysql
    Voucher DB
    Elasticsearch
    Materialize

    View for

    Store

    View Slide

  12. Summary
    > Challenge in LINE SPOT: Maintainability

    > Solution: Event Sourcing
    Thank You for Listening
    And welcome for further discussion:
    [email protected]

    View Slide