Stream Processing in LINE SPOT

Stream Processing in LINE SPOT

2019/07/31 LINE Developer Meetup 開發者小聚系列活動 #9
https://linegroup.kktix.cc/events/20190731

2102a6b8760bd6f57f672805723dd83a?s=128

line_developers_tw

July 31, 2019
Tweet

Transcript

  1. None
  2. 4USFBN1SPDFTTJOHJO-*/&4105 Julian Shen

  3. What’s it about? 

  4. .FEJB 4USFBNJOH #JH%BUB #JH%BUB (BNF Steam

  5. • .JDSPTFSWJDFT "SDIJUFDUVSF • &WFOU4PVSDJOH • $234 $PNNBOE2VFSZ3FTQPOTJCJMJUZ4FHSFHBUJPO

  6. Background   

  7. 8FBSF-*/&41055FBN

  8. None
  9. 8IBUT-*/&4105 10* 4FSWJD FT 4PDJBM $POUF OU 'PPE EFMJWFSZ 3FTFSWBUJPO

    5BYJ %FTJHOBUFE ESJWFS "SUJDMFT 3BOLJOHT &WFOUT 1SPNPUJPO T 6($ 'PMMPX
  10. Sounds promising !

  11. • %JGGFSFOULJOEPGEBUBGSPNEJGGFSFOUQMBDFT • -*/&JOUFSOBM • 1BSUOFST • 4UPSF0XOFST • /PSNBM6TFST

    • %JGGFSFOULJOEPGRVFSJFTGJMUFST • ,FZXPSE GVMMUFYUTFBSDI • #ZMPDBUJPO • #ZSBUJOH 8IBUXFOFFEUPQSPDFTT
  12. Getting back on topic 

  13. Let’s start from a simple example  

  14.     

  15. Thinking what’s your backend design for this?

  16. Web App API Server(s) API Server(s) API Server(s) API Server(s)

    API Server(s) Reverse proxy/API Gateway DB Let’s start from an old school way DB write read replication
  17. User Following Article Like View Share Comment

  18. • 8IPUIJTVTFSGPMMPXT 'PMMPXUBCMF • 8IBUUIFZQPTUFE "SUJDMFUBCMF • 6TFSOBNF QSPGJMFJNBHF 6TFSUBCMF

    • 8IPMJLFTMJLFDPVOU -JLFUBCMF • 8IPWJFXFEWJFXDPVOU 7JFXUBCMF • 8IPDPNNFOUFEDPNNFOUDPVOU $PNNFOU UBCMF 8IBUXFOFFEUPDPNQPTFUIFQBHF
  19. What your queries would like? Which DBMS you would pick?

    How many concurrent users to serve? How to cache the data? How you join data? What if user follows other 1000 users?
  20. Easy? or Complicated? 

  21. Web App Reverse proxy/API Gateway DB How about in Microservice

    world (1st try) Timeline service User service Comment/Like service Article service DB DB User & Follower Article Comment/Like Separation of Concerns But… Problem still unsolved
  22. What if … We could use one single query to

    get all data we need We join and prepare data before we need it But how?
  23. Web App Reverse proxy/API Gateway DB Timeline service User service

    Comment/Like service Article service DB DB User & Follower Article Comment /Like Messaging System publish Event consumer Materialized view consume
  24. How we do it?

  25. POI service Coupon service Product service POI DB Kafka Connect

    Coupon DB Product DB Kafka Connect Kafka Connect Store Locator Service Kafka Stream View DB Kafka Connect Process store view data store view data Elastic Search MySQL MySQL MySQL
  26. val stores: KStream[Long, Store] = storeDetails .mapValues(withOpeningHours _) .mapValues(withPhone _)

    .leftJoin(couponsByStore) { case (details, StoreCoupons(items)) => Store(details, images = List.empty, coupons = items) case (details, null) => Store(details, images = List.empty, coupons = List.empty) } .leftJoin(imagesByStore) { case (store, StoreImages(items)) => store.copy(images = items.sortBy(_.ordinal)) case (store, null) => store } .toStream .mapValues( store => if (store == null || store.details.deleted) { null } else { store } ) &YBNQMF KTable
  27. • 4FSWFEBUBXJUIPVUDPNQMJDBUFRVFSJFT • 3FDPWFSEBUBCZIJTUPSZ • &BTZUPNJHSBUFUPEJGGFSFOU%#TPMVUJPO • "QQMZSJHIUTPMVUJPOUPSJHIUEBUB

  28. What’s bad? 

  29. • %JGGFSFOUEFTJHOUIJOLJOH • /PUGSPNEBUBQFSTQFDUJWF • %BUBNJHIUOPUTZODJOSFBMUJNF FWFOUVBM DPOTJTUFODZ • %VQMJDBUFEEBUB

    • /FFEHPPENPOJUPSJOHNFDIBOJTN • /FFEUPNBJOUBJONVMUJQMFEJGGFSFOUTZTUFNT
  30. Thank You