Stream Processing in LINE SPOT

2019/07/31 LINE Developer Meetup 開發者小聚系列活動 #9
https://linegroup.kktix.cc/events/20190731

Transcript

2. 4USFBN
   1SPDFTTJOH
   JO
   -*/&
   4105 Julian Shen
3. What’s it about? 
   
4. .FEJB
   4USFBNJOH #JH
   %BUB #JH
   %BUB (BNF Steam
   
5. • .JDSPTFSWJDFT "SDIJUFDUVSF
   • &WFOU
   4PVSDJOH • $234
   $PNNBOE
   2VFSZ
   3FTQPOTJCJMJUZ
   4FHSFHBUJPO
6. Background  
   
7. 8F
   BSF
   -*/&
   4105
   5FBN
9. 8IBUT
   -*/&
   4105 10* 4FSWJD FT 4PDJBM $POUF OU 'PPE
   EFMJWFSZ 3FTFSWBUJPO 5BYJ %FTJHOBUFE
   ESJWFS "SUJDMFT 3BOLJOHT &WFOUT 1SPNPUJPO T 6($ 'PMMPX
10. Sounds promising !
11. • %JGGFSFOU
    LJOE
    PG
    EBUB
    GSPN
    EJGGFSFOU
    QMBDFT • -*/&
    JOUFSOBM • 1BSUOFST • 4UPSF
    0XOFST • /PSNBM
    6TFST • %JGGFSFOU
    LJOE
    PG
    RVFSJFTGJMUFST • ,FZXPSE
    GVMM
    UFYU
    TFBSDI • #Z
    MPDBUJPO • #Z
    SBUJOH 8IBU
    XF
    OFFE
    UP
    QSPDFTT
12. Getting back on topic 
    
13. Let’s start from a simple example 
     
14.  
     
    
15. Thinking what’s your backend design for this?
16. Web App API Server(s) API Server(s) API Server(s) API Server(s) API Server(s) Reverse proxy/API Gateway DB Let’s start from an old school way DB write read replication
17. User Following Article Like View Share Comment
18. • 8IP
    UIJT
    VTFS
    GPMMPXT
    'PMMPX
    UBCMF • 8IBU
    UIFZ
    QPTUFE
    "SUJDMF
    UBCMF • 6TFS
    OBNF
    QSPGJMF
    JNBHF
    6TFS
    UBCMF • 8IP
    MJLFT
    
    MJLF
    DPVOU -JLF
    UBCMF • 8IP
    WJFXFE
    
    WJFX
    DPVOU
    7JFX
    UBCMF • 8IP
    DPNNFOUFE
    
    DPNNFOU
    DPVOU
    $PNNFOU
    UBCMF 8IBU
    XF
    OFFE
    UP
    DPNQPTF
    UIF
    QBHF
19. What your queries would like? Which DBMS you would pick? How many concurrent users to serve? How to cache the data? How you join data? What if user follows other 1000 users?
20. Easy? or Complicated? 
    
21. Web App Reverse proxy/API Gateway DB How about in Microservice world (1st try) Timeline service User service Comment/Like service Article service DB DB User & Follower Article Comment/Like Separation of Concerns But… Problem still unsolved
22. What if … We could use one single query to get all data we need We join and prepare data before we need it But how?
23. Web App Reverse proxy/API Gateway DB Timeline service User service Comment/Like service Article service DB DB User & Follower Article Comment /Like Messaging System publish Event consumer Materialized view consume
24. How we do it?
25. POI service Coupon service Product service POI DB Kafka Connect Coupon DB Product DB Kafka Connect Kafka Connect Store Locator Service Kafka Stream View DB Kafka Connect Process store view data store view data Elastic Search MySQL MySQL MySQL
26. val stores: KStream[Long, Store] = storeDetails .mapValues(withOpeningHours _) .mapValues(withPhone _) .leftJoin(couponsByStore) { case (details, StoreCoupons(items)) => Store(details, images = List.empty, coupons = items) case (details, null) => Store(details, images = List.empty, coupons = List.empty) } .leftJoin(imagesByStore) { case (store, StoreImages(items)) => store.copy(images = items.sortBy(_.ordinal)) case (store, null) => store } .toStream .mapValues( store => if (store == null || store.details.deleted) { null } else { store } ) &YBNQMF KTable
27. • 4FSWF
    EBUB
    XJUIPVU
    DPNQMJDBUF
    RVFSJFT • 3FDPWFS
    EBUB
    CZ
    IJTUPSZ • &BTZ
    UP
    NJHSBUF
    UP
    EJGGFSFOU
    %#
    TPMVUJPO • "QQMZ
    SJHIU
    TPMVUJPO
    UP
    SJHIU
    EBUB
28. What’s bad? 
    
29. • %JGGFSFOU
    EFTJHO
    UIJOLJOH • /PU
    GSPN
    EBUB
    QFSTQFDUJWF • %BUB
    NJHIU
    OPU
    TZOD
    JO
    SFBM
    UJNF
    FWFOUVBM
    DPOTJTUFODZ • %VQMJDBUFE
    EBUB • /FFE
    HPPE
    NPOJUPSJOH
    NFDIBOJTN • /FFE
    UP
    NBJOUBJO
    NVMUJQMF
    EJGGFSFOU
    TZTUFNT
30. Thank You