How we deal with large traffic of LINE Official Account in realtime

Yusuke Todaka
LINE LINE OA Dev 2 Team Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S1-14

Transcript

1. 2019 DevDay How We Deal With Large Traffic of LINE Official Account in Realtime > Yusuke Todaka > LINE LINE OA Dev 2 Team Software Engineer
2. > Server Side Engineer > Developing LINE Official Account / LINE Bot SDK Ruby > Developer Center 4, OA Dev 1 team Todaka Yusuke
3. Ƃ LINE Official Account Manager https://manager.line.biz
4. Ƃ Event Tracking System
5. Ƃ Retargeting With User Action impression / click
6. Agenda > What Is Broadcast Targeting Estimation > Technical Difficulties > Solution - KMV Sketch > Accuracy & Benchmark Love Realtime,
 Love Approximation.
7. Ƃ Target Estimation impression / click
8. Large Set Operation in Sub-Second for UX > Complex set operations : Combination of Union, Exclude, Filtering > Everything should be done in less than 1 second for UX > Large User Group Size : ~ 100M users Technical Difficulties
9. Kth Minimum Values Sketch Solution : KMV Sketch > Distinct Counting Algorithm > Sampling With Hash Function > Supports Union, Intersection, Difference Operation
 > Use More Space Than HyperLogLog (~800KB for 60K Nominal Entries)
10. https://datasketches.github.io/ Implementation : Apache DataSketches
11. Architecture Events Sketch
 Storage Sketch Calculator Server Target Estimator Server
12. https://datasketches.github.io/docs/Theta/ThetaErrorTable.html 1 RSE (%) 0.01 0.1 1 10 Nominal Entries 2^8 2^12 2^16 2^20 2^24 RSE Accuracy vs Nominal Entries
13. Intersection Operation of 2 Sets. Read From Local File. Nominal Entries = 65K. Time (ms) 1 10 100 1000 10000 Entries 1K 10K 100K 1M 10M HashSet KMV Performance Benchmark https://github.com/toduq/theta-sketch-benchmark
14. Thank You