How we deal with large traffic of LINE Official Account in realtime

How we deal with large traffic of LINE Official Account in realtime

Yusuke Todaka
LINE LINE OA Dev 2 Team Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S1-14

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay How We Deal With Large Traffic of LINE

    Official Account in Realtime > Yusuke Todaka > LINE LINE OA Dev 2 Team Software Engineer
  2. > Server Side Engineer > Developing LINE Official Account /

    LINE Bot SDK Ruby > Developer Center 4, OA Dev 1 team Todaka Yusuke
  3. Ƃ LINE Official Account Manager https://manager.line.biz

  4. Ƃ Event Tracking System

  5. Ƃ Retargeting With User Action impression / click

  6. Agenda > What Is Broadcast Targeting Estimation > Technical Difficulties

    > Solution - KMV Sketch > Accuracy & Benchmark Love Realtime,
 Love Approximation.
  7. Ƃ Target Estimation impression / click

  8. Large Set Operation in Sub-Second for UX > Complex set

    operations : Combination of Union, Exclude, Filtering > Everything should be done in less than 1 second for UX > Large User Group Size : ~ 100M users Technical Difficulties
  9. Kth Minimum Values Sketch Solution : KMV Sketch > Distinct

    Counting Algorithm > Sampling With Hash Function > Supports Union, Intersection, Difference Operation
 > Use More Space Than HyperLogLog (~800KB for 60K Nominal Entries)
  10. https://datasketches.github.io/ Implementation : Apache DataSketches

  11. Architecture Events Sketch
 Storage Sketch Calculator Server Target Estimator Server

  12. https://datasketches.github.io/docs/Theta/ThetaErrorTable.html 1 RSE (%) 0.01 0.1 1 10 Nominal Entries

    2^8 2^12 2^16 2^20 2^24 RSE Accuracy vs Nominal Entries
  13. Intersection Operation of 2 Sets. Read From Local File. Nominal

    Entries = 65K. Time (ms) 1 10 100 1000 10000 Entries 1K 10K 100K 1M 10M HashSet KMV Performance Benchmark https://github.com/toduq/theta-sketch-benchmark
  14. Thank You