The internal of LINE Official Account's advanced broadcast

The internal of LINE Official Account's advanced broadcast

Haruki Okada
LINE OA Dev Team1 Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/B2-2

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 21, 2019
Tweet

Transcript

  1. 2019 DevDay The Internal of LINE Official Account's Advanced Broadcast

    > Haruki Okada > LINE OA Dev Team1 Software Engineer
  2. LINE Official Account

  3. > Businesses can create their own accounts on LINE >

    Communication with their customers LINE Official Account
  4. LINE Official Account Accounts Globally 11M Estimated
 Peak Event Traffic

    500K / Sec
  5. LINE Official Account Manager > Create and send various types

    of message > Show statistics of sent messages > 1 : 1 Chat > And so on
  6. Broadcasting

  7. • Campaign promotion • Release information of a new product

    • Send a coupon • And so on Broadcasting > Most widely used feature of LINE Official Account
  8. Message Insights

  9. Renewal Needed > We started to provide new pricing plan

    as part of “Redesign” project • For user: Get more valuable information • For businesses: Reach appropriate users > Needed to update message insights • Update timing: daily => realtime • Verbosity: only basic metrics => more detailed metrics
  10. Renewed in Sep 2019

  11. Message Insights Renewal

  12. > These metrics are updated in realtime > Detailed User

    Interaction Metrics • Number of people who viewed the message (unique impression) • Number of people who clicked the link in the message (unique click) • Total impression count of the message • And so on Message Insights Renewal
  13. > ✅ User reactions for broadcasted messages from LINE Official

    Account Target Events of Message Insights > ❌ User reactions for ordinary (≠ LINE Official Account) LINE messaging • e.g. ordinary “Mark as Read”
  14. A/B Testing > Businesses can send multiple message variants at

    once > Analyze user responses in realtime
  15. New Event Tracking System
 Needed

  16. > Update metrics in realtime > Maintain both of realtime-ness

    and persistency > Track events without impacting UX Challenges > Large traffic of message events
  17. Architecture Overview 1. Send Messages 2. Track Events 3. Stream

    Processing 4. Batch Processing
  18. Agenda > The Internal of 
 Message Event Tracking System

    > Future
  19. Broadcasting - Send Messages

  20. Send Messages LINE Official Account Manager MySQL Message Delivery Server

    Messaging API Schedule Poll
  21. Send Messages Messaging API Metadata MySQL Talk Server LINE Official

    Account Friends …
  22. > Possible architecture • 1. Embedding metadata in message itself,

    then send back when track an user reaction event • 2. Separate metadata from message itself. Streaming join later Send Messages > Message metadata contains information necessary for aggregation
  23. > Why ? • To minimize network usage so that

    never affecting UX Send Messages > We adopt 2. Separate metadata from message itself > Cons • The architecture becomes complicated a bit
  24. Track Events > Event Tracking SDK collects detailed user events

    • Impression of balloon 1,2,… • Click of url X in balloon 1,… • And so on Balloon 1 Balloon 2 Balloon 3 1 Broadcast
  25. Track Events LINE Official Account Friends Event Tracker Kafka Spark

    Streaming HDFS LINE Ads Platform Data Pipeline
  26. Stream Processing Kafka Event Processor Metadata MySQL Redis
 Cluster HBase

    Consume Join Realtime Data Raw Event Log
  27. Joining Metadata Event Processor Metadata MySQL Buffer (LinkedBlockingQueue) Event Event

    … Lookup Multiple Metadata
  28. Store Realtime Data To Redis Redis
 Cluster Event Processor Fetch

    Impression Count per Balloon Impression UU Per Broadcast …
  29. Counting Unique User > “Count-Distinct” problem > Generally, calculating count-distinct

    requires O(N) space > The space may not fit on memory for massive dataset
  30. HyperLogLog > A probabilistic algorithm solving “Count-Distinct” problem > Redis

    provides HyperLogLog via PFxxx command family • PFADD: Add an element to HyperLogLog data structure • PFCOUNT: Estimate distinct count • PFMERGE: Merge two HyperLogLog data structures into one
  31. Lua Scripting INCR {Balloon1_Impression} PFADD {Broadcast1_Impression} EXPIRE {Balloon1_Impression} {TTL} EXPIRE

    {Broadcast1_Impression} {TTL} …. Redis
 Cluster EVALSHA 1 Impression Event
  32. Store Raw Event Log to HBase Timestamp EventType UserId AccountId

    …. HBase PUT Raw Event Log Batch Processing Retargeting Realtime Data
 Recovery
  33. Retry Queueing Event Processor Kafka HBase RegionTooBusyException
 FAIL Queueing Reprocess

    After Backoff
  34. Batch Processing HDFS Spark Batch Raw Event Log Table Unique

    User Metrics Event Count Metrics HBase
 MapReduce Scan Put Import HBase
  35. HBase MapReduce Row Column Column impression_broadcastId1 userId1 … impression_broadcastId2 userId42

    userId47 impression_broadcastId3 userId55301 … … Raw Event Log Mapper
 Reducer
 MultiTableOutputFormat
 Count Columns Sum Up
  36. Gather Realtime Data and Batch Data Redis
 Cluster HBase Official

    Account Manager Realtime Batch
  37. Gather Realtime Data and Batch Data 2019-11-15 (Message Sent Date)

    2019-11-16 2019-11-17 2019-11-18 2019-11-19 (Last Batch Date) HBase Redis
 Cluster 2019-11-20 (Realtime) Sum Up
  38. Architecture Overview 1. Send Messages 2. Track Events 3. Stream

    Processing 4. Batch Processing
  39. > Provide message retargeting Future Plans > Integration with LINE

    Ads Platform > Look alike targeting
  40. Message Retargeting HBase Kafka LINE DMP User Segment Storage Segment

    Master SegmentA:
 Users Who Opened Message 1
 SegmentB:
 ….
  41. Integration With LINE Ads Platform LINE Ads Platform Message Event

    Tracking LINE NEWS, Timeline, Smart Channel,…
  42. Look Alike LINE Ads Platform Message Event Tracking LINE DMP

    Look Alike Engine
  43. > More new features are planned based on our system

    > We have built scalable, robust message event tracking system Conclusion
  44. Thank You