The internal of LINE Official Account's advanced broadcast

Haruki Okada
LINE OA Dev Team1 Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/B2-2

Transcript

1. 2019 DevDay
   The Internal of LINE Official
   Account's Advanced Broadcast
   > Haruki Okada
   > LINE OA Dev Team1 Software Engineer
   

   View Slide

2. LINE Official Account
   

   View Slide

3. > Businesses can create their own
   accounts on LINE
   > Communication with their customers
   LINE Official Account
   

   View Slide

4. LINE Official Account
   Accounts Globally
   11M
   Estimated

   Peak Event Traffic
   500K / Sec
   

   View Slide

5. LINE Official Account Manager
   > Create and send various types of
   message
   > Show statistics of sent messages
   > 1 : 1 Chat
   > And so on
   

   View Slide

6. Broadcasting
   

   View Slide

7. • Campaign promotion
   • Release information of a new product
   • Send a coupon
   • And so on
   Broadcasting
   > Most widely used feature of LINE Official Account
   

   View Slide

8. Message Insights
   

   View Slide

9. Renewal Needed
   > We started to provide new pricing plan as part of “Redesign” project
   • For user: Get more valuable information
   • For businesses: Reach appropriate users
   > Needed to update message insights
   • Update timing: daily => realtime
   • Verbosity: only basic metrics => more detailed metrics
   

   View Slide

10. Renewed in Sep 2019
    

    View Slide

11. Message Insights Renewal
    

    View Slide

12. > These metrics are updated in realtime
    > Detailed User Interaction Metrics
    • Number of people who viewed the message (unique impression)
    • Number of people who clicked the link in the message (unique click)
    • Total impression count of the message
    • And so on
    Message Insights Renewal
    

    View Slide

13. > ✅ User reactions for broadcasted messages from LINE Official Account
    Target Events of Message Insights
    > ❌ User reactions for ordinary (≠ LINE Official Account) LINE messaging
    • e.g. ordinary “Mark as Read”
    

    View Slide

14. A/B Testing
    > Businesses can send multiple
    message variants at once
    > Analyze user responses in
    realtime
    

    View Slide

15. New Event Tracking System

    Needed
    

    View Slide

16. > Update metrics in realtime
    > Maintain both of realtime-ness and persistency
    > Track events without impacting UX
    Challenges
    > Large traffic of message events
    

    View Slide

17. Architecture Overview
    1. Send Messages
    2. Track Events
    3. Stream Processing
    4. Batch Processing
    

    View Slide

18. Agenda > The Internal of 

    Message Event Tracking System
    > Future
    

    View Slide

19. Broadcasting - Send Messages
    

    View Slide

20. Send Messages
    LINE Official
    Account
    Manager
    MySQL
    Message
    Delivery Server
    Messaging API
    Schedule Poll
    

    View Slide

21. Send Messages
    Messaging API
    Metadata
    MySQL
    Talk Server
    LINE Official Account Friends
    …
    

    View Slide

22. > Possible architecture
    • 1. Embedding metadata in message itself, then send back when track an
    user reaction event
    • 2. Separate metadata from message itself. Streaming join later
    Send Messages
    > Message metadata contains information necessary for aggregation
    

    View Slide

23. > Why ?
    • To minimize network usage so that never affecting UX
    Send Messages
    > We adopt 2. Separate metadata from message itself
    > Cons
    • The architecture becomes complicated a bit
    

    View Slide

24. Track Events
    > Event Tracking SDK collects
    detailed user events
    • Impression of balloon 1,2,…
    • Click of url X in balloon 1,…
    • And so on
    Balloon 1
    Balloon 2
    Balloon 3
    1 Broadcast
    

    View Slide

25. Track Events
    LINE Official Account Friends
    Event Tracker
    Kafka
    Spark
    Streaming
    HDFS
    LINE Ads Platform Data Pipeline
    

    View Slide

26. Stream Processing
    Kafka Event
    Processor
    Metadata
    MySQL
    Redis

    Cluster
    HBase
    Consume
    Join
    Realtime Data
    Raw Event Log
    

    View Slide

27. Joining Metadata
    Event Processor
    Metadata
    MySQL
    Buffer (LinkedBlockingQueue)
    Event Event …
    Lookup
    Multiple Metadata
    

    View Slide

28. Store Realtime Data To Redis
    Redis

    Cluster
    Event
    Processor
    Fetch
    Impression Count per Balloon
    Impression UU Per Broadcast
    …
    

    View Slide

29. Counting Unique User
    > “Count-Distinct” problem
    > Generally, calculating count-distinct requires O(N) space
    > The space may not fit on memory for massive dataset
    

    View Slide

30. HyperLogLog
    > A probabilistic algorithm solving “Count-Distinct” problem
    > Redis provides HyperLogLog via PFxxx command family
    • PFADD: Add an element to HyperLogLog data structure
    • PFCOUNT: Estimate distinct count
    • PFMERGE: Merge two HyperLogLog data structures into one
    

    View Slide

31. Lua Scripting
    INCR {Balloon1_Impression}
    PFADD {Broadcast1_Impression}
    EXPIRE {Balloon1_Impression} {TTL}
    EXPIRE {Broadcast1_Impression} {TTL}
    ….
    Redis

    Cluster
    EVALSHA
    1 Impression Event
    

    View Slide

32. Store Raw Event Log to HBase
    Timestamp
    EventType
    UserId
    AccountId
    ….
    HBase
    PUT
    Raw Event Log
    Batch Processing
    Retargeting
    Realtime Data

    Recovery
    

    View Slide

33. Retry Queueing
    Event
    Processor
    Kafka
    HBase
    RegionTooBusyException

    FAIL
    Queueing
    Reprocess After Backoff
    

    View Slide

34. Batch Processing
    HDFS
    Spark Batch
    Raw Event Log Table
    Unique User Metrics
    Event Count Metrics
    HBase

    MapReduce
    Scan
    Put
    Import
    HBase
    

    View Slide

35. HBase MapReduce
    Row Column Column
    impression_broadcastId1 userId1 …
    impression_broadcastId2 userId42 userId47
    impression_broadcastId3 userId55301 …
    …
    Raw Event Log
    Mapper

    Reducer

    MultiTableOutputFormat

    Count Columns
    Sum Up
    

    View Slide

36. Gather Realtime Data and Batch Data
    Redis

    Cluster
    HBase
    Official Account Manager
    Realtime
    Batch
    

    View Slide

37. Gather Realtime Data and Batch Data
    2019-11-15 (Message Sent Date)
    2019-11-16
    2019-11-17
    2019-11-18
    2019-11-19 (Last Batch Date)
    HBase
    Redis

    Cluster
    2019-11-20 (Realtime)
    Sum Up
    

    View Slide

38. Architecture Overview
    1. Send Messages
    2. Track Events
    3. Stream Processing
    4. Batch Processing
    

    View Slide

39. > Provide message retargeting
    Future Plans
    > Integration with LINE Ads Platform
    > Look alike targeting
    

    View Slide

40. Message Retargeting
    HBase
    Kafka
    LINE DMP
    User
    Segment
    Storage
    Segment
    Master
    SegmentA:

    Users Who Opened Message 1

    SegmentB:

    ….
    

    View Slide

41. Integration With LINE Ads Platform
    LINE Ads Platform
    Message Event Tracking
    LINE NEWS, Timeline, Smart Channel,…
    

    View Slide

42. Look Alike
    LINE Ads Platform
    Message Event Tracking
    LINE DMP
    Look Alike
    Engine
    

    View Slide

43. > More new features are planned based on our system
    > We have built scalable, robust message event tracking system
    Conclusion
    

    View Slide

44. Thank You
    

    View Slide