Slide 1
Slide 1 text
2019 DevDay
The Internal of LINE Official
Account's Advanced Broadcast
> Haruki Okada
> LINE OA Dev Team1 Software Engineer
Slide 2
Slide 2 text
LINE Official Account
Slide 3
Slide 3 text
> Businesses can create their own
accounts on LINE
> Communication with their customers
LINE Official Account
Slide 4
Slide 4 text
LINE Official Account
Accounts Globally
11M
Estimated
Peak Event Traffic
500K / Sec
Slide 5
Slide 5 text
LINE Official Account Manager
> Create and send various types of
message
> Show statistics of sent messages
> 1 : 1 Chat
> And so on
Slide 6
Slide 6 text
Broadcasting
Slide 7
Slide 7 text
• Campaign promotion
• Release information of a new product
• Send a coupon
• And so on
Broadcasting
> Most widely used feature of LINE Official Account
Slide 8
Slide 8 text
Message Insights
Slide 9
Slide 9 text
Renewal Needed
> We started to provide new pricing plan as part of “Redesign” project
• For user: Get more valuable information
• For businesses: Reach appropriate users
> Needed to update message insights
• Update timing: daily => realtime
• Verbosity: only basic metrics => more detailed metrics
Slide 10
Slide 10 text
Renewed in Sep 2019
Slide 11
Slide 11 text
Message Insights Renewal
Slide 12
Slide 12 text
> These metrics are updated in realtime
> Detailed User Interaction Metrics
• Number of people who viewed the message (unique impression)
• Number of people who clicked the link in the message (unique click)
• Total impression count of the message
• And so on
Message Insights Renewal
Slide 13
Slide 13 text
> ✅ User reactions for broadcasted messages from LINE Official Account
Target Events of Message Insights
> ❌ User reactions for ordinary (≠ LINE Official Account) LINE messaging
• e.g. ordinary “Mark as Read”
Slide 14
Slide 14 text
A/B Testing
> Businesses can send multiple
message variants at once
> Analyze user responses in
realtime
Slide 15
Slide 15 text
New Event Tracking System
Needed
Slide 16
Slide 16 text
> Update metrics in realtime
> Maintain both of realtime-ness and persistency
> Track events without impacting UX
Challenges
> Large traffic of message events
Slide 17
Slide 17 text
Architecture Overview
1. Send Messages
2. Track Events
3. Stream Processing
4. Batch Processing
Slide 18
Slide 18 text
Agenda > The Internal of
Message Event Tracking System
> Future
Slide 19
Slide 19 text
Broadcasting - Send Messages
Slide 20
Slide 20 text
Send Messages
LINE Official
Account
Manager
MySQL
Message
Delivery Server
Messaging API
Schedule Poll
Slide 21
Slide 21 text
Send Messages
Messaging API
Metadata
MySQL
Talk Server
LINE Official Account Friends
…
Slide 22
Slide 22 text
> Possible architecture
• 1. Embedding metadata in message itself, then send back when track an
user reaction event
• 2. Separate metadata from message itself. Streaming join later
Send Messages
> Message metadata contains information necessary for aggregation
Slide 23
Slide 23 text
> Why ?
• To minimize network usage so that never affecting UX
Send Messages
> We adopt 2. Separate metadata from message itself
> Cons
• The architecture becomes complicated a bit
Slide 24
Slide 24 text
Track Events
> Event Tracking SDK collects
detailed user events
• Impression of balloon 1,2,…
• Click of url X in balloon 1,…
• And so on
Balloon 1
Balloon 2
Balloon 3
1 Broadcast
Slide 25
Slide 25 text
Track Events
LINE Official Account Friends
Event Tracker
Kafka
Spark
Streaming
HDFS
LINE Ads Platform Data Pipeline
Slide 26
Slide 26 text
Stream Processing
Kafka Event
Processor
Metadata
MySQL
Redis
Cluster
HBase
Consume
Join
Realtime Data
Raw Event Log
Slide 27
Slide 27 text
Joining Metadata
Event Processor
Metadata
MySQL
Buffer (LinkedBlockingQueue)
Event Event …
Lookup
Multiple Metadata
Slide 28
Slide 28 text
Store Realtime Data To Redis
Redis
Cluster
Event
Processor
Fetch
Impression Count per Balloon
Impression UU Per Broadcast
…
Slide 29
Slide 29 text
Counting Unique User
> “Count-Distinct” problem
> Generally, calculating count-distinct requires O(N) space
> The space may not fit on memory for massive dataset
Slide 30
Slide 30 text
HyperLogLog
> A probabilistic algorithm solving “Count-Distinct” problem
> Redis provides HyperLogLog via PFxxx command family
• PFADD: Add an element to HyperLogLog data structure
• PFCOUNT: Estimate distinct count
• PFMERGE: Merge two HyperLogLog data structures into one
Slide 31
Slide 31 text
Lua Scripting
INCR {Balloon1_Impression}
PFADD {Broadcast1_Impression}
EXPIRE {Balloon1_Impression} {TTL}
EXPIRE {Broadcast1_Impression} {TTL}
….
Redis
Cluster
EVALSHA
1 Impression Event
Slide 32
Slide 32 text
Store Raw Event Log to HBase
Timestamp
EventType
UserId
AccountId
….
HBase
PUT
Raw Event Log
Batch Processing
Retargeting
Realtime Data
Recovery
Slide 33
Slide 33 text
Retry Queueing
Event
Processor
Kafka
HBase
RegionTooBusyException
FAIL
Queueing
Reprocess After Backoff
Slide 34
Slide 34 text
Batch Processing
HDFS
Spark Batch
Raw Event Log Table
Unique User Metrics
Event Count Metrics
HBase
MapReduce
Scan
Put
Import
HBase
Slide 35
Slide 35 text
HBase MapReduce
Row Column Column
impression_broadcastId1 userId1 …
impression_broadcastId2 userId42 userId47
impression_broadcastId3 userId55301 …
…
Raw Event Log
Mapper
Reducer
MultiTableOutputFormat
Count Columns
Sum Up
Slide 36
Slide 36 text
Gather Realtime Data and Batch Data
Redis
Cluster
HBase
Official Account Manager
Realtime
Batch
Slide 37
Slide 37 text
Gather Realtime Data and Batch Data
2019-11-15 (Message Sent Date)
2019-11-16
2019-11-17
2019-11-18
2019-11-19 (Last Batch Date)
HBase
Redis
Cluster
2019-11-20 (Realtime)
Sum Up
Slide 38
Slide 38 text
Architecture Overview
1. Send Messages
2. Track Events
3. Stream Processing
4. Batch Processing
Slide 39
Slide 39 text
> Provide message retargeting
Future Plans
> Integration with LINE Ads Platform
> Look alike targeting
Slide 40
Slide 40 text
Message Retargeting
HBase
Kafka
LINE DMP
User
Segment
Storage
Segment
Master
SegmentA:
Users Who Opened Message 1
SegmentB:
….
Slide 41
Slide 41 text
Integration With LINE Ads Platform
LINE Ads Platform
Message Event Tracking
LINE NEWS, Timeline, Smart Channel,…
Slide 42
Slide 42 text
Look Alike
LINE Ads Platform
Message Event Tracking
LINE DMP
Look Alike
Engine
Slide 43
Slide 43 text
> More new features are planned based on our system
> We have built scalable, robust message event tracking system
Conclusion
Slide 44
Slide 44 text
Thank You