A Case Study of Issues Arising from LINE’s User Persona Systems Revamping

by LINE DEVDAY 2021

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Self introduction Machine Learning Solution Team 2 Tetsuroh Watanabe, Ph.D. - User persona system - Estimating users’ demographic and psychographic profiles - Feature store system Machine learning engineer, in charge of - Data scientist @ video game companies - Accounting - Researcher - Evolutionary computing / Multi-agent simulation - Travel by train, plane, ship, bus - Anime / Game / Manga - Quiz club activities Past work experiences Hobbies

Slide 3

Slide 3 text

Agenda - Introduction to LINE’s user persona system - Improvement points identified in the revamping - Future works

Slide 4

Slide 4 text

Agenda - Introduction to LINE’s user persona system - Improvement points identified in the revamping - Future works

Slide 5

Slide 5 text

LINE’s user persona system Systems overview - Estimating users’ persona - By machine learning methods - With the service logs from users who have consented - Used in various LINE services - For delivering useful content to LINE user • Female • Age: 20-24 • Interests: - Music - Fashion • Male • Age: 40-44 • Interests: - Travel - Games - Entertainment Estimated users’ persona (example) Demographic and psychographic profiles

Slide 6

Slide 6 text

LINE’s user persona system [Use case-1] LINE Display Ads https://linebizid.com/id-en/service/line-display-ads - External AD owners can specify their AD target users. - By estimated user demographic segments

Slide 7

Slide 7 text

LINE’s user persona system [Use case-2] LINE Official Account: Push message targeting - External account owners can specify target users who will receive their push messages. - By estimated user demographic segments

Slide 8

Slide 8 text

Slide 9

Slide 9 text

LINE’s user persona system [Use case-3] New LINE services: Dealing with Cold Start Problems Service log Very limited since the service has just started Item recommendation for User A User A L Too little data to generate recommendations for User A… For you . . . ??? ??? ???

Slide 10

Slide 10 text

LINE’s user persona system [Use case-3] New LINE services: Dealing with Cold Start Problems Service log Very limited since the service has just started Item recommendation for User A User A Popular Female Age: 20-24 . . . Item A Item B Item C J Popular items tailored to User A can be recommended! User demographics • Female • Age: 20-24

Slide 11

Slide 11 text

LINE’s user persona system Region MAU (As of June 2021) Feature dimensions (As of September 2021) Japan 89M 4.8M+ Taiwan 50M 1.4M+ Thailand 21M 1.1M+ Indonesia 10M 0.4M+ System scale

Slide 12

Slide 12 text

Data for machine learning z-features: Feature store storing cross-service logs

Slide 13

Slide 13 text

Data for machine learning z-features: Feature store storing cross-service logs

Slide 14

Slide 14 text

Data for machine learning z-features: Feature store storing cross-service logs From logs of users who have consented,

Slide 15

Slide 15 text

Data for machine learning z-features: Feature store storing cross-service logs only permitted data will be collected across services,

Slide 16

Slide 16 text

Data for machine learning z-features: Feature store storing cross-service logs and stored as the feature store.

Slide 17

Slide 17 text

Data for machine learning Ground truth (answer label) data Results of questionnaire surveys of users who gave their consent

Slide 18

Slide 18 text

Data for machine learning Ground truth (answer label) data Developing another model - Auto-label AD categories from small human-labeled data Results of questionnaire surveys of users who gave their consent

Slide 19

Slide 19 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game”

Slide 20

Slide 20 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game” AD ML model Train data Train label

Slide 21

Slide 21 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game” Estimated: “Game” AD (not annotated) AD ML model Train data Train label Predict

Slide 22

Slide 22 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game” Estimated: “Game” AD (not annotated) Ground truth: “Interests: Game” AD ML model Train data Train label Predict Click (Tap)

Slide 23

Slide 23 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game” Estimated: “Game” User ML model feature data AD (not annotated) Ground truth: “Interests: Game” AD ML model Train data Train label Predict Click (Tap) Train data Train label

Slide 24

Slide 24 text

Data for machine learning Auto-label AD categories from small human-labeled data AD (annotated) Ground truth: “Game” Estimated: “Game” User ML model feature data AD (not annotated) Ground truth: “Interests: Game” Estimated: “Interests: Game” AD ML model Train data Train label Predict Click (Tap) Predict Train data Train label

Slide 25

Slide 25 text

System history Further improved! GPU cluster introduced MPI introduced ghee & Kubernetes cluster Theano & Mesos cluster Neural network from scratch 2014 2016 2020 2014 2020 2021

Slide 26

Slide 26 text

System history Further improved! GPU cluster introduced MPI introduced ghee & Kubernetes cluster Theano & Mesos cluster Neural network from scratch 2014 2016 2020 2014 2020 2021

Slide 27

Slide 27 text

System history Further improved! GPU cluster introduced MPI introduced ghee & Kubernetes cluster Theano & Mesos cluster Neural network from scratch 2014 2016 2020 2014 2020 2021

Slide 28

Slide 28 text

System revamping Legacy system - Mesos cluster - Theano

Slide 29

Slide 29 text

System revamping Legacy system - Mesos cluster - Theano - Difficulties - Maintenance - Due to system age - Updating models - Due to inability to use current mainstream libraries (such as PyTorch)

Slide 30

Slide 30 text

System revamping Legacy system Revamped system - GPU cluster - Kubernetes cluster - Distributed processing - Using in-house library “ghee” - Available to use major deep learning libraries (including PyTorch) - Mesos cluster - Theano - Difficulties - Maintenance - Due to system age - Updating models - Due to inability to use current mainstream libraries (such as PyTorch)

Slide 31

Slide 31 text

System revamping Legacy system Revamped system Identified more room for improvement! - Mesos cluster - Theano - Difficulties - Maintenance - Due to system age - Updating models - Due to inability to use current mainstream libraries (such as PyTorch) - GPU cluster - Kubernetes cluster - Distributed processing - Using in-house library “ghee” - Available to use major deep learning libraries (including PyTorch)

Slide 32

Slide 32 text

Agenda - Introduction to LINE’s user persona system - Improvement points identified in the revamping - Future works

Slide 33

Slide 33 text

Improvement points 1. Further improvement of ML models 3. Further improvement of data and model reusability 2. Systems for stable delivery and monitoring

Slide 34

Slide 34 text

Improvement points 1. Further improvement of ML models 3. Further improvement of data and model reusability 2. Systems for stable delivery and monitoring

Slide 35

Slide 35 text

1. Further improvement of ML models Separated embedding by service Embedding layer Simple DNN Output from service A Feature data B C …… model

Slide 36

Slide 36 text

1. Further improvement of ML models Separated embedding by service Embedding layer Simple DNN Output from service A Feature data B C …… model

Slide 37

Slide 37 text

1. Further improvement of ML models Separated embedding by service Embedding layer Simple DNN Output from service A Feature data Splitted embedding layer ResNet (CNN) Output B C …… …… from service A Feature data B C …… model model

Slide 38

Slide 38 text

Slide 39

Slide 39 text

1. Further improvement of ML models Implementing a state-of-the-art model from service A Feature data Splitted embedding layer MLP-mixer (state-of-the-art) Output B C …… …… Service dropout model Model accuracy 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Old Old Old New New New Segment type (# of segments) B (13) A (22) C (11)

Slide 40

Slide 40 text

Improvement points 1. Further improvement of ML models 3. Further improvement of data and model reusability 2. Systems for stable delivery and monitoring

Slide 41

Slide 41 text

2-1. System for stable delivery [Issue] Variation in segment volume leads to inconvenience for biz side Date Segment volume Segment A C B D [Example] AD / Official account message delivery using estimated demographic segments

Slide 42

Slide 42 text

2-1. System for stable delivery [Issue] Variation in segment volume leads to inconvenience for biz side Delivery target users Date Segment volume Segment A d d +1 C B D [Example] AD / Official account message delivery using estimated demographic segments %FDSFBTFECZ

Slide 43

Slide 43 text

2-1. System for stable delivery [Issue] Variation in segment volume leads to inconvenience for biz side Date Segment volume Segment A d d +1 C B D %FDSFBTFECZ "GUFSKVTUPOFEBZ &WFOUIPVHIUIF"%BDDPVOUPXOFS IBTOPUDIBOHFEUIFEFMJWFSZTFUUJOH [Example] AD / Official account message delivery using estimated demographic segments Delivery target users

Slide 44

Slide 44 text

2-1. System for stable delivery [Solution-1] Smoothing: Reduce daily fluctuations Date Segment volume

Slide 45

Slide 45 text

2-1. System for stable delivery [Solution-1] Smoothing: Reduce daily fluctuations Date Segment volume Date Segment volume Smoothing Moving average

Slide 46

Slide 46 text

2-1. System for stable delivery [Solution-1] Smoothing: Reduce daily fluctuations Date Segment volume Date Smoothing API Segment volume Developing “Smoothing API” - Standardizing processes - Simplifying implementation Moving average

Slide 47

Slide 47 text

2-1. System for stable delivery [Solution-2] Volume calibration Example (dummy data and settings) Original (after smoothing) 300,000 250,000 200,000 150,000 100,000 50,000 0 Date Segment volume

Slide 48

Slide 48 text

2-1. System for stable delivery [Solution-2] Volume calibration Original (after smoothing) Calibrated Example (dummy data and settings) 300,000 250,000 200,000 150,000 100,000 50,000 0 Date Segment volume

Slide 49

Slide 49 text

2-1. System for stable delivery [Solution-2] Volume calibration Original (after smoothing) Calibrated Suppresses too large fluctuations (20% in this case) Example (dummy data and settings) 300,000 250,000 200,000 150,000 100,000 50,000 0 Date Segment volume

Slide 50

Slide 50 text

2-1. System for stable delivery [Solution-2] Volume calibration Original (after smoothing) Calibrated Guarantees the lower limit (150,000 in this case) Example (dummy data and settings) 300,000 250,000 200,000 150,000 100,000 50,000 0 Date Segment volume

Slide 51

Slide 51 text

Slide 52

Slide 52 text

2-1. System for stable delivery [Solution-3] Gradual rollout: Release model updates gradually Date OLD model applied NEW model applied

Slide 53

Slide 53 text

2-1. System for stable delivery [Solution-3] Gradual rollout: Release model updates gradually Date Segment volume [Example of actual data] Date OLD model applied NEW model applied

Slide 54

Slide 54 text

2-1. System for stable delivery [Solution-3] Gradual rollout: Release model updates gradually Date Segment volume [Example of actual data] Gradual rollout Old model 100% New model 100% Date OLD model applied NEW model applied

Slide 55

Slide 55 text

2-2. System for output monitoring As a framework for MLOps Dashboard tool “OASIS”

Slide 56

Slide 56 text

2-2. System for output monitoring As a framework for MLOps MLOps monitoring system “Lupus” Dashboard tool “OASIS”

Slide 57

Slide 57 text

2-2. System for output monitoring As a framework for MLOps MLOps monitoring system “Lupus” Dashboard tool “OASIS” Session No. ML-2 “Lupus - A monitoring system for accelerating MLOps” Just after this session!

Slide 58

Slide 58 text

Improvement points 1. Further improvement of ML models 3. Further improvement of data and model reusability 2. Systems for stable supply and monitoring

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

3. Improvement of data & model reusability [Issue] Similar processes implemented in different ways Feature data Preprocess A Processed feature data A Train model A Ground truth A Task A Create dataset A Dataset A Model A Preprocess B Processed feature data B Train model B Ground truth B Task B Create dataset B Dataset B Model B Preprocess C Processed feature data C Train model C Ground truth C Task C Create dataset C Dataset C Model C

Slide 64

Slide 64 text

3. Improvement of data & model reusability [Solution-1] Data pipeline integration Feature data Train model A Ground truth A Task A Dataset A Model A Preprocess Processed feature data Train model B Ground truth B Task B Dataset B Model B Train model C Ground truth C Task C Dataset C Model C Create dataset A Create dataset B Create dataset C

Slide 65

Slide 65 text

Slide 66

Slide 66 text

3. Improvement of data & model reusability [Solution-2] ghee-models API: Further simplifying the use and modification of models Feature data Ground truth A Task A Dataset A Model A Preprocess Processed feature data Ground truth B Task B Dataset B Model B Ground truth C Task C Dataset C Model C Create dataset API ghee- models API “ghee-models” = LINE’s proprietary ML models

Slide 67

Slide 67 text

3. Improvement of data & model reusability [Solution-3] felib: Organizing preprocessing as a library Feature data Ground truth A Task A Dataset A Model A Preprocess Processed feature data Ground truth B Task B Dataset B Model B Ground truth C Task C Dataset C Model C Create dataset API . . . Util library “felib” Util function 1 Util function 2 ghee- models API

Slide 68

Slide 68 text

System history Further improved! GPU cluster introduced MPI introduced ghee & Kubernetes cluster Theano & Mesos cluster Neural network from scratch 2014 2016 2020 2014 2020 2021

Slide 69

Slide 69 text

Agenda - Introduction to LINE’s user persona systems - Improvement points identified in the revamping - Future works

Slide 70

Slide 70 text

Future works [1] ghee-models upgrade: Reduce implementation for adding new models Dataset Preprocess code α ML Method α Train code α Predict code α Postprocess code α Model α Output α AS-IS

Slide 71

Slide 71 text

Future works [1] ghee-models upgrade: Reduce implementation for adding new models Dataset Preprocess code α ML Method α Train code α Predict code α Postprocess code α Model α Output α Preprocess code β ML Method β Train code β Predict code β Postprocess code β Model β Output β Preprocess code γ ML Method γ Train code γ Predict code γ Postprocess code γ Model γ Output γ AS-IS

Slide 72

Slide 72 text

Future works [1] ghee-models upgrade: Reduce implementation for adding new models Dataset ML Method α Model α Output α ML Method β Model β Output β ML Method γ Model γ Output γ Preprocess core code Train core code Predict core code Postprocess core code For α For α For α For α For β For β For β For β For γ For γ For γ For γ TO-BE

Slide 73

Slide 73 text

Future works [2] Auto-persona: Enableing original segment estimation by anyone in LINE Each LINE service side Original ground truth (Some users) • Each service can generate its original segments. • All they need is the label data mapped to users. • No feature data is required.

Slide 74

Slide 74 text

Future works [2] Auto-persona: Enableing original segment estimation by anyone in LINE Each LINE service side Original ground truth (Some users) Original estimated segments (All users) Auto-persona API • Each service can generate its original segments. • All they need is the label data mapped to users. • No feature data is required.

Slide 75

Slide 75 text

Future works [2] Auto-persona: Enableing original segment estimation by anyone in LINE Feature data Preprocess Processed feature data Auto-persona API Dataset Model Create dataset Train model Original ground truth (Some users) Each LINE service side Original estimated segments (All users) • Each service can generate its original segments. • All they need is the label data mapped to users. • No feature data is required.

Slide 76

Slide 76 text

Future works [2] Auto-persona: Enableing original segment estimation by anyone in LINE Feature data Preprocess Processed feature data Auto-persona API Dataset Model Create dataset Train model Original ground truth (Some users) Each LINE service side Original estimated segments (All users) Able to deliver more useful content to each and every LINE user • Each service can generate its original segments. • All they need is the label data mapped to users. • No feature data is required.

Slide 77

Slide 77 text

Thank you