Upgrade to Pro — share decks privately, control downloads, hide ads and more …

竟然有人說AI不能拿來做行銷

 竟然有人說AI不能拿來做行銷

Speaker: Johnson
Event: 台大AI社

LINE Developers Taiwan

November 01, 2022
Tweet

More Decks by LINE Developers Taiwan

Other Decks in Technology

Transcript


  1. __ɼᰈવ༗ਓ㘸AIෆೳ፤ိ၏ߦ᭖
    Data Scientist /Johnson Wu
    @LINE Taiwan

    View full-size slide

  2. Hello!
    - Johnson Wu


    - CSIE (B01, R05)


    - Intern @ Microsoft Research Asia


    - Poetry book writer (indirect)


    - Data Scientist @ LINE Taiwan
    2019 @LINE Fukuoka

    View full-size slide

  3. Imagination about Digital Marketing in LINE:
    *OUFSOFU
    DFMFCSJUZ
    4PDJBM
    NFEJB
    "EWFSUJTFNFOU

    View full-size slide

  4. Our actual thought of Marketing:
    Content & Experience
    Social & Relationship
    Commerce & Sales
    Management
    Data
    Advertising & Promotion
    • Talent Mgt.
    • Finance Mgt.
    • Quality Mgt.
    • CRM
    • Channels & Partners
    • Sales Automation & Intelligence
    • CDP/DMP
    • Business Intelligence
    • Data Governance
    • Customer Exp.
    • KOL
    • Community & Review
    • Content
    • Experience Mgt.
    • SEO
    • Marketing
    • Advertising
    • PR
    Align with user journey!

    View full-size slide

  5. Look into our problems with…
    (A)AARRR Funnel !
    How to let people visit my content?
    How to let people have action/buy?
    How to make people come back and buy?
    How to make people start to buy?
    How to let people refer new users to your content?
    Acquisition
    Activation
    Retention
    Revenue
    Referral
    1.25 2.5 3.75 5

    View full-size slide

  6. Commercial problems with the AARRR
    Real Cases in LINE Taiwan
    How to recommend OA push


    to users with limited number of pushes?
    How to attract most possibly


    new users to the service?
    How to value customers and


    Recommend based on values?
    How to deliver ad promotion


    with the best cost-profit balance?
    How do we understand users


    for AD targeting?
    Acquisition
    Activation
    Retention and activation
    AARRR
    Revenue

    View full-size slide

  7. Using knowledge graph solution for new user acquisition
    User Acquisition

    View full-size slide

  8. What are active users in the service?
    Where should I find the new users?
    How to connect new users with active users?
    Acquisition

    View full-size slide

  9. Users in LINE
    Correlate users with behaviors of other services
    -*/&
    50%":
    -*/&

    4)011*/(
    -*/&
    .64*$
    ?
    Acquisition
    0"

    View full-size slide

  10. Attracting New users
    Via SmartChannel with knowledge graph technique.
    smart channel


    And graph?
    Acquisition

    View full-size slide

  11. Graph embedding helps
    Service knowledge graph provides abundant information
    - GraphSAGE (inductive), GCN
    - Graph Attention network with deepwalk
    - Semantic matching based: ComplEX
    - Algorithm choice depends on data traits
    https://arxiv.org/pdf/1710.10903.pdf


    https://arxiv.org/pdf/1706.02216.pdf
    1
    2
    1
    2
    Acquisition

    View full-size slide

  12. How does graph work?
    Model predict the probability of the “link”.
    - Example: Graph Attention network with deepwalk
    1
    2
    Acquisition
    - Users are represented as edge connections between entity with meaningful
    actions
    - Users1: article1 read ->

    shop item2 -> join campaign 3….
    - Each node (entity) is aggregated

    by the weighted sum of neighbors.

    View full-size slide

  13. New acquired users
    Case study on LINE POINTS wall : how to acquire new users?
    -*/&10*/54XBMMVTFST
    OPO-*/&10*/54XBMMVTFST
    0
    ffi
    DJBM
    "DDPVOU
    GPMMPXJOH
    TUBUVT
    ,OPXMFEHF
    HSBQINPEFMJOH
    10*/540"

    $MVTUFSJOH
    ,OPXMFEHF
    HSBQIJOGFSFODF

    $MVTUFSJOH 0"

    OPO-*/&10*/54
    XBMMVTFST
    NPTUMJLFMZUP
    KPJO-*/&SFXBSE
    Compose triples
    Compose triples Top k similar users

    View full-size slide

  14. - Flexible and complex decision making


    - Applied to LINE SHOPPING, SPOT, POINTS, MUSIC
    - Embeddings show clustering effect w.r.t campaign
    Performance
    Case study on LINE POINTS : effectiveness and fully data orientation
    - ≥ 25% of new user growth comes from AI’s decision.
    - Easily fit for different scenario :

    How about recommend OA Push message to non-new users?


    - ≥200% CTR lift
    Acquisition
    Activation

    View full-size slide

  15. Apply uplift model to target the persuadable
    Retention:


    How to Maximize Campaign Profit?

    View full-size slide

  16. Who should receive OAP?
    How many OAP I should deliver?
    How to evaluate performance?
    Retention

    View full-size slide

  17. How to deliver ad promotion with the best cost-profit balance?
    $PVOUPGDBNQBJHO





    .POUIPG:FBS

    3FDFJWF
    0"NFTTBHF
    1VSDIBTF
    JODBNQBJHOQFSJPE
    &BSO
    -*/&$0*/10*/54
    Boost the Profits with Incentives
    Retention

    View full-size slide

  18. User Segments by Uplift
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Retention

    View full-size slide

  19. User Segments by Uplift
    EFMJWFSDPTU
    JODFOUJWFDPTU
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Retention

    View full-size slide

  20. User Segments by Uplift
    EFMJWFSZDPTU
    SFWFOVF
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Retention

    View full-size slide

  21. User Segments by Uplift
    EFMJWFSZDPTU
    High
    Low High
    Low
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Retention

    View full-size slide

  22. User Segments by Uplift
    EFMJWFSZDPTU
    *ODFOUJWFDPTU
    SFWFOVF
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Retention

    View full-size slide

  23. User Segments by Uplift
    QPTJUJWFVQMJGU
    OFHBUJWFVQMJGU
    OPVQMJGU
    OPVQMJGU
    Retention
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    Uplift M
    odel

    View full-size slide

  24. User Segments by Uplift Retention
    High
    Low
    Low High
    Buy if treated
    Buy if not treated
    Sure Thing
    Sleeping Dog
    Lost Cause
    The Persuadable
    3FTQPOTF.PEFM
    Uplift M
    odel

    View full-size slide

  25. A/B Testing to Collect Lookalike User Groups
    Treatment Group
    Control Group
    Retention

    View full-size slide

  26. #Control Group #Treatment Group
    Leverage Lookalike Groups to Estimate Uplift

    View full-size slide

  27. User 360 Degree Features
    Campaign Period
    Observation Period, 30 days
    Purchase


    or not
    Purchase


    or not
    Browsing behavior
    Demographics
    Purchase history
    OA engagement
    Ad event
    Sticker collections
    Treatment


    Group
    Control


    Group
    Retention

    View full-size slide

  28. User 360 Degree Features
    Campaign Period
    Observation Period, 30 days
    Purchase


    or not
    Purchase


    or not
    Browsing behavior
    Demographics
    Purchase history
    OA engagement
    Ad event
    Sticker collections
    Treatment


    Group
    Control


    Group
    If the campaign effect


    is significant

    If the sure thing exist

    Retention

    View full-size slide

  29. Meta-learners, S-Learner
    Campaign Period
    Observation Period, 30 days
    Browsing behavior
    Demographics
    Purchase history
    OA engagement
    Ad event
    Sticker collections
    Purchase


    probabilities
    Treatment


    Group
    Control


    Group
    $MBTTJ
    fi
    FS
    treatment = 1
    treatment = 0
    Purchase


    or not

    *OUVJUJWFBOEFBTZUPUSBJO
    -PXWBSJBODFBOE
    fl
    FYJCMF
    Retention

    View full-size slide

  30. Estimate Individual Treatment Effects
    Campaign Period
    Observation Period, 30 days
    Browsing behavior
    Demographics
    Purchase history
    OA engagement
    Ad event
    Sticker collections
    $MBTTJ
    fi
    FS
    Purchase probability
    Purchase probability
    treatment = 1
    treatment = 0
    candidate
    Retention

    View full-size slide

  31. Estimate Individual Treatment Effects
    Campaign Period
    Observation Period, 30 days
    Browsing behavior
    Demographics
    Purchase history
    OA engagement
    Ad event
    Sticker collections
    $MBTTJ
    fi
    FS
    treatment = 1
    treatment = 0
    Uplift
    Purchase probability
    candidate
    Retention

    View full-size slide

  32. Uplifts by Deciles
    6QMJGU







    The Persuadable Sure Thing/Lost Cause Sleeping Dog
    Retention

    View full-size slide

  33. Reduce Delivery Cost and Mute Rate in the Future
    6TFS-JTU
    %JTDPVOU

    6TFS-JTU
    $PJO1PJOU
    #BDL

    6TFS-JTU
    -VDLZESBX

    Day 3
    Day 2
    Day 1
    Campaign Period
    4UJDLFS
    %BUB
    %FW
    Retention

    View full-size slide

  34. Actionable and Explainable User Segmentation from RFM to CLV
    CRM:


    How to identify valuable customers?

    View full-size slide

  35. Know Your Customers
    F R
    M
    Historical Unknown
    Time
    Now
    From Buying Profile
    Retention
    Revenue

    View full-size slide

  36. Measure Your Customer Lifetime Value
    From RFM to CLV
    Past Now Future
    Measure
    Former Value
    Predict


    Future Value
    RFM CLV
    Retention
    Revenue

    View full-size slide

  37. Measure Your Customer Lifetime Value
    From RFM to CLV
    Past Now Future
    RFM CLV
    Measure
    Former Value
    Predict
    Retention
    Revenue

    View full-size slide

  38. Measure Former Value - RFM Modeling
    RFM


    Scores
    Historical data
    R
    F
    M
    Segments
    Retention
    Revenue

    View full-size slide

  39. Measure Former Value - RFM Modeling
    R
    F
    M
    RFM


    Segments
    Historical data
    Clustering
    Retention
    Revenue

    View full-size slide

  40. RFM Modeling - Clustering
    R
    F
    Retention
    Revenue

    View full-size slide

  41. RFM Modeling - Clustering
    R
    F
    Retention
    Revenue

    View full-size slide

  42. Measure Former Value - RFM Modeling
    R
    F
    M
    Historical data Clustering
    Retention
    Revenue

    View full-size slide

  43. Measure Your Customer Lifetime Value
    From RFM to CLV
    Past Now Future
    RFM CLV
    Measure
    Former Value
    Predict


    Future Value
    Retention
    Revenue

    View full-size slide

  44. Predict Future Value - CLV Modeling
    Demographic
    Browsing history
    Buying pro
    fi
    le
    OA interaction
    Feature
    Engineering
    Modeling Tags
    CLV High ↑
    CLV Median
    CLV Low ↓
    { > 50% High valued customers
    <= 50% Medium valued customers
    0, Churned customers
    450 days 180 days
    Now
    Cascade model
    Retention
    Revenue

    View full-size slide

  45. Predict Future Value - CLV Modeling
    Feature
    Engineering
    Modeling Tags
    CLV High ↑
    CLV Median
    CLV Low ↓
    { > 50% High valued customers
    <= 50% Medium valued customers
    0, Churned customers
    450 days 180 days
    Now
    User Embedding
    Demographic
    Browsing history
    Buying pro
    Retention
    Revenue

    View full-size slide

  46. Predict Future Value - CLV Modeling
    User Embedding
    Time of View
    Content Views from customers
    AD 1
    AD n
    Product 1
    Article 1
    Article n
    Product n
    … … …
    Context window
    Retention
    Revenue

    View full-size slide

  47. Explicit and explainable user segment
    R↑F↑M↑
    R↑F↑M↓
    R↑F↓M↑
    R↑F↓M↓
    R↓F↑M↑
    R↓F↑M↓
    R↓F↓M↑
    R↓F↓M↓
    ࠷ۙ༗ိత౔߽ৗ٬
    ࠷ۙ༗ိతখࢿৗ٬
    ࠷ۙ༗ိత౔߽ك٬
    ࠷ۙ༗ိతখࢿك٬
    ফࣦత౔߽ৗ٬
    ফࣦతখࢿৗ٬
    ফࣦత౔߽ك٬
    ফࣦతখࢿك٬
    CLV Low
    CLV Median
    CLV High ະိߴফඅ
    ະိதফඅ
    ະိෆফඅ
    RFM
    CLV
    Deliverables: Tags Retention
    Revenue

    View full-size slide

  48. ೗Կར༻໛ܕඪតʁ
    R↑F↑M↓
    Upselling !
    RFM
    RFM
    最近有來的⼩資常客
    R↑F↑M↑
    RFM
    最近有來的⼟豪常客
    Retention
    Revenue

    View full-size slide

  49. R↑F↑M↑
    CLV Low
    Retain !
    ೗Կར༻໛ܕඪតʁ
    RFM + CLV
    RFM
    CLV
    最近有來的⼟豪常客 未來不消費
    Retention
    Revenue

    View full-size slide

  50. How to adopt CLV service?
    Overview of CLV Service
    Retention
    Revenue

    View full-size slide

  51. Wait… is there…

    View full-size slide

  52. Any model that could handle everything?
    A good start of user understanding !
    #MBDLCPY
    Who among them are more likely to come?
    Which content is more likely be click by whom?
    Who among them are more likely persuadable?
    How to estimate user values?
    How to refer friends of users to us?

    View full-size slide

  53. Foundation models
    Pertaining on user behaviors, which come from our family services log!
    Click prediction


    User segment prediction


    Sales/revenue prediction

    View full-size slide

  54. Combined with MLOps:

    Scalable Automated MarTech Solution
    Data Source
    %BUB
    8BSFIPVTF
    Data Integration
    'FBUVSF4UPSF
    Application
    "DRVJSFOFXVTFS
    "%5BSHFUJOH
    #PPTUSFWFOVF
    *NQSPWF3FUFOUJPO

    ,(4PMVUJPO1JQFMJOF
    $-74PMVUJPO1JQFMJOF
    6QMJGU

    4PMVUJPO1JQFMJOF
    AI Solution
    6TFSTFHNFOUBUJPO

    4PMVUJPO1JQFMJOF

    LINE
    Family
    Services
    Demographic
    Log

    View full-size slide

  55. To sum up
    Swifter Scalable and Flexible Solution


    Faster Speed of Delivery


    Data-oriented Decision Making


    AI solves (almost) every Martech problem


    Developer


    Marketer


    YOU

    View full-size slide

  56. Thank you

    資料科學家大解密
    - LINE資料工程部門介紹
    - 資料科學生活
    - 職場眉眉角角
    Closing the Distance
    Share easily with friends in LINE Timeline

    View full-size slide

  57. Thank you
    Closing the Distance
    Official
    Account
    Ads
    圖⽚來源︓https://hub.line.me/

    View full-size slide

  58. Thank you
    Closing the Distance
    Official
    Account
    Ads
    圖⽚來源︓https://hub.line.me/
    如何推薦用戶會感興趣的商家?
    如何確保新聞品質? 如何判斷假新聞?
    如何判斷潛在的訂閱制流失戶?

    View full-size slide

  59. Thank you
    AI-enable
    Applications
    Business
    Intelligence Data Dev
    LINE
    Family
    Services
    LINE
    TODAY
    LINE
    SHOPPING
    LINE
    SPOT
    LINE
    MUSIC
    LINE
    Sticker
    LINE
    VOOM
    LINE
    Reward
    Official
    Account
    Fact
    Checker
    LINE
    HELP
    TW
    LINE
    Travel
    Ads
    獨立的資料工程部門,提供資料科學解決方案
    LINE
    TODAY

    View full-size slide

  60. Thank you
    Data Dev
    LINE
    Family
    Services
    LINE
    SHOPPING
    LINE
    SPOT
    LINE
    MUSIC
    LINE
    Sticker
    LINE
    VOOM
    LINE
    Reward
    Fact
    Checker
    LINE
    HELP
    TW
    LINE
    Travel
    NLP
    Knowledg
    e Graph
    Uplift
    Modeling
    NER
    Classifier
    Duplication
    Detector
    Auto
    completion
    Keyword
    Extraction
    Related
    Search
    Text
    Generation
    User
    Tagging
    Data
    Analytics
    Recom-
    mendation
    CLV
    從報表、分析洞見到預測模型
    LINE
    TODAY

    View full-size slide

  61. Thank you
    成員組成
    • Build and optimize da
    ta pipeline architectur
    e
    • Assemble large, com
    plex data sets that m
    eet requirements
    Data Engineer Data Analyst
    Big data infra, SQL, ET
    L, message queuing
    • Interpret data, analyz
    e results using statisti
    cal techniques
    • Identify, analyze, and
    interpret trends or pat
    terns in complex data
    sets
    Statistics, Data Visualiz
    ation, Business Knowle
    dge
    SKILL RESPONSIBILITY
    • Select appropriate da
    tasets and data repre
    sentation methods
    • Research and imple
    ment appropriate ML
    algorithms
    Data Scientist
    Machine learning, deep
    learning, CV, NLP, Spe
    ech
    ML Svc Engineer
    • Build and scale mach
    ine learning infrastruc
    ture
    • Monitor model perfor
    mance
    System infrastructure d
    esign, DevOps

    View full-size slide

  62. Thank you
    Subtitle
    成員組成
    • Build and optimize da
    ta pipeline architectur
    e
    • Assemble large, com
    plex data sets that m
    eet requirements
    Data Engineer Data Analyst
    Big data infra, SQL, ET
    L, message queuing
    • Interpret data, analyz
    e results using statisti
    cal techniques
    • Identify, analyze, and
    interpret trends or pat
    terns in complex data
    sets
    Statistics, Data Visualiz
    ation, Business Knowle
    dge
    SKILL RESPONSIBILITY
    Pipeline
    Biz
    • Select appropriate da
    tasets and data repre
    sentation methods
    • Research and imple
    ment appropriate ML
    algorithms
    Data Scientist
    Machine learning, deep
    learning, CV, NLP, Spe
    ech
    Model
    ML Svc Engineer
    • Build and scale mach
    ine learning infrastruc
    ture
    • Monitor model perfor
    mance
    System infrastructure d
    esign, DevOps
    Service

    View full-size slide