Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How LINE Ads Platform is Constantly Evolving

How LINE Ads Platform is Constantly Evolving

Jinsoo Oh
LINE Plus AD Performance Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/D1-3

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay
    How LINE Ads Platform Is
    Constantly Evolving
    > Jinsoo Oh
    > LINE Plus AD Performance Software Engineer

    View Slide

  2. Two and a Half Years Ago…

    View Slide

  3. The Direction of Evolution
    Become an Advertiser-Friendly Platform
    Support Smart Features Using Machine Learning & Algorithmic Techniques
    Comply with Industry Standards

    View Slide

  4. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  5. https://www.linebiz.com/jp/service/line-ads-platform/

    View Slide

  6. https://www.linebiz.com/jp/service/line-ads-platform/

    View Slide

  7. Online
    Account
    2019.11
    JP Migration
    Completed
    2019.03
    JP Migration
    Started
    2018. 08
    TH OPEN
    2017.10
    Two and a Half Years Ago…

    View Slide

  8. Two and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  9. Understanding Some Terms
    Conversion Event
    App Install
    Purchase
    CVR
    Conversion Rate
    =
    # of conversions / # of
    clicks
    Bidding Amount
    1,000 yen per app install
    2,000 yen per purchase

    View Slide

  10. Two and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  11. Machine Learning &
    Algorithms
    System Architecture
    Two and a Half Years Ago…

    View Slide

  12. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  13. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  14. System Architecture
    > Standard Protocols such as OpenRTB and VAST
    > Scalability
    > Components that Follow the Industry Standard

    View Slide

  15. System Architecture
    A Brief View of the AD System

    View Slide

  16. System Architecture
    A Brief View of the AD System
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP

    View Slide

  17. System Architecture
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    A Brief View of the AD System

    View Slide

  18. Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    System Architecture
    A Brief View of the AD System
    Ad Serving Ad Manager

    View Slide

  19. System Architecture
    A Brief View of the AD System
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    Data Management Platform
    DMP
    Data Pipeline
    Pipeline

    View Slide

  20. System Architecture
    Some Examples
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    OpenRTB

    View Slide

  21. Some Examples
    System Architecture
    Supply Side Platform
    FIVE SSP
    Demand Side Platform
    DSP
    OpenRTB

    View Slide

  22. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  23. Advertiser
    https://www.linebiz.com/jp/service/line-ads-platform/

    View Slide

  24. Advertiser

    View Slide

  25. Advertiser
    Supporting Strategies for Each Marketing Goal

    View Slide

  26. Marketing Goal

    View Slide

  27. Marketing Goal

    View Slide

  28. Marketing Funnel
    Interest
    Consideration Conversion Retention
    Brand
    Awareness

    View Slide

  29. Marketing Funnel
    Interest
    Consideration Conversion Retention
    Brand
    Awareness

    View Slide

  30. Interest
    Consideration Conversion Retention
    Brand
    Awareness
    Marketing Strategy

    View Slide

  31. Ad Performance
    Advertiser
    Interest
    Consideratio
    n
    Convers
    ion
    Retentio
    n
    Brand
    Awareness

    View Slide

  32. Advertiser
    Target Event: App Install
    Willingness To Pay: 5,000 Yen per App Install
    Couldn’t Be Happier
    If You Can Lower the Price Under 5,000 Yen

    View Slide

  33. Interest
    Consideration Conversion Retention
    Brand
    Awareness
    Marketing Strategy

    View Slide

  34. Advertiser
    Target Event: App Install
    Willingness To Pay: 5,000 Yen per App Install
    Auto Bidding With Bid Cap Supporting App Install Conversion
    :
    Couldn’t Be Happier
    If You Can Lower the Price Under 5,000 Yen

    View Slide

  35. Auto Bidding With Bid Cap Supporting App Install Conversion
    :
    CTR (Click Through Rate)
    CVR (Conversion Rate)
    BA (Bidding Amount)
    Building Block

    View Slide

  36. Before This,
    We Prioritize Other Issues

    View Slide

  37. 3 Important Things Before Logic
    Real-time Monitoring
    Real-time Serving
    Experimental Environment

    View Slide

  38. Real-Time
    Monitoring

    View Slide

  39. Real-Time Monitoring

    View Slide

  40. Prometheus & Grafana
    Real-Time Monitoring

    View Slide

  41. Druid / Imply
    OLAP

    View Slide

  42. Experiment Environment

    View Slide

  43. Experimental Environment
    Offline Test Online Test
    Data Pipeline

    View Slide

  44. Online Test
    Offline Test Multi A/B
    Test
    Experimental Environment

    View Slide

  45. Real-Time Serving

    View Slide

  46. Latency
    Data
    Pipeline
    Model
    Performance
    Real-Time Serving

    View Slide

  47. Latency
    Joiner Model
    Performance
    Real-Time Serving
    QA Tools
    Various Logs

    View Slide

  48. 3 Important Things Before Logic
    Real-time Monitoring
    Real-time Serving
    Experimental Environment

    View Slide

  49. Real-time Monitoring
    Real-time Serving
    Experimental Environment
    Interest
    Consideratio
    n
    Convers
    ion
    Retentio
    n
    Brand
    Awareness

    View Slide

  50. View Slide

  51. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  52. Latency
    Real-Time Model
    Performance
    CTR Prediction

    View Slide

  53. Latency
    Real-Time Model
    Performance
    CTR Prediction
    Logistic
    Regression
    Factorized
    Machine
    Field-Aware
    Factorized
    Machine
    FTRL-
    Proximal
    Gradient
    Boosting Tree
    Clustering

    View Slide

  54. CVR Prediction
    Impression
    Click Conversion Conversion
    30 Days Later…

    View Slide

  55. CVR Prediction
    Data
    Cleansing
    Rarity Delay Multi-Label

    View Slide

  56. CVR Prediction
    Data
    Cleansing
    Rarity Delay Multi-Label
    Logistic
    Regression
    Delayed
    Feedback Model
    L-BFGS FTRL-
    Proximal
    Exponential
    Distribution
    Poisson
    Distribution

    View Slide

  57. Given Total Budget
    Auto Bidding
    small
    Bidding Amount 3
    Time
    Winning Price
    Budget1 Budget2
    Bidding Amount 2
    Bidding Amount 1 Total Budget

    View Slide

  58. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  59. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  60. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  61. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding

    View Slide

  62. Look-Alike

    View Slide

  63. Look-Alike

    View Slide

  64. Look-Alike

    View Slide

  65. Speed
    Embedding
    Monitoring /
    Experimental
    Environment
    Look-Alike

    View Slide

  66. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine

    View Slide

  67. Simulator

    View Slide

  68. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine R&F Simulator

    View Slide

  69. Optimization Status
    0
    100
    200
    300
    400
    500
    600
    700
    800
    CPA Conversions Calibration

    View Slide

  70. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine R&F Simulator Optimization Status
    ?

    View Slide

  71. Auto
    Delivery
    Auto
    Bidding
    Auto
    Experiment
    Auto Series

    View Slide

  72. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic

    View Slide

  73. Thank You

    View Slide