How LINE Ads Platform is Constantly Evolving

Jinsoo Oh
LINE Plus AD Performance Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/D1-3

Transcript

1. 2019 DevDay
   How LINE Ads Platform Is
   Constantly Evolving
   > Jinsoo Oh
   > LINE Plus AD Performance Software Engineer
   

   View Slide

2. Two and a Half Years Ago…
   

   View Slide

3. The Direction of Evolution
   Become an Advertiser-Friendly Platform
   Support Smart Features Using Machine Learning & Algorithmic Techniques
   Comply with Industry Standards
   

   View Slide

4. Takeaways
   Gather Professionals in the Ad Industry!
   Don’t Focus on the Beauty of your Logic! Then What?
   Make your System the Industry Standard!
   Take a Look at the Features that are Related to Machine Learning & Logic
   

   View Slide

5. https://www.linebiz.com/jp/service/line-ads-platform/
   

   View Slide

6. https://www.linebiz.com/jp/service/line-ads-platform/
   

   View Slide

7. Online
   Account
   2019.11
   JP Migration
   Completed
   2019.03
   JP Migration
   Started
   2018. 08
   TH OPEN
   2017.10
   Two and a Half Years Ago…
   

   View Slide

8. Two and a Half Years Ago…
   Conversion
   Events Added
   CTR/CVR Conversion
   Auto Bidding
   Click
   Auto Bidding
   

   View Slide

9. Understanding Some Terms
   Conversion Event
   App Install
   Purchase
   CVR
   Conversion Rate
   =
   # of conversions / # of
   clicks
   Bidding Amount
   1,000 yen per app install
   2,000 yen per purchase
   

   View Slide

10. Two and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    

    View Slide

11. Machine Learning &
    Algorithms
    System Architecture
    Two and a Half Years Ago…
    

    View Slide

12. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic
    

    View Slide

13. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic
    

    View Slide

14. System Architecture
    > Standard Protocols such as OpenRTB and VAST
    > Scalability
    > Components that Follow the Industry Standard
    

    View Slide

15. System Architecture
    A Brief View of the AD System
    

    View Slide

16. System Architecture
    A Brief View of the AD System
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    

    View Slide

17. System Architecture
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    A Brief View of the AD System
    

    View Slide

18. Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    System Architecture
    A Brief View of the AD System
    Ad Serving Ad Manager
    

    View Slide

19. System Architecture
    A Brief View of the AD System
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    Data Management Platform
    DMP
    Data Pipeline
    Pipeline
    

    View Slide

20. System Architecture
    Some Examples
    Supply Side Platform
    SSP
    Demand Side Platform
    DSP
    OpenRTB
    

    View Slide

21. Some Examples
    System Architecture
    Supply Side Platform
    FIVE SSP
    Demand Side Platform
    DSP
    OpenRTB
    

    View Slide

22. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic
    

    View Slide

23. Advertiser
    https://www.linebiz.com/jp/service/line-ads-platform/
    

    View Slide

24. Advertiser
    

    View Slide

25. Advertiser
    Supporting Strategies for Each Marketing Goal
    

    View Slide

26. Marketing Goal
    

    View Slide

27. Marketing Goal
    

    View Slide

28. Marketing Funnel
    Interest
    Consideration Conversion Retention
    Brand
    Awareness
    

    View Slide

29. Marketing Funnel
    Interest
    Consideration Conversion Retention
    Brand
    Awareness
    

    View Slide

30. Interest
    Consideration Conversion Retention
    Brand
    Awareness
    Marketing Strategy
    

    View Slide

31. Ad Performance
    Advertiser
    Interest
    Consideratio
    n
    Convers
    ion
    Retentio
    n
    Brand
    Awareness
    

    View Slide

32. Advertiser
    Target Event: App Install
    Willingness To Pay: 5,000 Yen per App Install
    Couldn’t Be Happier
    If You Can Lower the Price Under 5,000 Yen
    

    View Slide

33. Interest
    Consideration Conversion Retention
    Brand
    Awareness
    Marketing Strategy
    

    View Slide

34. Advertiser
    Target Event: App Install
    Willingness To Pay: 5,000 Yen per App Install
    Auto Bidding With Bid Cap Supporting App Install Conversion
    :
    Couldn’t Be Happier
    If You Can Lower the Price Under 5,000 Yen
    

    View Slide

35. Auto Bidding With Bid Cap Supporting App Install Conversion
    :
    CTR (Click Through Rate)
    CVR (Conversion Rate)
    BA (Bidding Amount)
    Building Block
    

    View Slide

36. Before This,
    We Prioritize Other Issues
    

    View Slide

37. 3 Important Things Before Logic
    Real-time Monitoring
    Real-time Serving
    Experimental Environment
    

    View Slide

38. Real-Time
    Monitoring
    

    View Slide

39. Real-Time Monitoring
    

    View Slide

40. Prometheus & Grafana
    Real-Time Monitoring
    

    View Slide

41. Druid / Imply
    OLAP
    

    View Slide

42. Experiment Environment
    

    View Slide

43. Experimental Environment
    Offline Test Online Test
    Data Pipeline
    

    View Slide

44. Online Test
    Offline Test Multi A/B
    Test
    Experimental Environment
    

    View Slide

45. Real-Time Serving
    

    View Slide

46. Latency
    Data
    Pipeline
    Model
    Performance
    Real-Time Serving
    

    View Slide

47. Latency
    Joiner Model
    Performance
    Real-Time Serving
    QA Tools
    Various Logs
    

    View Slide

48. 3 Important Things Before Logic
    Real-time Monitoring
    Real-time Serving
    Experimental Environment
    

    View Slide

49. Real-time Monitoring
    Real-time Serving
    Experimental Environment
    Interest
    Consideratio
    n
    Convers
    ion
    Retentio
    n
    Brand
    Awareness
    

    View Slide

50. View Slide

51. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic
    

    View Slide

52. Latency
    Real-Time Model
    Performance
    CTR Prediction
    

    View Slide

53. Latency
    Real-Time Model
    Performance
    CTR Prediction
    Logistic
    Regression
    Factorized
    Machine
    Field-Aware
    Factorized
    Machine
    FTRL-
    Proximal
    Gradient
    Boosting Tree
    Clustering
    

    View Slide

54. CVR Prediction
    Impression
    Click Conversion Conversion
    30 Days Later…
    

    View Slide

55. CVR Prediction
    Data
    Cleansing
    Rarity Delay Multi-Label
    

    View Slide

56. CVR Prediction
    Data
    Cleansing
    Rarity Delay Multi-Label
    Logistic
    Regression
    Delayed
    Feedback Model
    L-BFGS FTRL-
    Proximal
    Exponential
    Distribution
    Poisson
    Distribution
    

    View Slide

57. Given Total Budget
    Auto Bidding
    small
    Bidding Amount 3
    Time
    Winning Price
    Budget1 Budget2
    Bidding Amount 2
    Bidding Amount 1 Total Budget
    

    View Slide

58. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    

    View Slide

59. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    

    View Slide

60. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    

    View Slide

61. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    

    View Slide

62. Look-Alike
    

    View Slide

63. Look-Alike
    

    View Slide

64. Look-Alike
    

    View Slide

65. Speed
    Embedding
    Monitoring /
    Experimental
    Environment
    Look-Alike
    

    View Slide

66. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine
    

    View Slide

67. Simulator
    

    View Slide

68. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine R&F Simulator
    

    View Slide

69. Optimization Status
    0
    100
    200
    300
    400
    500
    600
    700
    800
    CPA Conversions Calibration
    

    View Slide

70. About 2 and a Half Years Ago…
    Conversion
    Events Added
    CTR/CVR Conversion
    Auto Bidding
    Click
    Auto Bidding
    New

    Look-Alike Engine R&F Simulator Optimization Status
    ?
    

    View Slide

71. Auto
    Delivery
    Auto
    Bidding
    Auto
    Experiment
    Auto Series
    

    View Slide

72. Takeaways
    Gather Professionals in the Ad Industry!
    Don’t Focus on the Beauty of your Logic! Then What?
    Make your System the Industry Standard!
    Take a Look at the Features that are Related to Machine Learning & Logic
    

    View Slide

73. Thank You
    

    View Slide