How LINE Ads Platform is Constantly Evolving

Slide 1

Slide 1 text

2019 DevDay How LINE Ads Platform Is Constantly Evolving > Jinsoo Oh > LINE Plus AD Performance Software Engineer

Slide 2

Slide 2 text

Two and a Half Years Ago…

Slide 3

Slide 3 text

The Direction of Evolution Become an Advertiser-Friendly Platform Support Smart Features Using Machine Learning & Algorithmic Techniques Comply with Industry Standards

Slide 4

Slide 4 text

Takeaways Gather Professionals in the Ad Industry! Don’t Focus on the Beauty of your Logic! Then What? Make your System the Industry Standard! Take a Look at the Features that are Related to Machine Learning & Logic

Slide 5

Slide 5 text

https://www.linebiz.com/jp/service/line-ads-platform/

Slide 6

Slide 6 text

https://www.linebiz.com/jp/service/line-ads-platform/

Slide 7

Slide 7 text

Online Account 2019.11 JP Migration Completed 2019.03 JP Migration Started 2018. 08 TH OPEN 2017.10 Two and a Half Years Ago…

Slide 8

Slide 8 text

Two and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 9

Slide 9 text

Understanding Some Terms Conversion Event App Install Purchase CVR Conversion Rate = # of conversions / # of clicks Bidding Amount 1,000 yen per app install 2,000 yen per purchase

Slide 10

Slide 10 text

Two and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 11

Slide 11 text

Machine Learning & Algorithms System Architecture Two and a Half Years Ago…

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

System Architecture > Standard Protocols such as OpenRTB and VAST > Scalability > Components that Follow the Industry Standard

Slide 15

Slide 15 text

System Architecture A Brief View of the AD System

Slide 16

Slide 16 text

System Architecture A Brief View of the AD System Supply Side Platform SSP Demand Side Platform DSP

Slide 17

Slide 17 text

System Architecture Supply Side Platform SSP Demand Side Platform DSP A Brief View of the AD System

Slide 18

Slide 18 text

Supply Side Platform SSP Demand Side Platform DSP System Architecture A Brief View of the AD System Ad Serving Ad Manager

Slide 19

Slide 19 text

System Architecture A Brief View of the AD System Supply Side Platform SSP Demand Side Platform DSP Data Management Platform DMP Data Pipeline Pipeline

Slide 20

Slide 20 text

System Architecture Some Examples Supply Side Platform SSP Demand Side Platform DSP OpenRTB

Slide 21

Slide 21 text

Some Examples System Architecture Supply Side Platform FIVE SSP Demand Side Platform DSP OpenRTB

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Advertiser https://www.linebiz.com/jp/service/line-ads-platform/

Slide 24

Slide 24 text

Advertiser

Slide 25

Slide 25 text

Advertiser Supporting Strategies for Each Marketing Goal

Slide 26

Slide 26 text

Marketing Goal

Slide 27

Slide 27 text

Marketing Goal

Slide 28

Slide 28 text

Marketing Funnel Interest Consideration Conversion Retention Brand Awareness

Slide 29

Slide 29 text

Marketing Funnel Interest Consideration Conversion Retention Brand Awareness

Slide 30

Slide 30 text

Interest Consideration Conversion Retention Brand Awareness Marketing Strategy

Slide 31

Slide 31 text

Ad Performance Advertiser Interest Consideratio n Convers ion Retentio n Brand Awareness

Slide 32

Slide 32 text

Advertiser Target Event: App Install Willingness To Pay: 5,000 Yen per App Install Couldn’t Be Happier If You Can Lower the Price Under 5,000 Yen

Slide 33

Slide 33 text

Interest Consideration Conversion Retention Brand Awareness Marketing Strategy

Slide 34

Slide 34 text

Advertiser Target Event: App Install Willingness To Pay: 5,000 Yen per App Install Auto Bidding With Bid Cap Supporting App Install Conversion : Couldn’t Be Happier If You Can Lower the Price Under 5,000 Yen

Slide 35

Slide 35 text

Auto Bidding With Bid Cap Supporting App Install Conversion : CTR (Click Through Rate) CVR (Conversion Rate) BA (Bidding Amount) Building Block

Slide 36

Slide 36 text

Before This, We Prioritize Other Issues

Slide 37

Slide 37 text

3 Important Things Before Logic Real-time Monitoring Real-time Serving Experimental Environment

Slide 38

Slide 38 text

Real-Time Monitoring

Slide 39

Slide 39 text

Real-Time Monitoring

Slide 40

Slide 40 text

Prometheus & Grafana Real-Time Monitoring

Slide 41

Slide 41 text

Druid / Imply OLAP

Slide 42

Slide 42 text

Experiment Environment

Slide 43

Slide 43 text

Experimental Environment Offline Test Online Test Data Pipeline

Slide 44

Slide 44 text

Online Test Offline Test Multi A/B Test Experimental Environment

Slide 45

Slide 45 text

Real-Time Serving

Slide 46

Slide 46 text

Latency Data Pipeline Model Performance Real-Time Serving

Slide 47

Slide 47 text

Latency Joiner Model Performance Real-Time Serving QA Tools Various Logs

Slide 48

Slide 48 text

3 Important Things Before Logic Real-time Monitoring Real-time Serving Experimental Environment

Slide 49

Slide 49 text

Real-time Monitoring Real-time Serving Experimental Environment Interest Consideratio n Convers ion Retentio n Brand Awareness

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Latency Real-Time Model Performance CTR Prediction

Slide 53

Slide 53 text

Latency Real-Time Model Performance CTR Prediction Logistic Regression Factorized Machine Field-Aware Factorized Machine FTRL- Proximal Gradient Boosting Tree Clustering

Slide 54

Slide 54 text

CVR Prediction Impression Click Conversion Conversion 30 Days Later…

Slide 55

Slide 55 text

CVR Prediction Data Cleansing Rarity Delay Multi-Label

Slide 56

Slide 56 text

CVR Prediction Data Cleansing Rarity Delay Multi-Label Logistic Regression Delayed Feedback Model L-BFGS FTRL- Proximal Exponential Distribution Poisson Distribution

Slide 57

Slide 57 text

Given Total Budget Auto Bidding small Bidding Amount 3 Time Winning Price Budget1 Budget2 Bidding Amount 2 Bidding Amount 1 Total Budget

Slide 58

Slide 58 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 59

Slide 59 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 60

Slide 60 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 61

Slide 61 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding

Slide 62

Slide 62 text

Look-Alike

Slide 63

Slide 63 text

Look-Alike

Slide 64

Slide 64 text

Look-Alike

Slide 65

Slide 65 text

Speed Embedding Monitoring / Experimental Environment Look-Alike

Slide 66

Slide 66 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding New  Look-Alike Engine

Slide 67

Slide 67 text

Simulator

Slide 68

Slide 68 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding New  Look-Alike Engine R&F Simulator

Slide 69

Slide 69 text

Optimization Status 0 100 200 300 400 500 600 700 800 CPA Conversions Calibration

Slide 70

Slide 70 text

About 2 and a Half Years Ago… Conversion Events Added CTR/CVR Conversion Auto Bidding Click Auto Bidding New  Look-Alike Engine R&F Simulator Optimization Status ?