Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

ML Life Cycle for LINE Shopping Recommender

Slide 3

Slide 3 text

01 02 03 04 Transform as ML Problem Modeling Execute A/B test Problem Definition 05 Evaluation CONTENT 06 Decision

Slide 4

Slide 4 text

Rei Huang Machine Learning Engineer Education • Bachelor in Mathematics @ CCU • Master in Mathematics @ NTHU Work Experience • E.Sun Commercial Bank (2016 ~ 2022) • LINE Taiwan EC (2022 ~ Now)

Slide 5

Slide 5 text

Sketch of ML Life Cycle Business Issue Problem Definition ML problem ML model building A/B test Evaluation Decision

Slide 6

Slide 6 text

Identify the core issue Focus on the primary problem 01 Problem Definition

Slide 7

Slide 7 text

鎖定問題,對症下藥! Problem Definition 設計產品 折扣券 提升用戶 體驗

Slide 8

Slide 8 text

Analyze by the funnel chart Problem Definition Pageview Click Transfer page Orders Importance Sparsity

Slide 9

Slide 9 text

Describe problem by ML statement 02 Transform our Problem as a ML Problem

Slide 10

Slide 10 text

Transform our Problem as a ML Problem Precision@10 電話行銷 簡訊通知 email Precision@10k Recall@500k

Slide 11

Slide 11 text

Transform our Problem as a ML Problem Which one do you want to click? ( ) , ?future days Regession or Classifiction?

Slide 12

Slide 12 text

End-to-end model building 03 Modeling

Slide 13

Slide 13 text

Feature Engineering Model Data Modeling Model Training Model Inference Offline Evaluation Modeling Cycle

Slide 14

Slide 14 text

Design an effective experiment 04 Execute A/B testing

Slide 15

Slide 15 text

Execute A/B test Save to Database Request Recommendation experimental control 核心概念 1. 均勻分布 2. 執行同一個實驗,user 的組別是固定的 3. 同一個實驗,若增加實驗比率,實驗組不變 4. 高計算效率 experiment_id ratio Ref: A/B Test Bucketing using Hashing Sketch of Design Version Control

Slide 16

Slide 16 text

Execute A/B test Ref: A/B Test Bucketing using Hashing Core of hash function 每次進入網站時分組為 A or B 同一實驗調整 ratio 但組別重分配 同一 user 對到不同 model logic 均勻分布 全母體均勻 or acitve user 均勻 A control B control A control B control Session based User based 高計算效率

Slide 17

Slide 17 text

Confirm the performance 05 Evaluation

Slide 18

Slide 18 text

Evaluation Online experimental control Within a certain time Extract data CTR Precision MRR CVR ML metric business metric …

Slide 19

Slide 19 text

Evaluation Others Feature correlation Ref: https://byjus.com/maths/inversely-proportional/ Monitoring

Slide 20

Slide 20 text

What’s the next? 06 Decision

Slide 21

Slide 21 text

What’s wrong with us? Orders Clicks Model Performance Model Tuning Bad Good Does model solve click? 這時候才調整嗎? Monitoring

Slide 22

Slide 22 text

ML Life Cycle Business Issue Problem Definition ML problem ML model building A/B test Evaluation Decision

Slide 23

Slide 23 text

Ideal Corporation Mode Conclusion 項目標題文字 Engineering Product Owner

Slide 24

Slide 24 text

No content