Goal - We want users to read more articles - Improve and maximize FOR YOU clicks, impressions, and CTR LINE NEWS’s Goal - We want to increase LINE NEWS' pageviews & ad sales - Potential side effects: - The increase in impressions and clicks on FOR YOU do not exceed the overall decrease in impressions and clicks for non-FOR YOU components - FOR YOU clicks increase, but overall pageviews and/or ad sales decrease
Lift Lift = (treatment group’s values / control group’s values) - 1 1. FOR YOU Clicks, Impressions and CTR 2. Overall Pageviews per Session 3. Overall News Article Impressions 4. Overall Ad Sales FOR YOU’s KPI LINE NEWS KPI LINE NEWS KPI LINE NEWS KPI
the new recommendation engine. There may be an unintended bias in the recommended articles. Conclusion: There is most likely an unintended bias, especially towards men. The exposure frequency of entertainment category articles are clearly higher in the Treatment group.
testing again Examples of further AB tests conducted based on this result include: - Adding features to bring out more categories - Adding features to reflect long-term interests in recommendation results
the purpose - Aim for overall optimization, not partial optimization After the AB Test - Often does not go as designed - Can be caused by qualitative and ambiguous things - Quantify qualitative feedback - Verbalize with quantitative indicators - Prepare many dimensions for analysis