A/B Testing in LINE TODAY – Recommendation Tab Case @ TECHPULSE 2023

- Speaker: Wilson Wang
- Event: http://techpulse.line.me/

好的推薦功能讓用戶在看 LINE TODAY 時能夠看到關注的內容。本次將以 LINE TODAY 的推薦頁面為案例，帶大家了解 LINE TODAY 是如何透過 A/B Testing ，分析用戶對於不同功能的接受度，進一步了解用戶需求與體驗。

Transcript

2. Recommendation Tab Case 2
3. Recommendation Tab Case 3 auto play video or not
4. Recommendation Tab Case 4 auto play video or not thumbnail to the left or right
5. 5 User Research De fi ne Specs & Test Plan Development Conduct A / B Test Analyze Results A B A/B Testing in LINE TODAY 1 2 3
6. User Research Identify our goal Expected results Research method 6
7. User Research Identify our goal 7 Research method Expected results • Better understand usage of our app • Explore effectiveness of UI layout and response • A/B testing of different layout • User survey to different test groups • Get user feedback • How will the test results help future planning
8. Define Specs & Test Plan 8 Derive details from the previous step and decide what data to collect and how to collect Define how many percent of people should be included and how to segment the group Decide when to start and how long should we run the test Define Configuration Plan Schedule Test Specification 8
9. Development Internal Setting Flow Configure Group Settings Create A/B Test Groups 9
10. Development User Request Flow 10 Get User Information Identify User Group Respond with Page A or B
11. Conduct A/B Test 11 Con fi gure A/B test groups settings (optional) A/A test Start A/B test
12. Analyze Results 12 Quantitative Analysis Qualitative Analysis
13. Analyze Results 13 Quantitative Analysis •Monitor metrics such as Clicks, PV, CTR through internal dashboards •Validate and review our hypothesis from previous steps Collect User Metrics Generate Report
14. Analyze Results 14 Qualitative Analysis •Understand the reason why through user’s point of view •Analyze survey results to draw some conclusions about this test
15. Thank you 15