A/B Testing in LINE TODAY – Recommendation Tab Case @ TECHPULSE 2023

- Speaker: Wilson Wang
- Event: http://techpulse.line.me/

好的推薦功能讓用戶在看 LINE TODAY 時能夠看到關注的內容。本次將以 LINE TODAY 的推薦頁面為案例，帶大家了解 LINE TODAY 是如何透過 A/B Testing ，分析用戶對於不同功能的接受度，進一步了解用戶需求與體驗。

Transcript

1. View Slide

2. Recommendation Tab Case
   2
   

   View Slide

3. Recommendation Tab Case
   3
   auto play video or not
   

   View Slide

4. Recommendation Tab Case
   4
   auto play video or not
   thumbnail to the left or right
   

   View Slide

5. 5
   User
   Research
   De
   fi
   ne Specs
   & Test Plan
   Development
   Conduct A / B Test
   Analyze
   Results
   A B
   A/B Testing in
   LINE TODAY
   1
   2
   3
   

   View Slide

6. User Research
   Identify our
   goal
   Expected
   results
   Research
   method
   6
   

   View Slide

7. User Research
   Identify
   our goal
   7
   Research
   method
   Expected
   results
   • Better understand usage
   of our app
   • Explore effectiveness of
   UI layout and response
   • A/B testing of different
   layout
   • User survey to different
   test groups
   • Get user feedback
   • How will the test results
   help future planning
   

   View Slide

8. Define Specs & Test Plan
   8
   Derive details from the
   previous step and decide
   what data to collect and
   how to collect
   Define how many percent
   of people should be
   included and how to
   segment the group
   Decide when to start and
   how long should we run
   the test
   Define
   Configuration
   Plan
   Schedule
   Test
   Specification
   8
   

   View Slide

9. Development
   Internal Setting Flow
   Configure
   Group Settings
   Create A/B Test
   Groups
   9
   

   View Slide

10. Development
    User Request Flow
    10
    Get User
    Information
    Identify User
    Group
    Respond with
    Page A or B
    

    View Slide

11. Conduct A/B Test
    11
    Con
    fi
    gure A/B test groups settings (optional) A/A test Start A/B test
    

    View Slide

12. Analyze Results
    12
    Quantitative
    Analysis
    Qualitative
    Analysis
    

    View Slide

13. Analyze Results
    13
    Quantitative
    Analysis
    •Monitor metrics such as Clicks, PV, CTR
    through internal dashboards
    •Validate and review our hypothesis from
    previous steps
    Collect User
    Metrics
    Generate
    Report
    

    View Slide

14. Analyze Results
    14
    Qualitative
    Analysis
    •Understand the reason why through
    user’s point of view
    •Analyze survey results to draw some
    conclusions about this test
    

    View Slide

15. Thank you
    15
    

    View Slide