Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A/B Testing in LINE TODAY – Recommendation Tab Case @ TECHPULSE 2023

A/B Testing in LINE TODAY – Recommendation Tab Case @ TECHPULSE 2023

- Speaker: Wilson Wang
- Event: http://techpulse.line.me/

好的推薦功能讓用戶在看 LINE TODAY 時能夠看到關注的內容。本次將以 LINE TODAY 的推薦頁面為案例,帶大家了解 LINE TODAY 是如何透過 A/B Testing ,分析用戶對於不同功能的接受度,進一步了解用戶需求與體驗。

LINE Developers Taiwan

February 21, 2023
Tweet

More Decks by LINE Developers Taiwan

Other Decks in Technology

Transcript

  1. Recommendation Tab Case
    2

    View full-size slide

  2. Recommendation Tab Case
    3
    auto play video or not

    View full-size slide

  3. Recommendation Tab Case
    4
    auto play video or not
    thumbnail to the left or right

    View full-size slide

  4. 5
    User
    Research
    De
    fi
    ne Specs
    & Test Plan
    Development
    Conduct A / B Test
    Analyze
    Results
    A B
    A/B Testing in
    LINE TODAY
    1
    2
    3

    View full-size slide

  5. User Research
    Identify our
    goal
    Expected
    results
    Research
    method
    6

    View full-size slide

  6. User Research
    Identify
    our goal
    7
    Research
    method
    Expected
    results
    • Better understand usage
    of our app
    • Explore effectiveness of
    UI layout and response
    • A/B testing of different
    layout
    • User survey to different
    test groups
    • Get user feedback
    • How will the test results
    help future planning

    View full-size slide

  7. Define Specs & Test Plan
    8
    Derive details from the
    previous step and decide
    what data to collect and
    how to collect
    Define how many percent
    of people should be
    included and how to
    segment the group
    Decide when to start and
    how long should we run
    the test
    Define
    Configuration
    Plan
    Schedule
    Test
    Specification
    8

    View full-size slide

  8. Development
    Internal Setting Flow
    Configure
    Group Settings
    Create A/B Test
    Groups
    9

    View full-size slide

  9. Development
    User Request Flow
    10
    Get User
    Information
    Identify User
    Group
    Respond with
    Page A or B

    View full-size slide

  10. Conduct A/B Test
    11
    Con
    fi
    gure A/B test groups settings (optional) A/A test Start A/B test

    View full-size slide

  11. Analyze Results
    12
    Quantitative
    Analysis
    Qualitative
    Analysis

    View full-size slide

  12. Analyze Results
    13
    Quantitative
    Analysis
    •Monitor metrics such as Clicks, PV, CTR
    through internal dashboards
    •Validate and review our hypothesis from
    previous steps
    Collect User
    Metrics
    Generate
    Report

    View full-size slide

  13. Analyze Results
    14
    Qualitative
    Analysis
    •Understand the reason why through
    user’s point of view
    •Analyze survey results to draw some
    conclusions about this test

    View full-size slide