Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced A/B Testing Systems with Dynamic User ...

Advanced A/B Testing Systems with Dynamic User Segmentation

効果的なA/Bテストを行うには、適切なユーザーグループをターゲットにすることが重要です。このセッションでは、私たちのチームがこの課題をどのように解決したかを説明します。開発したA/Bテストシステムを紹介し、実際の使用例を共有します。

More Decks by LINEヤフーTech (LY Corporation Tech)

Other Decks in Technology

Transcript

  1. 1. What is A/B Test 2. General A/B Test System

    3. Advanced A/B Test System : Dynamic User Segmentation 4. Architecture 5. Use Case 6. Future Plans Index
  2. What is A/B Test A method to compare two or

    more versions to see which version works better
  3. What is A/B Test Example : Web Site Banner Hypothesis

    : The CTR (click-through rate) is likely to be higher when the banner color is blue. Current Version New Version Users seem to prefer blue banner.. A/B Test..! Marketer
  4. Why A/B Test? To find out which design or feature

    gives users a better experience. Data-Driven Decision To get reliable results instead of just guessing or relying on intuition. User Experience
  5. General A/B Test System Test Group Assignment : Random Control

    Group Test Group user_id Yes No hash(id)%2 = 0?
  6. Simplicity Easy to implement and manage. Also cost-effective in terms

    of time and resources General A/B Test System Reliability Since all users are assigned to test group randomly, It reduces selection bias by ensuring diverse groups Test Group Assignment : Random
  7. Advanced A/B Test System Dynamic User Segmentation Hypothesis : For

    iOS users living in Osaka, the click-through rate (CTR) is likely to be higher when the banner color is blue. Control Group Test Group iOS users living in Osaka seem to prefer blue banner… Advanced A/B Test Tool…! Marketer
  8. Advanced A/B Test System Dynamic User Segmentation Control Group Test

    Group device == iOS && area == Osaka No Yes
  9. Advanced A/B Test System Dynamic User Segmentation General A/B Test

    System Advanced A/B Test System Group Assign Random Based on User Segment Use Case General Testing (Entire User) Personalized Testing
  10. Architecture Targets specific user segments. A/B Test System Configure A/B

    test & Assign test groups to each user. Targeting System + Advanced A/B test system consists of the following two components.
  11. Architecture – Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1

    30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka HDFS Object Storage Spark Get iOS users living in Osaka Read Write
  12. Architecture – Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1

    30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 device == iOS && area == Osaka HDFS Object Storage Redis Spark Get iOS users living in Osaka Targeting Admin Operator Execute segment_id = 777 Read Write
  13. Architecture – Targeting System user_id visit_count 1 30 2 80

    3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 Object Storage Redis Spark Various Conditional Operations RDD : union(), Intersect(), subtract() device == iOS && area == Osaka device == iOS || area == Osaka device == iOS + area == Osaka device == iOS - area == Osaka
  14. Architecture – A/B Test System Test Group Assigner A/B Test

    Admin Targeting Admin Upstream Contents Log Store Client Dashboard Central Dogma Redis key value 1-777 1 3-777 1
  15. Architecture – A/B Test System Test Group Assigner A/B Test

    Admin Targeting Admin Upstream Contents Log Store 1) Request(user_id=1) 2) Get Experiment Info 3) Get assigned test group -> T1 ( user_id=1, segment_id=777 ) 4) Get Data based on test group 6) Store log (experiment, test group) Client Dashboard 5) Return Data (T1) segment_id=777 Central Dogma Redis key value 1-777 1 3-777 1
  16. Use Case Content Recommendation: Test which ML model performs better

    for particular user segment Model A? Model B? Content Recommendation Module Position Recommendation
  17. Use Case Shopping Discount Since discount coupons are unlikely to

    affect the order frequency of heavy users, the test should be conducted only on light users. User App Onboarding If you want to know whether onboarding screen is effective, you should run the test only on the new users, not on all users. Others
  18. Platform Expansion Expand the A/B Test platform to apply to

    various services within LY Corp. Future Plans Experiment Admin System Build an admin system that offers a unified interface for creating, editing, tracking history, and displaying experiment results via a dashboard.