Share
Embed
Advanced A/B Testing Systems with Dynamic User Segmentation
効果的なA/Bテストを行うには、適切なユーザーグループをターゲットにすることが重要です。このセッションでは、私たちのチームがこの課題をどのように解決したかを説明します。開発したA/Bテストシステムを紹介し、実際の使用例を共有します。
LINEヤフーTech (LY Corporation Tech) PRO
June 30, 2025
Transcript
27 slides
-
LINE+ Contents Service Server 1 Ganghun Cho Advanced A/B Testing
System: With Dynamic User Segmentation -
1. What is A/B Test 2. General A/B Test System
3. Advanced A/B Test System : Dynamic User Segmentation 4. Architecture 5. Use Case 6. Future Plans Index -
What is A/B Test A method to compare two or
more versions to see which version works better -
What is A/B Test Example : Web Site Banner Hypothesis
: The CTR (click-through rate) is likely to be higher when the banner color is blue. Current Version New Version Users seem to prefer blue banner.. A/B Test..! Marketer -
Why A/B Test? To find out which design or feature
gives users a better experience. Data-Driven Decision To get reliable results instead of just guessing or relying on intuition. User Experience -
A/B Test : Test Group Current Version 50% Control Group
Test Group New Version 50% -
A/B Test : Test Group Control Group Test Group Control
Group? Test Group? -
General A/B Test System Test Group Assignment : Random Control
Group Test Group user_id Yes No hash(id)%2 = 0? -
Simplicity Easy to implement and manage. Also cost-effective in terms
of time and resources General A/B Test System Reliability Since all users are assigned to test group randomly, It reduces selection bias by ensuring diverse groups Test Group Assignment : Random -
Advanced A/B Test System Dynamic User Segmentation Hypothesis : For
iOS users living in Osaka, the click-through rate (CTR) is likely to be higher when the banner color is blue. Control Group Test Group iOS users living in Osaka seem to prefer blue banner… Advanced A/B Test Tool…! Marketer -
Advanced A/B Test System Dynamic User Segmentation Control Group Test
Group device == iOS && area == Osaka No Yes -
Advanced A/B Test System Dynamic User Segmentation General A/B Test
System Advanced A/B Test System Group Assign Random Based on User Segment Use Case General Testing (Entire User) Personalized Testing -
Architecture Targets specific user segments. A/B Test System Configure A/B
test & Assign test groups to each user. Targeting System + Advanced A/B test system consists of the following two components. -
Architecture ‒ Targeting System UserInfo MobileInfo AppActivity HDFS Get iOS
users living in Osaka -
Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1
30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka HDFS Object Storage Spark Get iOS users living in Osaka Read Write -
Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1
30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 device == iOS && area == Osaka HDFS Object Storage Redis Spark Get iOS users living in Osaka Targeting Admin Operator Execute segment_id = 777 Read Write -
Architecture ‒ Targeting System user_id visit_count 1 30 2 80
3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 Object Storage Redis Spark Various Conditional Operations RDD : union(), Intersect(), subtract() device == iOS && area == Osaka device == iOS || area == Osaka device == iOS + area == Osaka device == iOS - area == Osaka -
Architecture ‒ A/B Test System Test Group Assigner A/B Test
Admin Targeting Admin Upstream Contents Log Store Client Dashboard Central Dogma Redis key value 1-777 1 3-777 1 -
Architecture ‒ A/B Test System Test Group Assigner A/B Test
Admin Targeting Admin Upstream Contents Log Store 1) Request(user_id=1) 2) Get Experiment Info 3) Get assigned test group -> T1 ( user_id=1, segment_id=777 ) 4) Get Data based on test group 6) Store log (experiment, test group) Client Dashboard 5) Return Data (T1) segment_id=777 Central Dogma Redis key value 1-777 1 3-777 1 -
Use Case Content Recommendation: Test which ML model performs better
for particular user segment Model A? Model B? Content Recommendation Module Position Recommendation -
Use Case Shopping Discount Since discount coupons are unlikely to
affect the order frequency of heavy users, the test should be conducted only on light users. User App Onboarding If you want to know whether onboarding screen is effective, you should run the test only on the new users, not on all users. Others -
Platform Expansion Expand the A/B Test platform to apply to
various services within LY Corp. Future Plans Experiment Admin System Build an admin system that offers a unified interface for creating, editing, tracking history, and displaying experiment results via a dashboard.