Advanced A/B Testing Systems with Dynamic User Segmentation

効果的なA/Bテストを行うには、適切なユーザーグループをターゲットにすることが重要です。このセッションでは、私たちのチームがこの課題をどのように解決したかを説明します。開発したA/Bテストシステムを紹介し、実際の使用例を共有します。

Transcript

1. LINE+ Contents Service Server 1 Ganghun Cho Advanced A/B Testing System: With Dynamic User Segmentation
2. 1. What is A/B Test 2. General A/B Test System 3. Advanced A/B Test System : Dynamic User Segmentation 4. Architecture 5. Use Case 6. Future Plans Index
3. What is A/B Test A method to compare two or more versions to see which version works better
4. What is A/B Test Example : Web Site Banner Hypothesis : The CTR (click-through rate) is likely to be higher when the banner color is blue. Current Version New Version Users seem to prefer blue banner.. A/B Test..! Marketer
5. Why A/B Test? To find out which design or feature gives users a better experience. Data-Driven Decision To get reliable results instead of just guessing or relying on intuition. User Experience
6. A/B Test : Test Group Current Version 50% Control Group Test Group New Version 50%
7. A/B Test : Test Group Control Group Test Group Control Group? Test Group?
8. General A/B Test System Test Group Assignment : Random Control Group Test Group user_id Yes No hash(id)%2 = 0?
9. Simplicity Easy to implement and manage. Also cost-effective in terms of time and resources General A/B Test System Reliability Since all users are assigned to test group randomly, It reduces selection bias by ensuring diverse groups Test Group Assignment : Random
10. Advanced A/B Test System Dynamic User Segmentation Hypothesis : For iOS users living in Osaka, the click-through rate (CTR) is likely to be higher when the banner color is blue. Control Group Test Group iOS users living in Osaka seem to prefer blue banner… Advanced A/B Test Tool…! Marketer
11. Advanced A/B Test System Dynamic User Segmentation Control Group Test Group device == iOS && area == Osaka No Yes
12. Advanced A/B Test System Dynamic User Segmentation General A/B Test System Advanced A/B Test System Group Assign Random Based on User Segment Use Case General Testing (Entire User) Personalized Testing
13. Architecture Targets specific user segments. A/B Test System Configure A/B test & Assign test groups to each user. Targeting System + Advanced A/B test system consists of the following two components.
14. Architecture ‒ Targeting System UserInfo MobileInfo AppActivity HDFS Get iOS users living in Osaka
15. Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka HDFS Object Storage Spark Get iOS users living in Osaka Read Write
16. Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 device == iOS && area == Osaka HDFS Object Storage Redis Spark Get iOS users living in Osaka Targeting Admin Operator Execute segment_id = 777 Read Write
17. Architecture ‒ Targeting System user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 Object Storage Redis Spark Various Conditional Operations RDD : union(), Intersect(), subtract() device == iOS && area == Osaka device == iOS || area == Osaka device == iOS + area == Osaka device == iOS - area == Osaka
18. Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator
19. Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator
20. Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator
21. Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator
22. Architecture ‒ A/B Test System Test Group Assigner A/B Test Admin Targeting Admin Upstream Contents Log Store Client Dashboard Central Dogma Redis key value 1-777 1 3-777 1
23. Architecture ‒ A/B Test System Test Group Assigner A/B Test Admin Targeting Admin Upstream Contents Log Store 1) Request(user_id=1) 2) Get Experiment Info 3) Get assigned test group -> T1 ( user_id=1, segment_id=777 ) 4) Get Data based on test group 6) Store log (experiment, test group) Client Dashboard 5) Return Data (T1) segment_id=777 Central Dogma Redis key value 1-777 1 3-777 1
24. Use Case Content Recommendation: Test which ML model performs better for particular user segment Model A? Model B? Content Recommendation Module Position Recommendation
25. Use Case Shopping Discount Since discount coupons are unlikely to affect the order frequency of heavy users, the test should be conducted only on light users. User App Onboarding If you want to know whether onboarding screen is effective, you should run the test only on the new users, not on all users. Others
26. Platform Expansion Expand the A/B Test platform to apply to various services within LY Corp. Future Plans Experiment Admin System Build an admin system that offers a unified interface for creating, editing, tracking history, and displaying experiment results via a dashboard.
27. 감사합니다 ありがとうございます。 Thank you