LINE+ Contents Service Server 1 Ganghun Cho Advanced A/B Testing System: With Dynamic User Segmentation 

1. What is A/B Test 2. General A/B Test System 3. Advanced A/B Test System : Dynamic User Segmentation 4. Architecture 5. Use Case 6. Future Plans Index 

What is A/B Test A method to compare two or more versions to see which version works better 

What is A/B Test Example : Web Site Banner Hypothesis : The CTR (click-through rate) is likely to be higher when the banner color is blue. Current Version New Version Users seem to prefer blue banner.. A/B Test..! Marketer 

Why A/B Test? To find out which design or feature gives users a better experience. Data-Driven Decision To get reliable results instead of just guessing or relying on intuition. User Experience 

A/B Test : Test Group Current Version 50% Control Group Test Group New Version 50% 

A/B Test : Test Group Control Group Test Group Control Group? Test Group? 

General A/B Test System Test Group Assignment : Random Control Group Test Group user_id Yes No hash(id)%2 = 0? 

Simplicity Easy to implement and manage. Also cost-effective in terms of time and resources General A/B Test System Reliability Since all users are assigned to test group randomly, It reduces selection bias by ensuring diverse groups Test Group Assignment : Random 

Advanced A/B Test System Dynamic User Segmentation Hypothesis : For iOS users living in Osaka, the click-through rate (CTR) is likely to be higher when the banner color is blue. Control Group Test Group iOS users living in Osaka seem to prefer blue banner… Advanced A/B Test Tool…! Marketer 

Advanced A/B Test System Dynamic User Segmentation Control Group Test Group device == iOS && area == Osaka No Yes 

Advanced A/B Test System Dynamic User Segmentation General A/B Test System Advanced A/B Test System Group Assign Random Based on User Segment Use Case General Testing (Entire User) Personalized Testing 

Architecture Targets specific user segments. A/B Test System Configure A/B test & Assign test groups to each user. Targeting System + Advanced A/B test system consists of the following two components. 

Architecture ‒ Targeting System UserInfo MobileInfo AppActivity HDFS Get iOS users living in Osaka 

Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka HDFS Object Storage Spark Get iOS users living in Osaka Read Write 

Architecture ‒ Targeting System UserInfo MobileInfo AppActivity user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 device == iOS && area == Osaka HDFS Object Storage Redis Spark Get iOS users living in Osaka Targeting Admin Operator Execute segment_id = 777 Read Write 

Architecture ‒ Targeting System user_id visit_count 1 30 2 80 3 170 user_id device_os 1 ios 2 aos 3 ios user_id area 1 osaka 2 tokyo 3 osaka key value {user_id}-{segment_id} 1 1-777 1 3-777 1 Object Storage Redis Spark Various Conditional Operations RDD : union(), Intersect(), subtract() device == iOS && area == Osaka device == iOS || area == Osaka device == iOS + area == Osaka device == iOS - area == Osaka 

Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator 

Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator 

Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator 

Central Dogma (Configuration Repository) Admin Architecture ‒ A/B Test Admin Operator 

Architecture ‒ A/B Test System Test Group Assigner A/B Test Admin Targeting Admin Upstream Contents Log Store Client Dashboard Central Dogma Redis key value 1-777 1 3-777 1 

Architecture ‒ A/B Test System Test Group Assigner A/B Test Admin Targeting Admin Upstream Contents Log Store 1) Request(user_id=1) 2) Get Experiment Info 3) Get assigned test group -> T1 ( user_id=1, segment_id=777 ) 4) Get Data based on test group 6) Store log (experiment, test group) Client Dashboard 5) Return Data (T1) segment_id=777 Central Dogma Redis key value 1-777 1 3-777 1 

Use Case Content Recommendation: Test which ML model performs better for particular user segment Model A? Model B? Content Recommendation Module Position Recommendation 

Use Case Shopping Discount Since discount coupons are unlikely to affect the order frequency of heavy users, the test should be conducted only on light users. User App Onboarding If you want to know whether onboarding screen is effective, you should run the test only on the new users, not on all users. Others 

Platform Expansion Expand the A/B Test platform to apply to various services within LY Corp. Future Plans Experiment Admin System Build an admin system that offers a unified interface for creating, editing, tracking history, and displaying experiment results via a dashboard. 

감사합니다 ありがとうございます。 Thank you