Slide 1
Slide 1 text
Jay Hori
A/B Testing at LINE NEWS
σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑ΔιϑτΣΞ։ൃઓུ
Slide 2
Slide 2 text
Software Engineer
Jay Hori
LINEגࣜձࣾ
ϑϩϯτΤϯυ։ൃηϯλʔ/UIT
Hartwick College (New York)
B.A. Cognitive Science
ݴޠɾϑϨʔϜϫʔΫ
TypeScript, JavaScript, React, Python
Slide 3
Slide 3 text
A/B Testing at LINE NEWS
σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑ΔιϑτΣΞ։ൃઓུ
1. LINE NEWSʹ͍ͭͯ
2. σϓϩΠͱϦϦʔεͷ
3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
4. Feature Con
fi
gurationͷ׆༻
5. ·ͱΊ
Slide 4
Slide 4 text
A/B Testing at LINE NEWS
σʔλʹجͮ͘ҙࢥܾఆΛࢧ͑ΔιϑτΣΞ։ൃઓུ
1. LINE NEWSʹ͍ͭͯ
2. σϓϩΠͱϦϦʔεͷ
3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
4. Feature Con
fi
gurationͷ׆༻
5. ·ͱΊ
Slide 5
Slide 5 text
7,700ສਓ
MAU (20218݄࣌)
Slide 6
Slide 6 text
• σʔλʹجͮ͘ҙࢥܾఆ
• සൟͳA/Bςετ͕ඞཁ
• A/Bςετӡ༻ͷੜ࢈ੑͱޮɺ
ͦͯ҆͠શͳϦϦʔε͕ඞਢ
LINE NEWS
Slide 7
Slide 7 text
σϓϩΠͱϦϦʔεͷ
Decoupling deployment from releases
σϓϩΠ
ιʔείʔυΛ
ϓϩμΫγϣϯڥ
σϓϩΠ͢Δ
ϦϦʔε
ػೳΛϢʔβʔ
ϦϦʔε͢Δ
Slide 8
Slide 8 text
A/B Testing at LINE NEWS
Taking full control of experimental features
1. LINE NEWSʹ͍ͭͯ
2. σϓϩΠͱϦϦʔεͷ
3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
4. Feature Con
fi
gurationͷ׆༻
5. ·ͱΊ
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Feature Flags
ػೳΛΦϯΦϑ͢Δ
O
ff
isEnabled: false
On
isEnabled: true
Slide 11
Slide 11 text
A/Bςετ
Control Treatment
ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
Slide 12
Slide 12 text
A/Bςετ
ίϯτϩʔϧάϧʔϓͱτϦʔτϝϯτάϧʔϓΛൺֱ
Control Treatment 1 Treatment 2
Slide 13
Slide 13 text
A/Bςετ
ΧϧʔηϧͱλΠϜϥΠϯΛൺֱ
Carousel
1%
Timeline
1%
Slide 14
Slide 14 text
Remote Config
τϦʔτϝϯτάϧʔϓͷόϦΤʔγϣϯύϥϝʔλʔઃఆ
Control
isEnabled: false
Treatment 1
isEnabled: true
numberOfArticlesToLoad: 30
Treatment 2
isEnabled: true
numberOfArticlesToLoad: 10
Slide 15
Slide 15 text
ΧφϦΞϦϦʔε
খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞτ͢Δ
Control
isEnabled: false
1%
Slide 16
Slide 16 text
ΧφϦΞϦϦʔε
খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞτ͢Δ
Control
isEnabled: false
50%
Slide 17
Slide 17 text
ΧφϦΞϦϦʔε
খ͞ͳαϒηοτ͔ΒঃʑʹϩʔϧΞτ͢Δ
Control
isEnabled: false
100%
Slide 18
Slide 18 text
A/B Testing at LINE NEWS
Taking full control of experimental features
1. LINE NEWSʹ͍ͭͯ
2. σϓϩΠͱϦϦʔεͷ
3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
4. Feature Con
fi
gurationͷ׆༻
5. ·ͱΊ
Slide 19
Slide 19 text
Feature
Configuration
Feature Flags
A/B Tests
Remote Config
Canary Releases
=
Slide 20
Slide 20 text
ϦϞʔτཧ
• JSONϑΥʔϚοτ
• ϦϞʔτϦϙδτϦͰཧ
• Con
fi
gurationαʔόʔ (Central Dogma) ʹอଘ͞ΕΔ
Feature Configuration
Slide 21
Slide 21 text
JSONྫ
• Feature
fl
ag
• isEnabled
• Remote con
fi
g
• numberOfArticlesToLoadInitially
• title
Feature Configuration
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
A/B Testing at LINE NEWS
Taking full control of experimental features
1. LINE NEWSʹ͍ͭͯ
2. σϓϩΠͱϦϦʔεͷ
3. ࣄྫ: For YouͷϢʔβʔΠϯλʔϑΣʔε
4. Feature Con
fi
gurationͷ׆༻
5. ·ͱΊ
Slide 26
Slide 26 text
·ͱΊ
1. େنϓϩμΫτͰA/Bςετͷੜ࢈ੑͱޮ͕ॏཁ
2. σϓϩΠͱϦϦʔεΛ͢ΔػߏΛ࡞Δ
3. Feature Con
fi
gurationͰޮԽ
4. ΧφϦΞϦϦʔεͰ҆શʹϦϦʔε͢Δ