AB Testing tips for Mobile

AB Testing tips for Mobile. Tips with Remote Config

Transcript

1. Mobile AB Testing tips ~ RemoteConfigΛఴ͑ͯ~ Mercari, Inc. Tomoaki Imai
2. twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ
3. Mercari in US Mercari USΦϑΟεΛ Sanfranciscoʹઃཱ 2014.9 ϦϦʔε 1200ສDLಥഁ ࠓ೔΋ݩؾʹαʔϏε ֦େத!
4. Why AB Testing is important?
5. Great things about AB Testing • ࡉ͔͍ௐ੔ʹΑΔޮՌͷ࠷େԽ • େ୾ͳมߋʹର͢ΔϦεΫͷ࠷খԽ • ݕূͷύϥϨϧԽʹΑΔظؒͷ୹ॖԽ
6. Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣ ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍
7. ࠷ߴͷAB Test͠Α͏ͥ • ࣮ྫΛ௨ͯ͡ޮՌతͳAB Testingͷํ๏Λ஌Δ • Firebase RemoteConfigͱ͍͏ڧྗͳAB Test πʔ ϧΛ࢖࣮ͬͯݱ͢Δํ๏Λ஌Δ
8. Before we go…
9. AB Testing at Mercari
10. History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ
11. Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣ ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ
12. AB Testing tool + ίϯιʔϧ্ͰτϥοΩϯάઃఆͰ͖Δ + ಈతʹUIΛมߋͰ͖Δ - τϥοΩϯάσʔλͷऔΓ͜΅͠༗ - σʔλͷexport͕ग़དྷͳ͍ - ஋ஈ - ࡉ͔͍λʔήοτઃఆ͕೉͍͠
13. AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal, Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ
14. What about Firebase RemoteConfig?
15. "QQUJNJ[F *O
    )PVTF 3FNPUF
    $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓ ̋ ̋ ઃఆͷ
    ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)
16. AB Testing Tips
17. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ - R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q
    TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O
    
    
    
    
    
    
    
    
    
    
    
    
    
    -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ
18. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ άϧʔϐϯά - ςετಉ͕࢜Өڹ͠ͳ͍Α͏௚ަͯ͠άϧʔϐϯά͞Ε͍ͯΔ͔ - ࣮ݧ܈, ରর܈ʹରͯ͠1 : 1ͰৼΓ෼͚ΒΕ͍ͯΔ͔ 5FTU
    "
    $POUSPM 5FTU
    "
    &YQFSJNFOU 5FTU
    #
    $POUSPM 5FTU
    #
    &YQFSJNFOU
19. Tips1. TargetઃఆΛద੾ʹ͢Δ • Firebase Ͱ͸ॊೈʹTargetΛઃఆͰ͖Δ&ॏෳΛڐ͢ ͷͰར༻ऀଆͰ஫ҙ͕ඞཁ!! μϝͳྫ ରর܈࣮ݧ܈ͷൺ཰ʹେ͖͕ࠩ͋͘ Δ )0-*%":@130.0ͷςετͱ௚ަ͠ ͍ͯͳ͍
    
    )0-*%":@130.0ͷ݁ ՌͷӨڹΛड͚ΔՄೳੑ͕͋Δ
    
    ಉ͡ൺ཰ʹ͢Δ͔ɺॏͳΒͳ͍Α ͏ʹ͢Δ
20. Tips1. TargetઃఆΛద੾ʹ͢Δ • Best practice? ׂΓৼΓͷ$POEJUJPOΛఆٛ $POEJUJPOΛઃఆ͢Δ ͦΕҎ ֎ͷઃఆ͸͠ͳ͍
21. Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭, ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU
    $POFOU  /PU
    JO
    5FTU EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    lz  $POUSPM EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    lz  &YQFSJNFOU EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    l4VNNFS
    DBNQBJHOz
22. • 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content -> { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT
    
    
    ͸ςετ֎ͱ͢Δ
    JOU͕EFGBVMU
    ͷͨΊ
23. remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task) { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ
24. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • AB Test͸֤έʔεͰͷϨϏϡʔϫʔɺQAͷෛ୲͕େ͖͍ => ϩʔΧϧ্Ͱ੾ସ͑ΒΕΔ࢓૊Έ ‣ ςετσʔλΛϩʔΧϧDBʹอଘ ‣ Stetho Ͱςετσʔλͷ஋Λมߋ
    42-Ͱ஋Λߋ৽Ͱ͖Δ
    DISPNFJOTQFDU
    
    ͰࢀরͰ͖Δ
25. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • RemoteConfig͸ίϯιʔϧ্ͰͷϦΞϧλΠϜͷςετσʔλมߋ͕Մೳ • มߋ࿙Ε/ϛεͰଞͷݕূऀ΁ͷΠϯύΫτ͕͋ΔͷͰ஫ҙ͕ඞཁ • DEV؀ڥͷΈςετσʔλΛDBͰ؅ཧ͢Δͱ͍ͬͨ࢓૊Έ͕͋Δͱྑͦ͞͏
26. • ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ
27. Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }
28. Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ