AB Testing tips for Mobile

Transcript

1. Mobile AB Testing tips ~ RemoteConfigΛఴ͑ͯ~ Mercari, Inc. Tomoaki Imai

2. twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari

   US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ

3. Mercari in US Mercari USΦϑΟεΛ Sanfranciscoʹઃཱ 2014.9 ϦϦʔε 1200ສDLಥഁ ࠓ೔΋ݩؾʹαʔϏε

   ֦େத!

4. Why AB Testing is important?

5. Great things about AB Testing • ࡉ͔͍ௐ੔ʹΑΔޮՌͷ࠷େԽ • େ୾ͳมߋʹର͢ΔϦεΫͷ࠷খԽ •

   ݕূͷύϥϨϧԽʹΑΔظؒͷ୹ॖԽ

6. Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣

   ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍

7. ࠷ߴͷAB Test͠Α͏ͥ • ࣮ྫΛ௨ͯ͡ޮՌతͳAB Testingͷํ๏Λ஌Δ • Firebase RemoteConfigͱ͍͏ڧྗͳAB Test πʔ

   ϧΛ࢖࣮ͬͯݱ͢Δํ๏Λ஌Δ

8. Before we go…

9. AB Testing at Mercari

10. History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing

    Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ

11. Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣

    ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ

12. AB Testing tool + ίϯιʔϧ্ͰτϥοΩϯάઃఆͰ͖Δ + ಈతʹUIΛมߋͰ͖Δ - τϥοΩϯάσʔλͷऔΓ͜΅͠༗ -

    σʔλͷexport͕ग़དྷͳ͍ - ஋ஈ - ࡉ͔͍λʔήοτઃఆ͕೉͍͠

13. AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal,

    Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ

14. What about Firebase RemoteConfig?

15. "QQUJNJ[F *O
    )PVTF 3FNPUF
    $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓

    ̋ ̋ ઃఆͷ
    ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)

16. AB Testing Tips

17. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ -

    R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q
    TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O
    
    
    
    
    
    
    
    
    
    
    
    
    
    -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ

18. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ άϧʔϐϯά - ςετಉ͕࢜Өڹ͠ͳ͍Α͏௚ަͯ͠άϧʔϐϯά͞Ε͍ͯΔ͔ - ࣮ݧ܈, ରর܈ʹରͯ͠1

    : 1ͰৼΓ෼͚ΒΕ͍ͯΔ͔ 5FTU
    "
    $POUSPM 5FTU
    "
    &YQFSJNFOU 5FTU
    #
    $POUSPM 5FTU
    #
    &YQFSJNFOU

19. Tips1. TargetઃఆΛద੾ʹ͢Δ • Firebase Ͱ͸ॊೈʹTargetΛઃఆͰ͖Δ&ॏෳΛڐ͢ ͷͰར༻ऀଆͰ஫ҙ͕ඞཁ!! μϝͳྫ ରর܈࣮ݧ܈ͷൺ཰ʹେ͖͕ࠩ͋͘ Δ )0-*%":@130.0ͷςετͱ௚ަ͠

    ͍ͯͳ͍
    
    )0-*%":@130.0ͷ݁ ՌͷӨڹΛड͚ΔՄೳੑ͕͋Δ
    
    ಉ͡ൺ཰ʹ͢Δ͔ɺॏͳΒͳ͍Α ͏ʹ͢Δ

20. Tips1. TargetઃఆΛద੾ʹ͢Δ • Best practice? ׂΓৼΓͷ$POEJUJPOΛఆٛ $POEJUJPOΛઃఆ͢Δ ͦΕҎ ֎ͷઃఆ͸͠ͳ͍

21. Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭,

    ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU
    $POFOU  /PU
    JO
    5FTU EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    lz  $POUSPM EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    lz  &YQFSJNFOU EJTDPVOU
    
    
    DPMPS
    
    SFE
    UFYU
    l4VNNFS
    DBNQBJHOz

22. • 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content ->

    { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT
    
    
    ͸ςετ֎ͱ͢Δ
    JOU͕EFGBVMU
    ͷͨΊ

23. remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task)

    { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ

24. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • AB Test͸֤έʔεͰͷϨϏϡʔϫʔɺQAͷෛ୲͕େ͖͍ => ϩʔΧϧ্Ͱ੾ସ͑ΒΕΔ࢓૊Έ ‣ ςετσʔλΛϩʔΧϧDBʹอଘ

    ‣ Stetho Ͱςετσʔλͷ஋Λมߋ
    42-Ͱ஋Λߋ৽Ͱ͖Δ
    DISPNFJOTQFDU
    
    ͰࢀরͰ͖Δ

25. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • RemoteConfig͸ίϯιʔϧ্ͰͷϦΞϧλΠϜͷςετσʔλมߋ͕Մೳ • มߋ࿙Ε/ϛεͰଞͷݕূऀ΁ͷΠϯύΫτ͕͋ΔͷͰ஫ҙ͕ඞཁ • DEV؀ڥͷΈςετσʔλΛDBͰ؅ཧ͢Δͱ͍ͬͨ࢓૊Έ͕͋Δͱྑͦ͞͏

26. • ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation

    ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ

27. Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String

    text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }

28. Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB

    Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ