AB Testing tips for Mobile

AB Testing tips for Mobile

AB Testing tips for Mobile. Tips with Remote Config

860bf040d996601213e05747dd661c23?s=128

Tomoaki Imai

July 13, 2016
Tweet

Transcript

  1. Mobile AB Testing tips ~ RemoteConfigΛఴ͑ͯ~ Mercari, Inc. Tomoaki Imai

  2. twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari

    US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ
  3. Mercari in US Mercari USΦϑΟεΛ Sanfranciscoʹઃཱ 2014.9 ϦϦʔε 1200ສDLಥഁ ࠓ೔΋ݩؾʹαʔϏε

    ֦େத!
  4. Why AB Testing is important?

  5. Great things about AB Testing • ࡉ͔͍ௐ੔ʹΑΔޮՌͷ࠷େԽ • େ୾ͳมߋʹର͢ΔϦεΫͷ࠷খԽ •

    ݕূͷύϥϨϧԽʹΑΔظؒͷ୹ॖԽ
  6. Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣

    ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍
  7. ࠷ߴͷAB Test͠Α͏ͥ • ࣮ྫΛ௨ͯ͡ޮՌతͳAB Testingͷํ๏Λ஌Δ • Firebase RemoteConfigͱ͍͏ڧྗͳAB Test πʔ

    ϧΛ࢖࣮ͬͯݱ͢Δํ๏Λ஌Δ
  8. Before we go…

  9. AB Testing at Mercari

  10. History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing

    Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ
  11. Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣

    ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ
  12. AB Testing tool + ίϯιʔϧ্ͰτϥοΩϯάઃఆͰ͖Δ + ಈతʹUIΛมߋͰ͖Δ - τϥοΩϯάσʔλͷऔΓ͜΅͠༗ -

    σʔλͷexport͕ग़དྷͳ͍ - ஋ஈ - ࡉ͔͍λʔήοτઃఆ͕೉͍͠
  13. AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal,

    Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ
  14. What about Firebase RemoteConfig?

  15. "QQUJNJ[F *O)PVTF 3FNPUF $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓

    ̋ ̋ ઃఆͷ ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)
  16. AB Testing Tips

  17. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ -

    R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ
  18. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ άϧʔϐϯά - ςετಉ͕࢜Өڹ͠ͳ͍Α͏௚ަͯ͠άϧʔϐϯά͞Ε͍ͯΔ͔ - ࣮ݧ܈, ରর܈ʹରͯ͠1

    : 1ͰৼΓ෼͚ΒΕ͍ͯΔ͔ 5FTU" $POUSPM 5FTU" &YQFSJNFOU 5FTU#$POUSPM 5FTU#&YQFSJNFOU
  19. Tips1. TargetઃఆΛద੾ʹ͢Δ • Firebase Ͱ͸ॊೈʹTargetΛઃఆͰ͖Δ&ॏෳΛڐ͢ ͷͰར༻ऀଆͰ஫ҙ͕ඞཁ!! μϝͳྫ ରর܈࣮ݧ܈ͷൺ཰ʹେ͖͕ࠩ͋͘ Δ )0-*%":@130.0ͷςετͱ௚ަ͠

    ͍ͯͳ͍)0-*%":@130.0ͷ݁ ՌͷӨڹΛड͚ΔՄೳੑ͕͋Δ ಉ͡ൺ཰ʹ͢Δ͔ɺॏͳΒͳ͍Α ͏ʹ͢Δ
  20. Tips1. TargetઃఆΛద੾ʹ͢Δ • Best practice? ׂΓৼΓͷ$POEJUJPOΛఆٛ $POEJUJPOΛઃఆ͢Δ ͦΕҎ ֎ͷઃఆ͸͠ͳ͍

  21. Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭,

    ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU$POFOU  /PUJO5FTU EJTDPVOU DPMPSSFE UFYUlz  $POUSPM EJTDPVOU DPMPSSFE UFYUlz  &YQFSJNFOU EJTDPVOU DPMPSSFE  UFYUl4VNNFSDBNQBJHOz
  22. • 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content ->

    { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT ͸ςετ֎ͱ͢Δ JOU͕EFGBVMUͷͨΊ
  23. remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task)

    { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ
  24. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • AB Test͸֤έʔεͰͷϨϏϡʔϫʔɺQAͷෛ୲͕େ͖͍ => ϩʔΧϧ্Ͱ੾ସ͑ΒΕΔ࢓૊Έ ‣ ςετσʔλΛϩʔΧϧDBʹอଘ

    ‣ Stetho Ͱςετσʔλͷ஋Λมߋ 42-Ͱ஋Λߋ৽Ͱ͖Δ DISPNFJOTQFDU ͰࢀরͰ͖Δ
  25. Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • RemoteConfig͸ίϯιʔϧ্ͰͷϦΞϧλΠϜͷςετσʔλมߋ͕Մೳ • มߋ࿙Ε/ϛεͰଞͷݕূऀ΁ͷΠϯύΫτ͕͋ΔͷͰ஫ҙ͕ඞཁ • DEV؀ڥͷΈςετσʔλΛDBͰ؅ཧ͢Δͱ͍ͬͨ࢓૊Έ͕͋Δͱྑͦ͞͏

  26. • ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation

    ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ
  27. Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String

    text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }
  28. Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB

    Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ