Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AB Testing tips for Mobile

AB Testing tips for Mobile

AB Testing tips for Mobile. Tips with Remote Config

Tomoaki Imai

July 13, 2016

Transcript

28 slides
  1. twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari

    US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ
  2. Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣

    ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍
  3. History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing

    Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ
  4. Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣

    ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ
  5. AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal,

    Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ
  6. "QQUJNJ[F *O)PVTF 3FNPUF $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓

    ̋ ̋ ઃఆͷ ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)
  7. Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ -

    R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ
  8. Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭,

    ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU$POFOU  /PUJO5FTU EJTDPVOU DPMPSSFE UFYUlz  $POUSPM EJTDPVOU DPMPSSFE UFYUlz  &YQFSJNFOU EJTDPVOU DPMPSSFE  UFYUl4VNNFSDBNQBJHOz
  9. • 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content ->

    { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT ͸ςετ֎ͱ͢Δ JOU͕EFGBVMUͷͨΊ
  10. remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task)

    { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ
  11. • ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation

    ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ
  12. Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String

    text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }
  13. Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB

    Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ

More Decks by Tomoaki Imai

Other Decks in Programming