Share
Embed
AB Testing tips for Mobile
AB Testing tips for Mobile. Tips with Remote Config
Tomoaki Imai
July 13, 2016
Transcript
28 slides
-
twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari
US ग़ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ -
Great things about AB Testing • ࡉ͔͍ௐʹΑΔޮՌͷ࠷େԽ • େͳมߋʹର͢ΔϦεΫͷ࠷খԽ •
ݕূͷύϥϨϧԽʹΑΔظؒͷॖԽ -
Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ͍ͬͯͳ͍͔ ‣
ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰσϑΥϧτʹͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍ -
History of AB Test AB Testಋೖ త: ͞ΒͳΔػೳվળ Apptimize(AB Testing
Tool) + Segment(τϥοΩϯά) + mixpanel(ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋεԽ ݱࡏ τϥοΩϯάͷΠϯϋεԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ੳͷ Big QueryͷҠߦ -
Type of AB Test • UI Test ‣ ݟͨɺϨΠΞτͷมߋ ‣
ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷՃ/εΩοϓ ‣ ྫ)ೖྗ߲ͷՃ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ -
AB Testing tool + ίϯιʔϧ্ͰτϥοΩϯάઃఆͰ͖Δ + ಈతʹUIΛมߋͰ͖Δ - τϥοΩϯάσʔλͷऔΓ͜΅͠༗ -
σʔλͷexport͕ग़དྷͳ͍ - ஈ - ࡉ͔͍λʔήοτઃఆ͕͍͠ -
AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal,
Big QueryͱΈ߹Θ༷ͤͯʑͳΓޱ͔Βੳ AB Test Client ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰཧ - iOSۙϦϦʔε༧ఆ -
"QQUJNJ[F *O)PVTF 3FNPUF $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓
̋ ̋ ઃఆͷ ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆͳ͠ ‣ ৽نࣄۀͰಋೖ༰қ͞ɺՁ֨ͷ؍ͰקΊ͍ͨ ‣ ·ͩΓͯͳ͍෦͕͋ΔͷͰ͏࣌ҙ(ޙड़) -
Tips1. TargetઃఆΛదʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹྀ ྫ) αϯϓϧ - AB TestͷԾઆʹରͯ͠దͳαϯϓϧ͕ઃఆͰ͖͍ͯΔ͔ -
R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯஅ͢Δ֬) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱஅ͢Δ֬) O -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ -
Tips1. TargetઃఆΛదʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹྀ άϧʔϐϯά - ςετಉ͕࢜Өڹ͠ͳ͍Α͏ަͯ͠άϧʔϐϯά͞Ε͍ͯΔ͔ - ࣮ݧ܈, ରর܈ʹରͯ͠1
: 1ͰৼΓ͚ΒΕ͍ͯΔ͔ 5FTU" $POUSPM 5FTU" &YQFSJNFOU 5FTU#$POUSPM 5FTU#&YQFSJNFOU -
Tips1. TargetઃఆΛదʹ͢Δ • Firebase ͰॊೈʹTargetΛઃఆͰ͖Δ&ॏෳΛڐ͢ ͷͰར༻ऀଆͰҙ͕ඞཁ!! μϝͳྫ ରর܈࣮ݧ܈ͷൺʹେ͖͕ࠩ͋͘ Δ )0-*%":@130.0ͷςετͱަ͠
͍ͯͳ͍)0-*%":@130.0ͷ݁ ՌͷӨڹΛड͚ΔՄೳੑ͕͋Δ ಉ͡ൺʹ͢Δ͔ɺॏͳΒͳ͍Α ͏ʹ͢Δ -
Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷํ ‣ Keyϕʔε ‣ Buttonͷ৭,
ςΩετ,ͦΕͧΕʹΛઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU$POFOU /PUJO5FTU EJTDPVOU DPMPSSFE UFYUlz $POUSPM EJTDPVOU DPMPSSFE UFYUlz &YQFSJNFOU EJTDPVOU DPMPSSFE UFYUl4VNNFSDBNQBJHOz -
• 1ͭͷςετͷઃఆ߲͕૿͑ΔͱKeyϕʔεࡶʹͳΔ => ࡉ͔͍ॲཧClientଆʹͤͨํ͕҆શ • αʔόଆVariantΛৼΓ͚Δ͚ͩʹ͢Δͱɺςετ༰͕ ཧ͍͢͠ Siberi.runTest(Test.HOLIDAY_PROMO, content ->
{ switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT ςετ֎ͱ͢Δ JOU͕EFGBVMUͷͨΊ -
remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener<Void>() { @Override public void onComplete(@NonNull Task<Void> task)
{ switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͕ औΕͳ͍߹͕ฦΔ -
Tips 3. ݕূ͍͢͠ΈΛ࡞Δ • AB Test֤έʔεͰͷϨϏϡʔϫʔɺQAͷෛ୲͕େ͖͍ => ϩʔΧϧ্Ͱସ͑ΒΕΔΈ ‣ ςετσʔλΛϩʔΧϧDBʹอଘ
‣ Stetho ͰςετσʔλͷΛมߋ 42-ͰΛߋ৽Ͱ͖Δ DISPNFJOTQFDU ͰࢀরͰ͖Δ -
• ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠Θͳ͍ϑΟʔϧυม/ϝιουΛফ͠ ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม/ϝιου͔அͣ͠Β͍ • @ForExperiment Annotation
‣ ςετίʔυʹՃ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷཧ -
Tips 4. AnnotationʹΑΔςετίʔυͷཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String
text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ Experiments.javaͷStringΛফͤফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD}) public @interface ForExperiment { String value(); } -
Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig Φεεϝ(͚Ͳར༻ʹ·ͩ·ͩҙ) • ͪΐͬͱྀͨ͠Ͱҙຯͷ͋ΔAB
Test ͕ޮྑ࣮͘ ࢪͰ͖Δ • దͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͍͢͠Έ • AnnotationͰςετ༻ίʔυΛཧ