Mobile AB Testing tips ~ RemoteConfigΛఴ͑ͯ~ Mercari, Inc. Tomoaki Imai

twitter: @tomoaki_imai github: tomoima525 ࠓҪ ஐষ Android Engineer 2016.1 Mercari US ग़޲ ඒຯ͍͠ΧϨʔͷΑ͏ʹ ਓΛ޾ͤʹग़དྷΔΑ͏ʹͳΓ͍ͨ

Mercari in US Mercari USΦϑΟεΛ Sanfranciscoʹઃཱ 2014.9 ϦϦʔε 1200ສDLಥഁ ࠓ೔΋ݩؾʹαʔϏε ֦େத!

Why AB Testing is important?

Great things about AB Testing • ࡉ͔͍ௐ੔ʹΑΔޮՌͷ࠷େԽ • େ୾ͳมߋʹର͢ΔϦεΫͷ࠷খԽ • ݕূͷύϥϨϧԽʹΑΔظؒͷ୹ॖԽ

Things need to take care • ਖ਼͍͠ޮՌଌఆ ‣ ҙຯͷͳ͍ςετΛ΍͍ͬͯͳ͍͔ ‣ ฼਺ʁάϧʔϐϯάʁ༗ҙࠩ? • ͍ͭͰ΋σϑΥϧτʹ໭ͤΔ ‣ Ξοϓσʔτͳ͘ରԠͰ͖Δ ‣ ιʔείʔυΛԚ͞ͳ͍

࠷ߴͷAB Test͠Α͏ͥ • ࣮ྫΛ௨ͯ͡ޮՌతͳAB Testingͷํ๏Λ஌Δ • Firebase RemoteConfigͱ͍͏ڧྗͳAB Test πʔ ϧΛ࢖࣮ͬͯݱ͢Δํ๏Λ஌Δ

Before we go…

AB Testing at Mercari

History of AB Test AB Testಋೖ ໨త: ͞ΒͳΔػೳվળ Apptimize(AB Testing Tool) + Segment(τϥοΩϯά) + mixpanel(෼ੳ) 2014 Ն 2015 य़ AB TestͷΠϯϋ΢εԽ ݱࡏ τϥοΩϯάͷΠϯϋ΢εԽ (Pascal) 2015 Ն ৗ࣌ 10݅લޙ ͷAB TestΛ࣮ࢪ ෼ੳͷ Big Query΁ͷҠߦ

Type of AB Test • UI Test ‣ ݟͨ໨ɺϨΠΞ΢τͷมߋ ‣ ྫ) Ձ֨දࣔํ๏ͷมߋ • Flow Test ‣ ಛఆϑϩʔͷ௥Ճ/εΩοϓ ‣ ྫ)ೖྗ߲໨ͷ௥Ճ/εΩοϓ • New feature Test ‣ ৽ػೳͷදࣔ/ඇදࣔ ‣ ྫ) ύʔιφϥΠζػೳ

AB Testing tool + ίϯιʔϧ্ͰτϥοΩϯάઃఆͰ͖Δ + ಈతʹUIΛมߋͰ͖Δ - τϥοΩϯάσʔλͷऔΓ͜΅͠༗ - σʔλͷexport͕ग़དྷͳ͍ - ஋ஈ - ࡉ͔͍λʔήοτઃఆ͕೉͍͠

AB Testing tool • In House + αʔόαΠυ͔Βॊೈͳλʔήοτઃఆ + Pascal, Big Queryͱ૊Έ߹Θ༷ͤͯʑͳ੾Γޱ͔Β෼ੳ AB Test Client ͸ ϥΠϒϥϦԽ https://github.com/mercari/siberi-android - Apptimize like ͳ Interface - ςετσʔλΛDBͰ؅ཧ - iOS΋ۙ೔ϦϦʔε༧ఆ

What about Firebase RemoteConfig?

"QQUJNJ[F *O)PVTF 3FNPUF $POpH ಋೖ༰қ͞ ̋ ˚ ˕ Ձ֨ ☓ ̋ ̋ ઃఆͷ ॊೈੑ ☓ ̋ ˚ ֦ுੑ ☓ ˕ ˚ • MercariͰͷಋೖ༧ఆ͸ͳ͠ ‣ ৽نࣄۀͰ͸ಋೖ༰қ͞ɺՁ֨ͷ؍఺ͰקΊ͍ͨ ‣ ·ͩ଍Γͯͳ͍෦෼͕͋ΔͷͰ࢖͏࣌͸஫ҙ(ޙड़)

AB Testing Tips

Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ ྫ) αϯϓϧ਺ - AB TestͷԾઆʹରͯ͠ద੾ͳαϯϓϧ਺͕ઃఆͰ͖͍ͯΔ͔ - R ͰܭࢉՄೳ QPXFSQSPQUFTU Q Q TJHMFWFM QPXFS ex) AB ςετͷ݁Ռߪೖ཰͕ 5% -> 6%ͱͳΔͱ͍͏Ծઆ ࢀߟ: http://oku.edu.mie-u.ac.jp/~okumura/stat/150922.html sig.level -> ༗ҙਫ४ (Ծઆ͕ਖ਼͍͠ͱޡͬͯ൑அ͢Δ֬཰) power -> ݕఆྗ (Ծઆ͕ਖ਼͍͠ͱ൑அ͢Δ֬཰) O -> 8100݅Ҏ্ͷαϯϓϧ͕ඞཁ

Tips1. TargetઃఆΛద੾ʹ͢Δ • ౷ܭֶతʹҙຯ͕͋Δ͔ʹ഑ྀ άϧʔϐϯά - ςετಉ͕࢜Өڹ͠ͳ͍Α͏௚ަͯ͠άϧʔϐϯά͞Ε͍ͯΔ͔ - ࣮ݧ܈, ରর܈ʹରͯ͠1 : 1ͰৼΓ෼͚ΒΕ͍ͯΔ͔ 5FTU" $POUSPM 5FTU" &YQFSJNFOU 5FTU#$POUSPM 5FTU#&YQFSJNFOU

Tips1. TargetઃఆΛద੾ʹ͢Δ • Firebase Ͱ͸ॊೈʹTargetΛઃఆͰ͖Δ&ॏෳΛڐ͢ ͷͰར༻ऀଆͰ஫ҙ͕ඞཁ!! μϝͳྫ ରর܈࣮ݧ܈ͷൺ཰ʹେ͖͕ࠩ͋͘ Δ )0-*%":@130.0ͷςετͱ௚ަ͠ ͍ͯͳ͍)0-*%":@130.0ͷ݁ ՌͷӨڹΛड͚ΔՄೳੑ͕͋Δ ಉ͡ൺ཰ʹ͢Δ͔ɺॏͳΒͳ͍Α ͏ʹ͢Δ

Tips1. TargetઃఆΛద੾ʹ͢Δ • Best practice? ׂΓৼΓͷ$POEJUJPOΛఆٛ $POEJUJPOΛઃఆ͢Δ ͦΕҎ ֎ͷઃఆ͸͠ͳ͍

Tips 2. VariantϕʔεͰςετ͢Δ • AB Testͷ࢓ํ ‣ Keyϕʔε ‣ Buttonͷ৭, ςΩετ,ͦΕͧΕʹ஋Λઃఆ͢Δ ‣ Variantϕʔε ‣ ςετ಺༰ΛέʔεผͰఆٛ͢Δ 7BSJBOU &YQFSJNFOU$POFOU /PUJO5FTU EJTDPVOU DPMPSSFE UFYUlz $POUSPM EJTDPVOU DPMPSSFE UFYUlz &YQFSJNFOU EJTDPVOU DPMPSSFE UFYUl4VNNFSDBNQBJHOz

• 1ͭͷςετͷઃఆ߲໨͕૿͑ΔͱKeyϕʔε͸൥ࡶʹͳΔ => ࡉ͔͍ॲཧ͸Clientଆʹ೚ͤͨํ͕҆શ • αʔόଆ͸VariantΛৼΓ෼͚Δ͚ͩʹ͢Δͱɺςετ಺༰͕؅ ཧ͠΍͍͢ Siberi.runTest(Test.HOLIDAY_PROMO, content -> { switch(content.getVariant()){ case 0: break; // not test group case 1: break; // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } }); Tips 2. VariantϕʔεͰςετ͢Δ 5JQT ͸ςετ֎ͱ͢Δ JOU͕EFGBVMUͷͨΊ

remoteConfig.fetch(len).addOnCompleteListener(this, new OnCompleteListener() { @Override public void onComplete(@NonNull Task task) { switch ((int)remoteConfig.getLong(HOLIDAY_PROMO)){ case 0: // not test group case 1: // control group case 2: setDiscount(30); button.setBackgroundColor(red); button.setText("Summer Campaign"); break; } } }); Tips 2. VariantϕʔεͰςετ͢Δ • RemoteConfig SFNPUF$POpHHFU-POH͸஋͕ औΕͳ͍৔߹͸͕ฦΔ

Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • AB Test͸֤έʔεͰͷϨϏϡʔϫʔɺQAͷෛ୲͕େ͖͍ => ϩʔΧϧ্Ͱ੾ସ͑ΒΕΔ࢓૊Έ ‣ ςετσʔλΛϩʔΧϧDBʹอଘ ‣ Stetho Ͱςετσʔλͷ஋Λมߋ 42-Ͱ஋Λߋ৽Ͱ͖Δ DISPNFJOTQFDU ͰࢀরͰ͖Δ

Tips 3. ݕূ͠΍͍͢࢓૊ΈΛ࡞Δ • RemoteConfig͸ίϯιʔϧ্ͰͷϦΞϧλΠϜͷςετσʔλมߋ͕Մೳ • มߋ࿙Ε/ϛεͰଞͷݕূऀ΁ͷΠϯύΫτ͕͋ΔͷͰ஫ҙ͕ඞཁ • DEV؀ڥͷΈςετσʔλΛDBͰ؅ཧ͢Δͱ͍ͬͨ࢓૊Έ͕͋Δͱྑͦ͞͏

• ςετ࣮ࢪޙͷ͋Δ͋Δ ‣ ͋ΔςετͰ͔͠࢖Θͳ͍ϑΟʔϧυม਺/ϝιουΛফ͠ ๨ΕΔ ‣ ͲͷςετͷͨΊͷϑΟʔϧυม਺/ϝιου͔൑அͣ͠Β͍ • @ForExperiment Annotation ‣ ςετίʔυʹ෇Ճ͢ΔAnnotation ‣ @ForExperiment(Test໊) ͱ͢Δ͜ͱͰՄಡੑΛ্͛Δ Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ

Tips 4. AnnotationʹΑΔςετίʔυͷ؅ཧ @ForExperiment(Experiments.TEST_002) TextView testTv; @ForExperiment(Experiments.TEST_002) private void changeText(String text){ testTv.setText(text); } public interface Experiments { String TEST_001 = "test_001_change_button_color"; String TEST_002 = "test_002_change_text"; } • @ForExperimentͰͲͷTest༻ͷม਺/ϝιουͳͷ͔໌ࣔతʹͳΔ • Testऴྃޙ͸ Experiments.java಺ͷStringΛফͤ͹ফ͠࿙Ε͕ແ͘ͳΔ @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.FIELD, ElementType.TYPE, ElementType.LOCAL_VARIABLE, ElementType.METHOD})
 public @interface ForExperiment {
 String value();
 }

Wrap up • ࠓ͙͢ʹAB TestΛ࢝ΊΔͳΒ Firebase RemoteConfig ͸Φεεϝ(͚Ͳར༻ʹ͸·ͩ·ͩ஫ҙ) • ͪΐͬͱͨ͠഑ྀͰҙຯͷ͋ΔAB Test ͕ޮ཰ྑ࣮͘ ࢪͰ͖Δ • ద੾ͳλʔήοτઃఆ • Variant ϕʔεͰͷςετ • ಈ࡞ݕূ͠΍͍͢࢓૊Έ • AnnotationͰςετ༻ίʔυΛ؅ཧ