Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

社内DS論文読み会の資料です

Duan, Weitao, Shan Ba, and Chunzhe Zhang. "Online Experimentation with Surrogate Metrics: Guidelines and a Case Study." Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021.

D77e6b2d469947a4792ab062d466350b?s=128

Takashi Nishibayashi

April 02, 2021
Tweet

More Decks by Takashi Nishibayashi

Other Decks in Technology

Transcript

  1. ࿦จ঺հ 0OMJOF&YQFSJNFOUBUJPOXJUI4VSSPHBUF.FUSJDT (VJEFMJOFTBOEB$BTF4UVEZ %4࿦จಡΈձ! ੢ྛ 5BLBTIJ/JTIJCBZBTIJ %VBO 8FJUBP 4IBO#B BOE$IVO[IF;IBOH0OMJOF&YQFSJNFOUBUJPOXJUI4VSSPHBUF.FUSJDT(VJEFMJOFTBOEB$BTF4UVEZ

    1SPDFFEJOHTPGUIFUI"$.*OUFSOBUJPOBM$POGFSFODFPO8FC4FBSDIBOE%BUB.JOJOH IUUQTEMBDNPSHEPJ
  2. ֓ཁ w 84%.࠾୒ w IUUQTXXXXTENDPOGFSFODFPSHBDDFQUFEQBQFSTQIQ w ୅ସࢦඪΛ࢖ͬͨ"#5FTUʹ͍ͭͯ w ِཅੑ͕େ͖͘ͳΔࣄΛࣔͨ͠ w

    ِཅੑΛ཈͑ΔͨΊͷൺֱ࣌ʹௐઅ͢Δख๏ΛఏҊ w ྑ͍୅ସࢦඪΛબͿ࣮ફతͳΨΠυϥΠϯΛఏҊ w -JOLFE*Oʹ͓͚ΔέʔεελσΟ
  3. എܠ • "#5FTU΍ΔΑͶ • ੑೳ͕ະ஌ͷࢪࡦ͸Ұ෦ͷϢʔβʔʹಋೖͯ͠ϏδωεࢦඪͳͲΛ֬ೝ͢Δ • ͍͔ʹͯ͠଎͘ਖ਼֬ͳ݁࿦ΛԼͤΔ͔ʹΑͬͯΠϊϕʔγϣϯ͸཯଎͞ΕΔ • -JOLFE*O͸4QFFE 2VBMJUZ

    3JTLͷόϥϯεΛͱ࣮ͬͨݧϑϨʔϜϫʔΫΛ ։ൃ͍ͯ͠Δ • ࢪࡦ͕-57 -JGF5JNF7BMVF ʹͲ͏Өڹ͢Δ͔ධՁ͍ͨ͠ • ͚ͲͦΕ͸͕͔͔࣌ؒΔ ޿ࠂͩͱ30"4ͱ͔ίϯόʔδϣϯ  • -57ͳͲͷ஗ߦࢦඪͷ୅ΘΓʹ࢖͏ͷ͕୅ସࢦඪ 4VSSPHBUF.FUSJDT  • ଥ౰ͳ୅ସࢦඪΛ࢖ͬͯҙࢥܾఆΛߴ଎Խ͍ͨ͠
  4. ิ଍୅ସࢦඪ 4VSSPHBUF.FUSJDT 1SPYZ.FUSJDTʜ ࢀߟ IUUQTOPUFDPNLFOKJLBUPPPOOFECED  Ҿ༻l/FUGMJYʹ͓͍ͯɺϓϩμΫτશମͷ඼࣭ΛධՁ͢ΔͨΊʹ࢖༻ͨ͠ࢦඪ͸ɺʮ ༗ྉձһͷ ܧଓ཰ʯ Ͱͨ͠ɻ͜ͷɺ͍ΘΏΔ/PSUI4UBS.FUSJD

    ࠷ॏཁࢦඪ ͸ɺ೥ؒͰେ͖͘վળ͞Ε·ͨ͠ɻ͸͡Ίͷࠒ ͸ɺ༗ྉձһͷ໿ˋ͕ຖ݄ղ໿͍ͯ͠·͕ͨ͠ɺ೥ʹ͸ɺ໿ˋ·Ͱվળ͠ɺݱࡏͰ͸ˋۙ͘ͷղ ໿཰ʹͳΓ·ͨ͠ɻ  ͨͩ͠ɺ͢΂ͯͷϓϩδΣΫτͷࢦඪͰ࠷ॏཁࢦඪͰ͋Δܧଓ཰Λ࢖༻͢Δ͜ͱ͸ݱ࣮తͰ͸͋Γ·ͤΜɻͳ ͥͳΒɺܧଓ཰ͱ͍͏ࢦඪࣗମɺ਺஋Λ͙͢ʹվળ͢Δ͜ͱ͕೉͘͠ɺվળΛূ໌͢Δʹ͸େن໛ͳ"#ςε τ͕ඞཁ͔ͩΒͰ͢ɻͦͷͨΊɺΑΓܭଌ͕͠΍͘͢ɺߴ଎ͰݕূͰ͖ΔΑ͏ͳɺ/PSUI4UBS.FUSJDͷ୅ ΘΓͱͳΔΑ͏ͳࢦඪ͕ඞཁʹͳΓ·͢ɻ͜Ε͕ϓϩΩγϝτϦΫεͰ͢ɻ QSPYZ͸୅ΘΓͷͷҙຯ l  lͦͯ͠ɺʮγϯϓϧͳମݧʯΛܭଌ͢ΔϓϩΩγϝτϦΫε͸ɺʮ࠷ॳͷηογϣϯதʹɺগͳ͘ͱ΋ͭͷ өըΛݟ͍ͨ΋ͷϦετʹ௥Ճͨ͠৽نϢʔβʔͷׂ߹ʯʹઃఆ͠·ͨ͠ɻz
  5. ߩݙ • ୅ସࢦඪِ͕ཅੑΛߴΊΔ࢓૊ΈΛ໌Β͔ʹͨ͠ • ͲͷΑ͏ͳ୅ସࢦඪΛ࢖͏΂͖͔ٞ࿦ͨ͠ • ࣮ફɾཧ࿦ͷ྆໘ʹجͮ͘୅ସࢦඪͷબఆΨΠυϥΠϯΛࣔͨ͠ • ୅ସࢦඪΛ࢖ͬͨߴ଎ͳҙࢥܾఆͷͨΊͷϫʔΫϑϩʔΛߏஙͨ͠ •

    -JOLFE*Oʹ͓͚ΔέʔεελσΟΛࣔͨ͠
  6. /PUBUJPOT ه߸ ҙຯ Y_i ϢʔβʔJͷ௕ظࢦඪ -57ͳͲ Y_i (0) ϢʔβʔJ͕USFBUNFOUΛड͚ͳ͔ͬͨͱ͖ͷPVUDPNF Y_i

    (1) ϢʔβʔJ͕USFBUNFOUΛड͚ͨͱ͖ͷPVUDPNF W_i ϢʔβʔJ͕࣮ݧ܈͔Ͳ͏͔\ ^ f(X) ୅ସࢦඪΛܭࢉ͢Δؔ਺G S_i ϢʔβʔJͷ୅ସࢦඪ TVSSPHBUFNFUSJDT µ_Y, µ_S ࢦඪʹର͢Δ"WFSBHF5SFBUNFOU&GGFDU
  7. 3FWJFXPO$POUSPMMFE&YQFSJNFOUT • ϧʔϏϯͷҼՌϞσϧ • જࡏΞ΢τΧϜϞσϧ • 4657" 4UBCMF6OJU5SFBUNFOU7BMVF"TTVNQUJPO  •

    ݸਓͷΞ΢τΧϜ͸ɺͦͷݸਓʹର͢Δ5SFBUNFOUʹΑͬͯͷΈܾ·Δ • ˠωοτϫʔΫޮՌ͕ແ͍Ծఆ • "WFSBHF5SFBUNFOU&GGFDU "5&
  8. ୅ସࢦඪʹΑΔ"5&

  9. ୅ସࢦඪબఆΨΠυϥΠϯ • )JHIQSFEJDUJWFQPXFSPOUIFUSVFOPSUI • ෆภੑɺਅͷϝτϦΫεͱͷ૬͕ؔڧ͍ DMPTFUP  • 'PDVTJOHPONFUSJDTXFDBODIBOHFBOENFBTVSFJOUIFTIPSUUFSN •

    ୹ظؒʹ؍ଌͰ͖Δ • $VTUPNJ[BUJPOGPSEJGGFSFOUUSFBUNFOUGFBUVSFT • ྫ͑͹ϞόΠϧͱ%FTLUPQͷ྆ํͰαʔϏεΛఏڙ͍ͯͨ͠ͱ͖ʹϞόΠϧʹ͍ͭͯ ͸ɺϞόΠϧ͚ͩͰऔಘͰ͖ΔϝτϦΫεΛ࢖ͬͨํ͕ޮՌͷ༧ଌ͕ྑ͘ͳΔɻϞόΠ ϧͱ%FTLUPQڞ௨ͷϝτϦΫεΛར༻͢ΔΑΓ΋ɻ • *OUFSQSFUBCJMJUZ • ෳࡶͳͷ͸΍ΊΑ͏ɻྫ͑͹ϢʔβʔߦಈͷTFRVFODFΛݩʹͨ͠ݕࡧຬ଍౓ͷਪఆɻ ػցֶशʹΑͬͯ༧ଌ͞Εͨ୅ସࢦඪ͸Ͳ͏ͯͦ͠ͷ஋্͕Լ͢Δͷ͔ཧղ͕೉͍͠ɻ • .BOBHFNFOUPWFSIFBE • l8FOFFEUPFEVDBUFPVSVTFSTʜzVTFSͬͯ୭ͩΖ  • ୅ସࢦඪ͕ࣗಈͰఏҊ͞ΕΔΑ͏ʹ͢Δ
  10. 4UBUJTUJDBM7BMJEJUZ3FRVJSFNFOU • ୅ସࢦඪͱ௕ظࢦඪͷؔ܎ʹ5SFBUNFOUΛड͚Δ͔Ͳ͏͔͕ಠཱͰ͋Δ͜ ͱ͕๬·͍͠ • ৚݅෇͖ಠཱͰॻ͘ͱ:⫫8c4 • ೋ܈ͷ&YDIBOHFBCJMJUZ͚ͩͰ͸ෆे෼

  11. ِཅੑ͕ੜ͡Δ࢓૊Έ ୅ସࢦඪͷ"5&͸ෆภੑ͸͋Δ͕ɺ෼ࢄΛաখධՁ͍ͯ͠Δ "5&ͷظ଴஋͸ಉ͡ͳͷ ͰVOCJBTFEFTUJNBUPS ୅ସࢦඪͱ௕ظࢦඪͷޡ ࠩʹ༝དྷ͢Δ෼ࢄ Var(µ_Y) > Var(µ_S) ͳͷͰ

    ୅ସࢦඪͷATEͷ෼ࢄ͸௕ظࢦ ඪͷATEͷ෼ࢄΑΓখ͍͞ ୅ସࢦඪ͸௕ظࢦඪΑΓૣ͘؍ ଌͰ͖Δ΋ͷͱߟ͑Δͱ௚ײతʹ Θ͔Δɻ޿ࠂͩͱCTRͷ෼ࢄΑ Γ΋ CTR×CVRͷ෼ࢄͷํ͕େ͖ ͍
  12. ِཅੑΛ཈͑Δʹ͸ • ୅ସࢦඪͷੜ੒Ϟσϧ͔Β෼ࢄΛಘͯ࢖͏ • ˠෆ͔֬͞Λར༻ • όοΫςετͯ͠୅ସࢦඪͱ௕ظࢦඪͷޡࠩΛఆྔԽ • ୅ସࢦඪΛͲ͏ิਖ਼ͨ͠Β௕ظࢦඪͱҰக͢Δ͔

  13. $BTF4UVEZ • ໨ඪ • ʣٻ৬ऀ͕ؔ࿈͢Δٻਓ৘ใΛΑΓΑ͘ൃݟͰ͖ΔΑ͏ʹ͢Δ • ʣٻਓ޿ࠂओʹద֨ͳٻ৬ऀͷԠืΛఏڙ͢Δ • ʣ֤ٻਓ޿ࠂʹे෼ͳ਺ͷٻ৬ऀͷԠื͕ࡴ౸͗͢͠ͳ͍Α͏ʹ͢Δ •

    -JOLFE*Oʹ͓͚Δ௕ظࢦඪ͸DPOGJSNFEIJSFT $)  • -JOLFE*Oͷॿ͚Λ͔Γͯ࢓ࣄΛΈ͚ͭͨϝϯόʔ • ͨͩ͠స৬׬ྃ·Ͱʹ͸͕͔͔࣌ؒΔ • ୅ସࢦඪͱͯ͠ͷQSFEJDUFEDPOGJSNFEIJSF 1$)  • ͜ΕΛ࡞ͬͨ
  14. -JOLFEJOʹ͓͚Δհೖ ਪન ͔Β௕ظࢦඪୡ੒·ͰͷྲྀΕ

  15. 1$) 1SFEJDUFE$POGJSNFE)JSF Linkedinʹ͓͚Δ స৬׬ྃ (௕ظࢦඪ) ͷ୅ସࢦඪ

  16. 1$)ͷධՁ૬ؔ ૬ؔͷ͋Δͳ͠͸όοΫςετ (ΦϑϥΠϯݕূ) ͰධՁՄೳ े෼ͳ૬͕ؔ͋Δࣄ͕֬ೝͰ͖ ͍ͯΔ

  17. 1$)ͷධՁҼՌͷํ޲ w ૬͕ؔ͋Δ͚ͩͰ͸ෆे෼ w 1$)ΛߴΊΔͱ$)΋ߴ͘ͳΔ͔Λ֬ೝ͍ͨ͠ ͷͰϓϩμΫγϣϯ؀ڥͰ࣮ݧΛͨ͠ ΦϯϥΠϯςετ  w ྫ

    w ମॏͱ਎௕ʹ૬͕ؔ͋ͬͨͱ͖ʹ w ମॏΛ૿΍͢ͱ਎௕͸৳ͼΔͷ͔
  18. 1$)ͷධՁTUBUJTUJDBMWBMJEJUZBTTVNQUJPO հೖͷ༗ແͰ$)ͱ1$)ͷؔ܎ʹ͕ࠩग़ͳ͍͜ͱͷ֬ೝ ৚݅෇͖ಠཱ  1$)ͷόέοτ ຖʹ ͷ෼෍Λϓϩοτ j λWk j

    = E(CH|PCHJOCVDLFUj, Wk) E(CH|PCHJOCVDLFUj)
  19. 1$)ͷධՁ෼ࢄͷਪఆ w 1$)Λ࢖ͬͨͷ࣮ݧͷ͏ͪ͸༗ҙਫ४Ͱ౷ܭతʹ༗ҙͩͬͨ w 1$)ͷU౷ܭݕఆྔͱ$)ͷU౷ܭݕఆྔΛϓϩοτ͢Δͱઢܗͷؔ܎ʹ͋Δ ͷ͕Θ͔ͬͨɺ ܾఆ܎਺ɻ͜ͷ͜ͱ͔Β1$)ʹΑΔૣظ൑அ͕ ༗ޮͩͱݴ͑Δ R2 PCHͷt౷ܭݕఆྔ

    ͸CHͷʹൺ΂ͯߴ ͘ͳΓ΍͍͢
  20. 1$)ͷධՁ෼ࢄͷਪఆ ଓ w ͨͩ͠1$)ͷ౷ܭݕఆྔ͸෼ࢄ͕աগධՁ͞Εͯ༗ҙʹͳΓ΍͍͢ͷͰ ิਖ਼ͨ͠෼ࢄ Λߟ͑Δ ࣜ VarAdj(μPCH) w ࣜͷ

    ΛࣜͰਪఆ͢Δɻ ͸1$)ͷ༧ଌޡࠩͷਪఆྔͱͳΔ w Ͱ΋ ͕େ͖͍ͷͰ༗ҙࠩͷग़Δ࣮ݧ͕͔Βʹݮͬͯ͠·ͬͨ ͋͞ࠔͬͨͧ σ2 ̂ σ2 ̂ σ2
  21. ڞมྔΛ༻͍ͨ෼ࢄݮগ๏ w ෼ࢄΛิਖ਼͚ͨ͠Ͳ࣮ݧ݁Ռͷ൑அ͕Ͱ͖ͳ͍ͷͰ͸୅ସࢦඪͷҙຯ͕ͳ͍ w TFOTJUJWJUZΛߴΊΔඞཁ͕͋Δ w Ұͭͷํ๏͸෼ࢄݮগ๏ WBSJBODFSFEVDUJPO  w

    ෼ࢄݮগ๏ͷதͰ΋ΦϯϥΠϯςετͰΑ͘࢖ΘΕΔͷ͕$61&% w $61&%ʹΑΓ෼ࢄ͕ʙݮগͰ͖ͨ w $61&%ద༻ޙʹ༗ҙͳ࣮ݧ͸͔Βʹ૿͑ͨ
  22. ·ͱΊ w ΦϯϥΠϯ࣮ݧʹ͓͚Δ୅ཧࢦඪͷΨΠυϥΠϯΛ·ͱΊͨ w ୅ཧࢦඪΛ࢖͏্Ͱ෼ࢄͷิਖ਼͕ඞཁʹͳΔࣄΛࣔͨ͠ w ୅ཧࢦඪͷTFOTJUJWJUZΛߴΊΔʹ͸෼ࢄݮগ๏͕༗ޮ w $)Λ࢖͏ʹ͸਺ϲ݄͔͔Δ͕1$)͸਺೔Ͱར༻ՄೳͳͷͰ਺ϲ݄ͷ͕࣌ؒઅ ໿Ͱ͖Δ

    w 1$)͕FOHJOFFSJOHUFBN QSPEVDUUFBNʹ޿͘ड͚͍ΕΒΕͨͷͰ-JOLFEJO ͷ6OJGJFE.FUSJD1MBUGPSNʹ౥ࡌͨ͠