Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

社内DS論文読み会の資料です

Duan, Weitao, Shan Ba, and Chunzhe Zhang. "Online Experimentation with Surrogate Metrics: Guidelines and a Case Study." Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021.

Takashi Nishibayashi

April 02, 2021
Tweet

More Decks by Takashi Nishibayashi

Other Decks in Technology

Transcript

  1. ࿦จ঺հ 0OMJOF&YQFSJNFOUBUJPOXJUI4VSSPHBUF.FUSJDT (VJEFMJOFTBOEB$BTF4UVEZ %4࿦จಡΈձ! ੢ྛ 5BLBTIJ/JTIJCBZBTIJ %VBO 8FJUBP 4IBO#B BOE$IVO[IF;IBOH0OMJOF&YQFSJNFOUBUJPOXJUI4VSSPHBUF.FUSJDT(VJEFMJOFTBOEB$BTF4UVEZ

    1SPDFFEJOHTPGUIFUI"$.*OUFSOBUJPOBM$POGFSFODFPO8FC4FBSDIBOE%BUB.JOJOH IUUQTEMBDNPSHEPJ
  2. ֓ཁ w 84%.࠾୒ w IUUQTXXXXTENDPOGFSFODFPSHBDDFQUFEQBQFSTQIQ w ୅ସࢦඪΛ࢖ͬͨ"#5FTUʹ͍ͭͯ w ِཅੑ͕େ͖͘ͳΔࣄΛࣔͨ͠ w

    ِཅੑΛ཈͑ΔͨΊͷൺֱ࣌ʹௐઅ͢Δख๏ΛఏҊ w ྑ͍୅ସࢦඪΛબͿ࣮ફతͳΨΠυϥΠϯΛఏҊ w -JOLFE*Oʹ͓͚ΔέʔεελσΟ
  3. എܠ • "#5FTU΍ΔΑͶ • ੑೳ͕ະ஌ͷࢪࡦ͸Ұ෦ͷϢʔβʔʹಋೖͯ͠ϏδωεࢦඪͳͲΛ֬ೝ͢Δ • ͍͔ʹͯ͠଎͘ਖ਼֬ͳ݁࿦ΛԼͤΔ͔ʹΑͬͯΠϊϕʔγϣϯ͸཯଎͞ΕΔ • -JOLFE*O͸4QFFE 2VBMJUZ

    3JTLͷόϥϯεΛͱ࣮ͬͨݧϑϨʔϜϫʔΫΛ ։ൃ͍ͯ͠Δ • ࢪࡦ͕-57 -JGF5JNF7BMVF ʹͲ͏Өڹ͢Δ͔ධՁ͍ͨ͠ • ͚ͲͦΕ͸͕͔͔࣌ؒΔ ޿ࠂͩͱ30"4ͱ͔ίϯόʔδϣϯ  • -57ͳͲͷ஗ߦࢦඪͷ୅ΘΓʹ࢖͏ͷ͕୅ସࢦඪ 4VSSPHBUF.FUSJDT  • ଥ౰ͳ୅ସࢦඪΛ࢖ͬͯҙࢥܾఆΛߴ଎Խ͍ͨ͠
  4. ิ଍୅ସࢦඪ 4VSSPHBUF.FUSJDT 1SPYZ.FUSJDTʜ ࢀߟ IUUQTOPUFDPNLFOKJLBUPPPOOFECED  Ҿ༻l/FUGMJYʹ͓͍ͯɺϓϩμΫτશମͷ඼࣭ΛධՁ͢ΔͨΊʹ࢖༻ͨ͠ࢦඪ͸ɺʮ ༗ྉձһͷ ܧଓ཰ʯ Ͱͨ͠ɻ͜ͷɺ͍ΘΏΔ/PSUI4UBS.FUSJD

    ࠷ॏཁࢦඪ ͸ɺ೥ؒͰେ͖͘վળ͞Ε·ͨ͠ɻ͸͡Ίͷࠒ ͸ɺ༗ྉձһͷ໿ˋ͕ຖ݄ղ໿͍ͯ͠·͕ͨ͠ɺ೥ʹ͸ɺ໿ˋ·Ͱվળ͠ɺݱࡏͰ͸ˋۙ͘ͷղ ໿཰ʹͳΓ·ͨ͠ɻ  ͨͩ͠ɺ͢΂ͯͷϓϩδΣΫτͷࢦඪͰ࠷ॏཁࢦඪͰ͋Δܧଓ཰Λ࢖༻͢Δ͜ͱ͸ݱ࣮తͰ͸͋Γ·ͤΜɻͳ ͥͳΒɺܧଓ཰ͱ͍͏ࢦඪࣗମɺ਺஋Λ͙͢ʹվળ͢Δ͜ͱ͕೉͘͠ɺվળΛূ໌͢Δʹ͸େن໛ͳ"#ςε τ͕ඞཁ͔ͩΒͰ͢ɻͦͷͨΊɺΑΓܭଌ͕͠΍͘͢ɺߴ଎ͰݕূͰ͖ΔΑ͏ͳɺ/PSUI4UBS.FUSJDͷ୅ ΘΓͱͳΔΑ͏ͳࢦඪ͕ඞཁʹͳΓ·͢ɻ͜Ε͕ϓϩΩγϝτϦΫεͰ͢ɻ QSPYZ͸୅ΘΓͷͷҙຯ l  lͦͯ͠ɺʮγϯϓϧͳମݧʯΛܭଌ͢ΔϓϩΩγϝτϦΫε͸ɺʮ࠷ॳͷηογϣϯதʹɺগͳ͘ͱ΋ͭͷ өըΛݟ͍ͨ΋ͷϦετʹ௥Ճͨ͠৽نϢʔβʔͷׂ߹ʯʹઃఆ͠·ͨ͠ɻz
  5. ߩݙ • ୅ସࢦඪِ͕ཅੑΛߴΊΔ࢓૊ΈΛ໌Β͔ʹͨ͠ • ͲͷΑ͏ͳ୅ସࢦඪΛ࢖͏΂͖͔ٞ࿦ͨ͠ • ࣮ફɾཧ࿦ͷ྆໘ʹجͮ͘୅ସࢦඪͷબఆΨΠυϥΠϯΛࣔͨ͠ • ୅ସࢦඪΛ࢖ͬͨߴ଎ͳҙࢥܾఆͷͨΊͷϫʔΫϑϩʔΛߏஙͨ͠ •

    -JOLFE*Oʹ͓͚ΔέʔεελσΟΛࣔͨ͠
  6. /PUBUJPOT ه߸ ҙຯ Y_i ϢʔβʔJͷ௕ظࢦඪ -57ͳͲ Y_i (0) ϢʔβʔJ͕USFBUNFOUΛड͚ͳ͔ͬͨͱ͖ͷPVUDPNF Y_i

    (1) ϢʔβʔJ͕USFBUNFOUΛड͚ͨͱ͖ͷPVUDPNF W_i ϢʔβʔJ͕࣮ݧ܈͔Ͳ͏͔\ ^ f(X) ୅ସࢦඪΛܭࢉ͢Δؔ਺G S_i ϢʔβʔJͷ୅ସࢦඪ TVSSPHBUFNFUSJDT µ_Y, µ_S ࢦඪʹର͢Δ"WFSBHF5SFBUNFOU&GGFDU
  7. 3FWJFXPO$POUSPMMFE&YQFSJNFOUT • ϧʔϏϯͷҼՌϞσϧ • જࡏΞ΢τΧϜϞσϧ • 4657" 4UBCMF6OJU5SFBUNFOU7BMVF"TTVNQUJPO  •

    ݸਓͷΞ΢τΧϜ͸ɺͦͷݸਓʹର͢Δ5SFBUNFOUʹΑͬͯͷΈܾ·Δ • ˠωοτϫʔΫޮՌ͕ແ͍Ծఆ • "WFSBHF5SFBUNFOU&GGFDU "5&
  8. ୅ସࢦඪʹΑΔ"5&

  9. ୅ସࢦඪબఆΨΠυϥΠϯ • )JHIQSFEJDUJWFQPXFSPOUIFUSVFOPSUI • ෆภੑɺਅͷϝτϦΫεͱͷ૬͕ؔڧ͍ DMPTFUP  • 'PDVTJOHPONFUSJDTXFDBODIBOHFBOENFBTVSFJOUIFTIPSUUFSN •

    ୹ظؒʹ؍ଌͰ͖Δ • $VTUPNJ[BUJPOGPSEJGGFSFOUUSFBUNFOUGFBUVSFT • ྫ͑͹ϞόΠϧͱ%FTLUPQͷ྆ํͰαʔϏεΛఏڙ͍ͯͨ͠ͱ͖ʹϞόΠϧʹ͍ͭͯ ͸ɺϞόΠϧ͚ͩͰऔಘͰ͖ΔϝτϦΫεΛ࢖ͬͨํ͕ޮՌͷ༧ଌ͕ྑ͘ͳΔɻϞόΠ ϧͱ%FTLUPQڞ௨ͷϝτϦΫεΛར༻͢ΔΑΓ΋ɻ • *OUFSQSFUBCJMJUZ • ෳࡶͳͷ͸΍ΊΑ͏ɻྫ͑͹ϢʔβʔߦಈͷTFRVFODFΛݩʹͨ͠ݕࡧຬ଍౓ͷਪఆɻ ػցֶशʹΑͬͯ༧ଌ͞Εͨ୅ସࢦඪ͸Ͳ͏ͯͦ͠ͷ஋্͕Լ͢Δͷ͔ཧղ͕೉͍͠ɻ • .BOBHFNFOUPWFSIFBE • l8FOFFEUPFEVDBUFPVSVTFSTʜzVTFSͬͯ୭ͩΖ  • ୅ସࢦඪ͕ࣗಈͰఏҊ͞ΕΔΑ͏ʹ͢Δ
  10. 4UBUJTUJDBM7BMJEJUZ3FRVJSFNFOU • ୅ସࢦඪͱ௕ظࢦඪͷؔ܎ʹ5SFBUNFOUΛड͚Δ͔Ͳ͏͔͕ಠཱͰ͋Δ͜ ͱ͕๬·͍͠ • ৚݅෇͖ಠཱͰॻ͘ͱ:⫫8c4 • ೋ܈ͷ&YDIBOHFBCJMJUZ͚ͩͰ͸ෆे෼

  11. ِཅੑ͕ੜ͡Δ࢓૊Έ ୅ସࢦඪͷ"5&͸ෆภੑ͸͋Δ͕ɺ෼ࢄΛաখධՁ͍ͯ͠Δ "5&ͷظ଴஋͸ಉ͡ͳͷ ͰVOCJBTFEFTUJNBUPS ୅ସࢦඪͱ௕ظࢦඪͷޡ ࠩʹ༝དྷ͢Δ෼ࢄ Var(µ_Y) > Var(µ_S) ͳͷͰ

    ୅ସࢦඪͷATEͷ෼ࢄ͸௕ظࢦ ඪͷATEͷ෼ࢄΑΓখ͍͞ ୅ସࢦඪ͸௕ظࢦඪΑΓૣ͘؍ ଌͰ͖Δ΋ͷͱߟ͑Δͱ௚ײతʹ Θ͔Δɻ޿ࠂͩͱCTRͷ෼ࢄΑ Γ΋ CTR×CVRͷ෼ࢄͷํ͕େ͖ ͍
  12. ِཅੑΛ཈͑Δʹ͸ • ୅ସࢦඪͷੜ੒Ϟσϧ͔Β෼ࢄΛಘͯ࢖͏ • ˠෆ͔֬͞Λར༻ • όοΫςετͯ͠୅ସࢦඪͱ௕ظࢦඪͷޡࠩΛఆྔԽ • ୅ସࢦඪΛͲ͏ิਖ਼ͨ͠Β௕ظࢦඪͱҰக͢Δ͔

  13. $BTF4UVEZ • ໨ඪ • ʣٻ৬ऀ͕ؔ࿈͢Δٻਓ৘ใΛΑΓΑ͘ൃݟͰ͖ΔΑ͏ʹ͢Δ • ʣٻਓ޿ࠂओʹద֨ͳٻ৬ऀͷԠืΛఏڙ͢Δ • ʣ֤ٻਓ޿ࠂʹे෼ͳ਺ͷٻ৬ऀͷԠื͕ࡴ౸͗͢͠ͳ͍Α͏ʹ͢Δ •

    -JOLFE*Oʹ͓͚Δ௕ظࢦඪ͸DPOGJSNFEIJSFT $)  • -JOLFE*Oͷॿ͚Λ͔Γͯ࢓ࣄΛΈ͚ͭͨϝϯόʔ • ͨͩ͠స৬׬ྃ·Ͱʹ͸͕͔͔࣌ؒΔ • ୅ସࢦඪͱͯ͠ͷQSFEJDUFEDPOGJSNFEIJSF 1$)  • ͜ΕΛ࡞ͬͨ
  14. -JOLFEJOʹ͓͚Δհೖ ਪન ͔Β௕ظࢦඪୡ੒·ͰͷྲྀΕ

  15. 1$) 1SFEJDUFE$POGJSNFE)JSF Linkedinʹ͓͚Δ స৬׬ྃ (௕ظࢦඪ) ͷ୅ସࢦඪ

  16. 1$)ͷධՁ૬ؔ ૬ؔͷ͋Δͳ͠͸όοΫςετ (ΦϑϥΠϯݕূ) ͰධՁՄೳ े෼ͳ૬͕ؔ͋Δࣄ͕֬ೝͰ͖ ͍ͯΔ

  17. 1$)ͷධՁҼՌͷํ޲ w ૬͕ؔ͋Δ͚ͩͰ͸ෆे෼ w 1$)ΛߴΊΔͱ$)΋ߴ͘ͳΔ͔Λ֬ೝ͍ͨ͠ ͷͰϓϩμΫγϣϯ؀ڥͰ࣮ݧΛͨ͠ ΦϯϥΠϯςετ  w ྫ

    w ମॏͱ਎௕ʹ૬͕ؔ͋ͬͨͱ͖ʹ w ମॏΛ૿΍͢ͱ਎௕͸৳ͼΔͷ͔
  18. 1$)ͷධՁTUBUJTUJDBMWBMJEJUZBTTVNQUJPO հೖͷ༗ແͰ$)ͱ1$)ͷؔ܎ʹ͕ࠩग़ͳ͍͜ͱͷ֬ೝ ৚݅෇͖ಠཱ  1$)ͷόέοτ ຖʹ ͷ෼෍Λϓϩοτ j λWk j

    = E(CH|PCHJOCVDLFUj, Wk) E(CH|PCHJOCVDLFUj)
  19. 1$)ͷධՁ෼ࢄͷਪఆ w 1$)Λ࢖ͬͨͷ࣮ݧͷ͏ͪ͸༗ҙਫ४Ͱ౷ܭతʹ༗ҙͩͬͨ w 1$)ͷU౷ܭݕఆྔͱ$)ͷU౷ܭݕఆྔΛϓϩοτ͢Δͱઢܗͷؔ܎ʹ͋Δ ͷ͕Θ͔ͬͨɺ ܾఆ܎਺ɻ͜ͷ͜ͱ͔Β1$)ʹΑΔૣظ൑அ͕ ༗ޮͩͱݴ͑Δ R2 PCHͷt౷ܭݕఆྔ

    ͸CHͷʹൺ΂ͯߴ ͘ͳΓ΍͍͢
  20. 1$)ͷධՁ෼ࢄͷਪఆ ଓ w ͨͩ͠1$)ͷ౷ܭݕఆྔ͸෼ࢄ͕աগධՁ͞Εͯ༗ҙʹͳΓ΍͍͢ͷͰ ิਖ਼ͨ͠෼ࢄ Λߟ͑Δ ࣜ VarAdj(μPCH) w ࣜͷ

    ΛࣜͰਪఆ͢Δɻ ͸1$)ͷ༧ଌޡࠩͷਪఆྔͱͳΔ w Ͱ΋ ͕େ͖͍ͷͰ༗ҙࠩͷग़Δ࣮ݧ͕͔Βʹݮͬͯ͠·ͬͨ ͋͞ࠔͬͨͧ σ2 ̂ σ2 ̂ σ2
  21. ڞมྔΛ༻͍ͨ෼ࢄݮগ๏ w ෼ࢄΛิਖ਼͚ͨ͠Ͳ࣮ݧ݁Ռͷ൑அ͕Ͱ͖ͳ͍ͷͰ͸୅ସࢦඪͷҙຯ͕ͳ͍ w TFOTJUJWJUZΛߴΊΔඞཁ͕͋Δ w Ұͭͷํ๏͸෼ࢄݮগ๏ WBSJBODFSFEVDUJPO  w

    ෼ࢄݮগ๏ͷதͰ΋ΦϯϥΠϯςετͰΑ͘࢖ΘΕΔͷ͕$61&% w $61&%ʹΑΓ෼ࢄ͕ʙݮগͰ͖ͨ w $61&%ద༻ޙʹ༗ҙͳ࣮ݧ͸͔Βʹ૿͑ͨ
  22. ·ͱΊ w ΦϯϥΠϯ࣮ݧʹ͓͚Δ୅ཧࢦඪͷΨΠυϥΠϯΛ·ͱΊͨ w ୅ཧࢦඪΛ࢖͏্Ͱ෼ࢄͷิਖ਼͕ඞཁʹͳΔࣄΛࣔͨ͠ w ୅ཧࢦඪͷTFOTJUJWJUZΛߴΊΔʹ͸෼ࢄݮগ๏͕༗ޮ w $)Λ࢖͏ʹ͸਺ϲ݄͔͔Δ͕1$)͸਺೔Ͱར༻ՄೳͳͷͰ਺ϲ݄ͷ͕࣌ؒઅ ໿Ͱ͖Δ

    w 1$)͕FOHJOFFSJOHUFBN QSPEVDUUFBNʹ޿͘ड͚͍ΕΒΕͨͷͰ-JOLFEJO ͷ6OJGJFE.FUSJD1MBUGPSNʹ౥ࡌͨ͠