Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

社内DS論文読み会の資料です

Duan, Weitao, Shan Ba, and Chunzhe Zhang. "Online Experimentation with Surrogate Metrics: Guidelines and a Case Study." Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021.

Takashi Nishibayashi

April 02, 2021
Tweet

More Decks by Takashi Nishibayashi

Other Decks in Technology

Transcript

  1. ֓ཁ w 84%.࠾୒ w IUUQTXXXXTENDPOGFSFODFPSHBDDFQUFEQBQFSTQIQ w ୅ସࢦඪΛ࢖ͬͨ"#5FTUʹ͍ͭͯ w ِཅੑ͕େ͖͘ͳΔࣄΛࣔͨ͠ w

    ِཅੑΛ཈͑ΔͨΊͷൺֱ࣌ʹௐઅ͢Δख๏ΛఏҊ w ྑ͍୅ସࢦඪΛબͿ࣮ફతͳΨΠυϥΠϯΛఏҊ w -JOLFE*Oʹ͓͚ΔέʔεελσΟ
  2. എܠ • "#5FTU΍ΔΑͶ • ੑೳ͕ະ஌ͷࢪࡦ͸Ұ෦ͷϢʔβʔʹಋೖͯ͠ϏδωεࢦඪͳͲΛ֬ೝ͢Δ • ͍͔ʹͯ͠଎͘ਖ਼֬ͳ݁࿦ΛԼͤΔ͔ʹΑͬͯΠϊϕʔγϣϯ͸཯଎͞ΕΔ • -JOLFE*O͸4QFFE 2VBMJUZ

    3JTLͷόϥϯεΛͱ࣮ͬͨݧϑϨʔϜϫʔΫΛ ։ൃ͍ͯ͠Δ • ࢪࡦ͕-57 -JGF5JNF7BMVF ʹͲ͏Өڹ͢Δ͔ධՁ͍ͨ͠ • ͚ͲͦΕ͸͕͔͔࣌ؒΔ ޿ࠂͩͱ30"4ͱ͔ίϯόʔδϣϯ  • -57ͳͲͷ஗ߦࢦඪͷ୅ΘΓʹ࢖͏ͷ͕୅ସࢦඪ 4VSSPHBUF.FUSJDT  • ଥ౰ͳ୅ସࢦඪΛ࢖ͬͯҙࢥܾఆΛߴ଎Խ͍ͨ͠
  3. ิ଍୅ସࢦඪ 4VSSPHBUF.FUSJDT 1SPYZ.FUSJDTʜ ࢀߟ IUUQTOPUFDPNLFOKJLBUPPPOOFECED  Ҿ༻l/FUGMJYʹ͓͍ͯɺϓϩμΫτશମͷ඼࣭ΛධՁ͢ΔͨΊʹ࢖༻ͨ͠ࢦඪ͸ɺʮ ༗ྉձһͷ ܧଓ཰ʯ Ͱͨ͠ɻ͜ͷɺ͍ΘΏΔ/PSUI4UBS.FUSJD

    ࠷ॏཁࢦඪ ͸ɺ೥ؒͰେ͖͘վળ͞Ε·ͨ͠ɻ͸͡Ίͷࠒ ͸ɺ༗ྉձһͷ໿ˋ͕ຖ݄ղ໿͍ͯ͠·͕ͨ͠ɺ೥ʹ͸ɺ໿ˋ·Ͱվળ͠ɺݱࡏͰ͸ˋۙ͘ͷղ ໿཰ʹͳΓ·ͨ͠ɻ  ͨͩ͠ɺ͢΂ͯͷϓϩδΣΫτͷࢦඪͰ࠷ॏཁࢦඪͰ͋Δܧଓ཰Λ࢖༻͢Δ͜ͱ͸ݱ࣮తͰ͸͋Γ·ͤΜɻͳ ͥͳΒɺܧଓ཰ͱ͍͏ࢦඪࣗମɺ਺஋Λ͙͢ʹվળ͢Δ͜ͱ͕೉͘͠ɺվળΛূ໌͢Δʹ͸େن໛ͳ"#ςε τ͕ඞཁ͔ͩΒͰ͢ɻͦͷͨΊɺΑΓܭଌ͕͠΍͘͢ɺߴ଎ͰݕূͰ͖ΔΑ͏ͳɺ/PSUI4UBS.FUSJDͷ୅ ΘΓͱͳΔΑ͏ͳࢦඪ͕ඞཁʹͳΓ·͢ɻ͜Ε͕ϓϩΩγϝτϦΫεͰ͢ɻ QSPYZ͸୅ΘΓͷͷҙຯ l  lͦͯ͠ɺʮγϯϓϧͳମݧʯΛܭଌ͢ΔϓϩΩγϝτϦΫε͸ɺʮ࠷ॳͷηογϣϯதʹɺগͳ͘ͱ΋ͭͷ өըΛݟ͍ͨ΋ͷϦετʹ௥Ճͨ͠৽نϢʔβʔͷׂ߹ʯʹઃఆ͠·ͨ͠ɻz
  4. /PUBUJPOT ه߸ ҙຯ Y_i ϢʔβʔJͷ௕ظࢦඪ -57ͳͲ Y_i (0) ϢʔβʔJ͕USFBUNFOUΛड͚ͳ͔ͬͨͱ͖ͷPVUDPNF Y_i

    (1) ϢʔβʔJ͕USFBUNFOUΛड͚ͨͱ͖ͷPVUDPNF W_i ϢʔβʔJ͕࣮ݧ܈͔Ͳ͏͔\ ^ f(X) ୅ସࢦඪΛܭࢉ͢Δؔ਺G S_i ϢʔβʔJͷ୅ସࢦඪ TVSSPHBUFNFUSJDT µ_Y, µ_S ࢦඪʹର͢Δ"WFSBHF5SFBUNFOU&GGFDU
  5. 3FWJFXPO$POUSPMMFE&YQFSJNFOUT • ϧʔϏϯͷҼՌϞσϧ • જࡏΞ΢τΧϜϞσϧ • 4657" 4UBCMF6OJU5SFBUNFOU7BMVF"TTVNQUJPO  •

    ݸਓͷΞ΢τΧϜ͸ɺͦͷݸਓʹର͢Δ5SFBUNFOUʹΑͬͯͷΈܾ·Δ • ˠωοτϫʔΫޮՌ͕ແ͍Ծఆ • "WFSBHF5SFBUNFOU&GGFDU "5&
  6. ୅ସࢦඪબఆΨΠυϥΠϯ • )JHIQSFEJDUJWFQPXFSPOUIFUSVFOPSUI • ෆภੑɺਅͷϝτϦΫεͱͷ૬͕ؔڧ͍ DMPTFUP  • 'PDVTJOHPONFUSJDTXFDBODIBOHFBOENFBTVSFJOUIFTIPSUUFSN •

    ୹ظؒʹ؍ଌͰ͖Δ • $VTUPNJ[BUJPOGPSEJGGFSFOUUSFBUNFOUGFBUVSFT • ྫ͑͹ϞόΠϧͱ%FTLUPQͷ྆ํͰαʔϏεΛఏڙ͍ͯͨ͠ͱ͖ʹϞόΠϧʹ͍ͭͯ ͸ɺϞόΠϧ͚ͩͰऔಘͰ͖ΔϝτϦΫεΛ࢖ͬͨํ͕ޮՌͷ༧ଌ͕ྑ͘ͳΔɻϞόΠ ϧͱ%FTLUPQڞ௨ͷϝτϦΫεΛར༻͢ΔΑΓ΋ɻ • *OUFSQSFUBCJMJUZ • ෳࡶͳͷ͸΍ΊΑ͏ɻྫ͑͹ϢʔβʔߦಈͷTFRVFODFΛݩʹͨ͠ݕࡧຬ଍౓ͷਪఆɻ ػցֶशʹΑͬͯ༧ଌ͞Εͨ୅ସࢦඪ͸Ͳ͏ͯͦ͠ͷ஋্͕Լ͢Δͷ͔ཧղ͕೉͍͠ɻ • .BOBHFNFOUPWFSIFBE • l8FOFFEUPFEVDBUFPVSVTFSTʜzVTFSͬͯ୭ͩΖ  • ୅ସࢦඪ͕ࣗಈͰఏҊ͞ΕΔΑ͏ʹ͢Δ
  7. ِཅੑ͕ੜ͡Δ࢓૊Έ ୅ସࢦඪͷ"5&͸ෆภੑ͸͋Δ͕ɺ෼ࢄΛաখධՁ͍ͯ͠Δ "5&ͷظ଴஋͸ಉ͡ͳͷ ͰVOCJBTFEFTUJNBUPS ୅ସࢦඪͱ௕ظࢦඪͷޡ ࠩʹ༝དྷ͢Δ෼ࢄ Var(µ_Y) > Var(µ_S) ͳͷͰ

    ୅ସࢦඪͷATEͷ෼ࢄ͸௕ظࢦ ඪͷATEͷ෼ࢄΑΓখ͍͞ ୅ସࢦඪ͸௕ظࢦඪΑΓૣ͘؍ ଌͰ͖Δ΋ͷͱߟ͑Δͱ௚ײతʹ Θ͔Δɻ޿ࠂͩͱCTRͷ෼ࢄΑ Γ΋ CTR×CVRͷ෼ࢄͷํ͕େ͖ ͍
  8. $BTF4UVEZ • ໨ඪ • ʣٻ৬ऀ͕ؔ࿈͢Δٻਓ৘ใΛΑΓΑ͘ൃݟͰ͖ΔΑ͏ʹ͢Δ • ʣٻਓ޿ࠂओʹద֨ͳٻ৬ऀͷԠืΛఏڙ͢Δ • ʣ֤ٻਓ޿ࠂʹे෼ͳ਺ͷٻ৬ऀͷԠื͕ࡴ౸͗͢͠ͳ͍Α͏ʹ͢Δ •

    -JOLFE*Oʹ͓͚Δ௕ظࢦඪ͸DPOGJSNFEIJSFT $)  • -JOLFE*Oͷॿ͚Λ͔Γͯ࢓ࣄΛΈ͚ͭͨϝϯόʔ • ͨͩ͠స৬׬ྃ·Ͱʹ͸͕͔͔࣌ؒΔ • ୅ସࢦඪͱͯ͠ͷQSFEJDUFEDPOGJSNFEIJSF 1$)  • ͜ΕΛ࡞ͬͨ
  9. 1$)ͷධՁ෼ࢄͷਪఆ ଓ w ͨͩ͠1$)ͷ౷ܭݕఆྔ͸෼ࢄ͕աগධՁ͞Εͯ༗ҙʹͳΓ΍͍͢ͷͰ ิਖ਼ͨ͠෼ࢄ Λߟ͑Δ ࣜ VarAdj(μPCH) w ࣜͷ

    ΛࣜͰਪఆ͢Δɻ ͸1$)ͷ༧ଌޡࠩͷਪఆྔͱͳΔ w Ͱ΋ ͕େ͖͍ͷͰ༗ҙࠩͷग़Δ࣮ݧ͕͔Βʹݮͬͯ͠·ͬͨ ͋͞ࠔͬͨͧ σ2 ̂ σ2 ̂ σ2
  10. ڞมྔΛ༻͍ͨ෼ࢄݮগ๏ w ෼ࢄΛิਖ਼͚ͨ͠Ͳ࣮ݧ݁Ռͷ൑அ͕Ͱ͖ͳ͍ͷͰ͸୅ସࢦඪͷҙຯ͕ͳ͍ w TFOTJUJWJUZΛߴΊΔඞཁ͕͋Δ w Ұͭͷํ๏͸෼ࢄݮগ๏ WBSJBODFSFEVDUJPO  w

    ෼ࢄݮগ๏ͷதͰ΋ΦϯϥΠϯςετͰΑ͘࢖ΘΕΔͷ͕$61&% w $61&%ʹΑΓ෼ࢄ͕ʙݮগͰ͖ͨ w $61&%ద༻ޙʹ༗ҙͳ࣮ݧ͸͔Βʹ૿͑ͨ