Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介 Online Experimentation with Surrogate Metr...

論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study

社内DS論文読み会の資料です

Duan, Weitao, Shan Ba, and Chunzhe Zhang. "Online Experimentation with Surrogate Metrics: Guidelines and a Case Study." Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021.

Takashi Nishibayashi

April 02, 2021

Transcript

22 slides
  1. ֓ཁ w 84%.࠾୒ w IUUQTXXXXTENDPOGFSFODFPSHBDDFQUFEQBQFSTQIQ w ୅ସࢦඪΛ࢖ͬͨ"#5FTUʹ͍ͭͯ w ِཅੑ͕େ͖͘ͳΔࣄΛࣔͨ͠ w

    ِཅੑΛ཈͑ΔͨΊͷൺֱ࣌ʹௐઅ͢Δख๏ΛఏҊ w ྑ͍୅ସࢦඪΛબͿ࣮ફతͳΨΠυϥΠϯΛఏҊ w -JOLFE*Oʹ͓͚ΔέʔεελσΟ
  2. എܠ • "#5FTU΍ΔΑͶ • ੑೳ͕ະ஌ͷࢪࡦ͸Ұ෦ͷϢʔβʔʹಋೖͯ͠ϏδωεࢦඪͳͲΛ֬ೝ͢Δ • ͍͔ʹͯ͠଎͘ਖ਼֬ͳ݁࿦ΛԼͤΔ͔ʹΑͬͯΠϊϕʔγϣϯ͸཯଎͞ΕΔ • -JOLFE*O͸4QFFE 2VBMJUZ

    3JTLͷόϥϯεΛͱ࣮ͬͨݧϑϨʔϜϫʔΫΛ ։ൃ͍ͯ͠Δ • ࢪࡦ͕-57 -JGF5JNF7BMVF ʹͲ͏Өڹ͢Δ͔ධՁ͍ͨ͠ • ͚ͲͦΕ͸͕͔͔࣌ؒΔ ޿ࠂͩͱ30"4ͱ͔ίϯόʔδϣϯ  • -57ͳͲͷ஗ߦࢦඪͷ୅ΘΓʹ࢖͏ͷ͕୅ସࢦඪ 4VSSPHBUF.FUSJDT  • ଥ౰ͳ୅ସࢦඪΛ࢖ͬͯҙࢥܾఆΛߴ଎Խ͍ͨ͠
  3. ิ଍୅ସࢦඪ 4VSSPHBUF.FUSJDT 1SPYZ.FUSJDTʜ ࢀߟ IUUQTOPUFDPNLFOKJLBUPPPOOFECED  Ҿ༻l/FUGMJYʹ͓͍ͯɺϓϩμΫτશମͷ඼࣭ΛධՁ͢ΔͨΊʹ࢖༻ͨ͠ࢦඪ͸ɺʮ ༗ྉձһͷ ܧଓ཰ʯ Ͱͨ͠ɻ͜ͷɺ͍ΘΏΔ/PSUI4UBS.FUSJD

    ࠷ॏཁࢦඪ ͸ɺ೥ؒͰେ͖͘վળ͞Ε·ͨ͠ɻ͸͡Ίͷࠒ ͸ɺ༗ྉձһͷ໿ˋ͕ຖ݄ղ໿͍ͯ͠·͕ͨ͠ɺ೥ʹ͸ɺ໿ˋ·Ͱվળ͠ɺݱࡏͰ͸ˋۙ͘ͷղ ໿཰ʹͳΓ·ͨ͠ɻ  ͨͩ͠ɺ͢΂ͯͷϓϩδΣΫτͷࢦඪͰ࠷ॏཁࢦඪͰ͋Δܧଓ཰Λ࢖༻͢Δ͜ͱ͸ݱ࣮తͰ͸͋Γ·ͤΜɻͳ ͥͳΒɺܧଓ཰ͱ͍͏ࢦඪࣗମɺ਺஋Λ͙͢ʹվળ͢Δ͜ͱ͕೉͘͠ɺվળΛূ໌͢Δʹ͸େن໛ͳ"#ςε τ͕ඞཁ͔ͩΒͰ͢ɻͦͷͨΊɺΑΓܭଌ͕͠΍͘͢ɺߴ଎ͰݕূͰ͖ΔΑ͏ͳɺ/PSUI4UBS.FUSJDͷ୅ ΘΓͱͳΔΑ͏ͳࢦඪ͕ඞཁʹͳΓ·͢ɻ͜Ε͕ϓϩΩγϝτϦΫεͰ͢ɻ QSPYZ͸୅ΘΓͷͷҙຯ l  lͦͯ͠ɺʮγϯϓϧͳମݧʯΛܭଌ͢ΔϓϩΩγϝτϦΫε͸ɺʮ࠷ॳͷηογϣϯதʹɺগͳ͘ͱ΋ͭͷ өըΛݟ͍ͨ΋ͷϦετʹ௥Ճͨ͠৽نϢʔβʔͷׂ߹ʯʹઃఆ͠·ͨ͠ɻz
  4. /PUBUJPOT ه߸ ҙຯ Y_i ϢʔβʔJͷ௕ظࢦඪ -57ͳͲ Y_i (0) ϢʔβʔJ͕USFBUNFOUΛड͚ͳ͔ͬͨͱ͖ͷPVUDPNF Y_i

    (1) ϢʔβʔJ͕USFBUNFOUΛड͚ͨͱ͖ͷPVUDPNF W_i ϢʔβʔJ͕࣮ݧ܈͔Ͳ͏͔\ ^ f(X) ୅ସࢦඪΛܭࢉ͢Δؔ਺G S_i ϢʔβʔJͷ୅ସࢦඪ TVSSPHBUFNFUSJDT µ_Y, µ_S ࢦඪʹର͢Δ"WFSBHF5SFBUNFOU&GGFDU
  5. 3FWJFXPO$POUSPMMFE&YQFSJNFOUT • ϧʔϏϯͷҼՌϞσϧ • જࡏΞ΢τΧϜϞσϧ • 4657" 4UBCMF6OJU5SFBUNFOU7BMVF"TTVNQUJPO  •

    ݸਓͷΞ΢τΧϜ͸ɺͦͷݸਓʹର͢Δ5SFBUNFOUʹΑͬͯͷΈܾ·Δ • ˠωοτϫʔΫޮՌ͕ແ͍Ծఆ • "WFSBHF5SFBUNFOU&GGFDU "5&
  6. ୅ସࢦඪબఆΨΠυϥΠϯ • )JHIQSFEJDUJWFQPXFSPOUIFUSVFOPSUI • ෆภੑɺਅͷϝτϦΫεͱͷ૬͕ؔڧ͍ DMPTFUP  • 'PDVTJOHPONFUSJDTXFDBODIBOHFBOENFBTVSFJOUIFTIPSUUFSN •

    ୹ظؒʹ؍ଌͰ͖Δ • $VTUPNJ[BUJPOGPSEJGGFSFOUUSFBUNFOUGFBUVSFT • ྫ͑͹ϞόΠϧͱ%FTLUPQͷ྆ํͰαʔϏεΛఏڙ͍ͯͨ͠ͱ͖ʹϞόΠϧʹ͍ͭͯ ͸ɺϞόΠϧ͚ͩͰऔಘͰ͖ΔϝτϦΫεΛ࢖ͬͨํ͕ޮՌͷ༧ଌ͕ྑ͘ͳΔɻϞόΠ ϧͱ%FTLUPQڞ௨ͷϝτϦΫεΛར༻͢ΔΑΓ΋ɻ • *OUFSQSFUBCJMJUZ • ෳࡶͳͷ͸΍ΊΑ͏ɻྫ͑͹ϢʔβʔߦಈͷTFRVFODFΛݩʹͨ͠ݕࡧຬ଍౓ͷਪఆɻ ػցֶशʹΑͬͯ༧ଌ͞Εͨ୅ସࢦඪ͸Ͳ͏ͯͦ͠ͷ஋্͕Լ͢Δͷ͔ཧղ͕೉͍͠ɻ • .BOBHFNFOUPWFSIFBE • l8FOFFEUPFEVDBUFPVSVTFSTʜzVTFSͬͯ୭ͩΖ  • ୅ସࢦඪ͕ࣗಈͰఏҊ͞ΕΔΑ͏ʹ͢Δ
  7. ِཅੑ͕ੜ͡Δ࢓૊Έ ୅ସࢦඪͷ"5&͸ෆภੑ͸͋Δ͕ɺ෼ࢄΛաখධՁ͍ͯ͠Δ "5&ͷظ଴஋͸ಉ͡ͳͷ ͰVOCJBTFEFTUJNBUPS ୅ସࢦඪͱ௕ظࢦඪͷޡ ࠩʹ༝དྷ͢Δ෼ࢄ Var(µ_Y) > Var(µ_S) ͳͷͰ

    ୅ସࢦඪͷATEͷ෼ࢄ͸௕ظࢦ ඪͷATEͷ෼ࢄΑΓখ͍͞ ୅ସࢦඪ͸௕ظࢦඪΑΓૣ͘؍ ଌͰ͖Δ΋ͷͱߟ͑Δͱ௚ײతʹ Θ͔Δɻ޿ࠂͩͱCTRͷ෼ࢄΑ Γ΋ CTR×CVRͷ෼ࢄͷํ͕େ͖ ͍
  8. $BTF4UVEZ • ໨ඪ • ʣٻ৬ऀ͕ؔ࿈͢Δٻਓ৘ใΛΑΓΑ͘ൃݟͰ͖ΔΑ͏ʹ͢Δ • ʣٻਓ޿ࠂओʹద֨ͳٻ৬ऀͷԠืΛఏڙ͢Δ • ʣ֤ٻਓ޿ࠂʹे෼ͳ਺ͷٻ৬ऀͷԠื͕ࡴ౸͗͢͠ͳ͍Α͏ʹ͢Δ •

    -JOLFE*Oʹ͓͚Δ௕ظࢦඪ͸DPOGJSNFEIJSFT $)  • -JOLFE*Oͷॿ͚Λ͔Γͯ࢓ࣄΛΈ͚ͭͨϝϯόʔ • ͨͩ͠స৬׬ྃ·Ͱʹ͸͕͔͔࣌ؒΔ • ୅ସࢦඪͱͯ͠ͷQSFEJDUFEDPOGJSNFEIJSF 1$)  • ͜ΕΛ࡞ͬͨ
  9. 1$)ͷධՁ෼ࢄͷਪఆ ଓ w ͨͩ͠1$)ͷ౷ܭݕఆྔ͸෼ࢄ͕աগධՁ͞Εͯ༗ҙʹͳΓ΍͍͢ͷͰ ิਖ਼ͨ͠෼ࢄ Λߟ͑Δ ࣜ VarAdj(μPCH) w ࣜͷ

    ΛࣜͰਪఆ͢Δɻ ͸1$)ͷ༧ଌޡࠩͷਪఆྔͱͳΔ w Ͱ΋ ͕େ͖͍ͷͰ༗ҙࠩͷग़Δ࣮ݧ͕͔Βʹݮͬͯ͠·ͬͨ ͋͞ࠔͬͨͧ σ2 ̂ σ2 ̂ σ2
  10. ڞมྔΛ༻͍ͨ෼ࢄݮগ๏ w ෼ࢄΛิਖ਼͚ͨ͠Ͳ࣮ݧ݁Ռͷ൑அ͕Ͱ͖ͳ͍ͷͰ͸୅ସࢦඪͷҙຯ͕ͳ͍ w TFOTJUJWJUZΛߴΊΔඞཁ͕͋Δ w Ұͭͷํ๏͸෼ࢄݮগ๏ WBSJBODFSFEVDUJPO  w

    ෼ࢄݮগ๏ͷதͰ΋ΦϯϥΠϯςετͰΑ͘࢖ΘΕΔͷ͕$61&% w $61&%ʹΑΓ෼ࢄ͕ʙݮগͰ͖ͨ w $61&%ద༻ޙʹ༗ҙͳ࣮ݧ͸͔Βʹ૿͑ͨ

More Decks by Takashi Nishibayashi

Other Decks in Technology