Policy-Adaptive Estimator Selection for Off-Policy Evaluation

Policy Adaptive Estimator Selection
for Off-Policy Evaluation
Takuma Udagawa, Haruka Kiyohara, Yusuke Narita,
Yuta Saito, Kei Tateno
Haruka Kiyohara, Tokyo Institute of Technology
https://sites.google.com/view/harukakiyohara
September 2022 Policy Adaptive Estimator Selection (PAS-IF) 1

View Slide

Content
• Introduction to Off-Policy Evaluation (OPE)
• Estimator Selection for OPE
• Our proposal: Policy-Adaptive Estimator Selection via Importance Fitting (PAS-IF)
• Synthetic Experiments
• Estimator Selection
• Policy Selection

View Slide

Off-Policy Evaluation
Motivation towards Estimator Selection

View Slide

Interactions in recommender systems
A behavior policy interacts with users and collects logged data.
a user feedback
(reward)
a coming user
(context)
an item
(action)

View Slide

Interactions in recommender systems
A behavior policy interacts with users and collects logged data.
a user feedback
(reward)
a coming user
(context)
an item
(action)
logged bandit feedback
behavior
policy 𝝅𝒃

View Slide

Off-Policy Evaluation
The goal is to evaluate the performance of an evaluation policy 𝜋
𝑒
.
offline A/B test
logged bandit feedback
behavior
policy 𝝅𝒃
OPE estimator
(policy performance)

View Slide

Representative OPE estimators
We aim to reduce both bias and variance to enable an accurate OPE.
✓
✓
✓
✓ ✓
model-based
importance
sampling-based
bias variance
Direct Method (DM)
[Beygelzimer&Langford,09]
--- high low
Inverse Propensity Scoring (IPS)
[Precup+,00] [Strehl+,10]
--- unbiased very high
Doubly Robust (DR)
[Dudík+,14]
unbiased
lower than IPS,
but still high
(reward predictor) (importance weight)

View Slide

✓
✓
✓
✓ ✓
model-based
importance
sampling-based
bias variance
Direct Method (DM)
--- high low
Doubly Robust (DR)
[Dudík+,14]
unbiased
lower than IPS,
but still high
✓
reward predictor

View Slide

✓
✓
✓
✓ ✓
model-based
importance
sampling-based
bias variance
Direct Method (DM)
--- high low
Doubly Robust (DR)
[Dudík+,14]
unbiased
lower than IPS,
but still high
importance weight

View Slide

✓
✓
✓
✓ ✓
model-based
importance
sampling-based
bias variance
Direct Method (DM)
--- high low
Doubly Robust (DR)
[Dudík+,14]
unbiased
lower than IPS,
but still high
importance weight
evaluation behavior

View Slide

✓
✓
✓
✓ ✓
model-based
importance
sampling-based
bias variance
Direct Method (DM)
--- high low
Doubly Robust (DR)
[Dudík+,14]
unbiased
lower than IPS,
but still high
control variate

View Slide

To reduce the variance of IPS/ DR, we make importance weights smaller.
modification on importance weights
Self-Normalized (IPS/ DR)
[Swaminathan&Joachims,15]
Clipped (IPS/ DR) *
[Su+,20a]
Switch (DR) *
[Wang+,17]
Optimistic Shrinkage (DR) *
[Su+,20a]
Subgaussian (IPS/ DR) *
[Metelli+,21]
Advanced OPE estimators
* requires hyperparameter tuning of 𝜆, e.g., SLOPE [Su+,20b] [Tucker&Lee,21]

View Slide

To reduce the variance of IPS/ DR, we make importance weights smaller.
modification on importance weights
Self-Normalized (IPS/ DR)
[Swaminathan&Joachims,15]
Clipped (IPS/ DR) *
[Su+,20a]
Switch (DR) *
[Wang+,17]
Optimistic Shrinkage (DR) *
[Su+,20a]
Subgaussian (IPS/DR) *
[Metelli+,21]
Advanced OPE estimators
* requires hyperparameter tuning, e.g., SLOPE [Su+,20b]
Which OPE estimator should be used to enable an accurate OPE?

View Slide

Motivation towards data-driven estimator selection
𝜋
𝑏
𝜋
𝑒

View Slide

𝜋
𝑏
𝜋
𝑒
Estimator Selection is important!

View Slide

𝜋
𝑏
but..
The best estimator can be different
under different situations.

View Slide

𝜋
𝑏
among the best
but..

View Slide

𝜋
𝑏
among the best
among the worst
but..

View Slide

𝜋
𝑏
among the best
among the worst
but..
- data size
- evaluation policy
- reward noise
matter.

View Slide

𝜋
𝑏
Estimator Selection: How to identify the most accurate OPE estimator
using only the available logged data?
but..
- data size
- evaluation policy
- reward noise
matter.

View Slide

Estimator Selection for OPE

View Slide

Objective for estimator selection
The goal is to identify the most accurate OPE estimator in terms of MSE.

View Slide

true policy value
(estimand)

View Slide

estimated from the logged data

View Slide

Baseline – non-adaptive heuristic [Saito+,21a] [Saito+,21b]
Suppose we have logged data from previous A/B tests.

View Slide

pseudo-evaluation policy

View Slide

※ 𝑆 is a set of random states for bootstrapping.
OPE estimate on-policy policy value

View Slide

Does non-adaptive heuristic work?
Do these estimators
really work well?
non-adaptive heuristic
(estimation)
෠
𝑉 𝜋
𝐴
; 𝐷
𝐵

View Slide

𝜋
𝑏
𝜋
𝐴
Do these estimators
really work well?
true performance
non-adaptive heuristic
(estimation)
෠
𝑉 𝜋
𝐴
; 𝐷
𝐵

View Slide

𝜋
𝑏
𝜋
𝐴
Do these estimators
really work well?
non-adaptive heuristic true performance
(estimation)
෠
𝑉 𝜋
𝐴
; 𝐷
𝐵

View Slide

𝜋
𝑏
𝜋
𝐴
Do these estimators
really work well?
non-adaptive heuristic true performance
How to choose OPE estimators adaptively
to the given OPE task (e.g., evaluation policy)?
(estimation)
෠
𝑉 𝜋
𝐴
; 𝐷
𝐵

View Slide

PAS-IF
Policy Adaptive Estimator Selection via Importance Fitting

View Slide

Is it possible to make pseudo-policies adaptive?
Non-adaptive heuristic calculates MSE using two datasets collected by A/B tests.
~𝝅
𝒃
~𝝅
𝑩
~𝝅
𝑨
pseudo-behavior policy
total amount
of logged data

View Slide

~𝝅
𝒃
~𝝅
𝑩
~𝝅
𝑨
total amount
of logged data
behavior
evaluation

View Slide

We aim to split the logged datasets adaptive to the given OPE task.
~𝝅
𝒃
~𝝅
𝑩
~𝝅
𝑨
total amount
of logged data
~෥
𝝅
𝒃
~෥
𝝅
𝒆

View Slide

Subsampling function controls the pseudo-policies
We now introduce a subsampling function .
~𝝅
𝒃
total amount
of logged data
~෥
𝝅
𝒃
~෥
𝝅
𝒆

View Slide

How to optimize the subsampling function?
PAS-IF optimizes 𝜌 to reproduce the bias-variance tradeoff of the original OPE.
Subsampling function

View Slide

How to optimize the subsampling function?
PAS-IF optimizes 𝜌 to reproduce the bias-variance tradeoff of the original OPE.
Objective of importance fitting:
Subsampling function

View Slide

Example case of importance fitting
When we have
Note: the simplified case of .

View Slide

When we have

View Slide

When we have
⇒ PAS-IF can produce a similar distribution shift!

View Slide

Summary of the key points
• PAS-IF produces the pseudo OPE datasets via adaptive subsampling.
• thereby enables an accurate and adaptive estimator selection.

View Slide

Synthetic Experiment

View Slide

Experimental settings
We compare PAS-IF and non-adaptive heuristic in two tasks.
1. Estimator Selection
2. Policy Selection using the selected estimator

View Slide

hyperparam tuning* estimator selection
* SLOPE [Su+,20b] [Tucker&Lee,21]

View Slide

PAS-IF enables an accurate estimator selection
PAS-IF enables far more accurate estimator selection by being adaptive.
PAS-IF is accurate across
various evaluation policies
lower, the better
𝜋
𝑏1
𝜋
𝑏2
𝜋
𝑏1
𝜋
𝑏2
ෝ
𝑚 -- selected
𝑚 ∗ -- true best

View Slide

hyperparam tuning estimator selection

View Slide

hyperparam tuning estimator selection
෠
𝑉
1
෠
𝑉
2
෠
𝑉
3
PAS-IF
different estimator
for each policy
non-adaptive
෠
𝑉
universal estimator
for all policies
෠
𝑉
෠
𝑉

View Slide

Moreover, PAS-IF also benefits policy selection
PAS-IF also reveals a favorable result in the policy selection task.
PAS-IF can identify better policies among many candidates
by using different (appropriate) estimator for each policy!
lower, the better
ො
𝜋 -- selected 𝜋 ∗ -- true best

View Slide

Summary
• Estimator Selection is important to enable an accurate OPE.
• PAS-IF enables an adaptive and accurate estimator selection by subsampling
and optimizing the pseudo OPE datasets.
• The empirical results show that PAS-IF is beneficial in both estimator selection
and the downstream policy selection tasks.
PAS-IF will help practitioners identify an accurate OPE estimator!

View Slide

Thank you for listening!
contact: [email protected]

View Slide

Detailed optimization procedure of PAS-IF
We optimize the subsampling rule 𝜌
𝜃
via gradient decent.
To maintain the similar data size with the original OPE task, PAS-IF also imposes
the regularization on the data size.
We tune 𝜆 so that .

View Slide

Key idea of PAS-IF
How about sampling the logged data and constructing a pseudo-evaluation policy
that has a bias-variance tradeoff similar to the given OPE task?
(𝑆 is a set of random states for bootstrapping)

View Slide

References

View Slide

References (1/4)
[Beygelzimer&Langford,00] Alina Beygelzimer and John Langford. “The Offset Tree
for Learning with Partial Labels.” KDD, 2009.
[Precup+,00] Doina Precup, Richard S. Sutton, and Satinder Singh. “Eligibility Traces
for Off-Policy Policy Evaluation.” ICML, 2000.
https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1079&context=cs_facult
y_pubs
[Strehl+,10] Alex Strehl, John Langford, Sham Kakade, and Lihong Li. “Learning from
Logged Implicit Exploration Data.” NeurIPS, 2010. https://arxiv.org/abs/1003.0120
[Dudík+,14] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. “Doubly
Robust Policy Evaluation and Optimization.” ICML, 2011.
https://arxiv.org/abs/1503.02834

View Slide

References (2/4)
[Swaminathan&Joachims,15] Adith Swaminathan and Thorsten Joachims. “The Self-
Normalized Estimator for Counterfactual Learning.” NeurIPS, 2015.
https://dl.acm.org/doi/10.5555/2969442.2969600
[Wang+,17] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudík. “Optimal and
Adaptive Off-policy Evaluation in Contextual Bandits.” ICML, 2017.
[Metelli+,21] Alberto M. Metelli, Alessio Russo, Marcello Restelli. “Subgaussian and
Differentiable Importance Sampling for Off-Policy Evaluation and Learning.” NeurIPS,
2021.
https://proceedings.neurips.cc/paper/2021/hash/4476b929e30dd0c4e8bdbcc82c6b
a23a-Abstract.html

View Slide

References (3/4)
[Su+,20a] Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík.
“Doubly Robust Off-policy Evaluation with Shrinkage.” ICML, 2020.
[Su+,20b] Yi Su, Pavithra Srinath, and Akshay Krishnamurthy. “Adaptive Estimator
Selection for Off-Policy Evaluation.” ICML, 2020. https://arxiv.org/abs/1907.09623
[Tucker&Lee, 21] George Tucker and Jonathan Lee. “Improved Estimator Selection
for Off-Policy Evaluation.” 2021.
https://lyang36.github.io/icml2021_rltheory/camera_ready/79.pdf
[Narita+,21] Yusuke Narita, Shota Yasui, and Kohei Yata. ”Debiased Off-Policy
Evaluation for Recommendation Systems.” RecSys, 2021.

View Slide

References (4/4)
[Saito+,21a] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita.
“Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy
Evaluation.” NeurIPS dataset&benchmark, 2021. https://arxiv.org/abs/2008.07146
[Saito+,21b] Yuta Saito, Takuma Udagawa, Haruka Kiyohara, Kazuki Mogi, Yusuke
Narita, and Kei Tateno. “Evaluating the Robustness of Off-Policy Evaluation.” RecSys,
2021. https://arxiv.org/abs/2108.13703

View Slide

Policy-Adaptive Estimator Selection for Off-Policy Evaluation

Policy-Adaptive Estimator Selection for Off-Policy Evaluation

Haruka Kiyohara

More Decks by Haruka Kiyohara

Other Decks in Research

Featured

Transcript