KDD 2023 Workshop - Causal Inference and Machine Learning in Practice An IPW-based Unbiased Ranking Metric in Two-sided Markets Keisho Oh1, Naoki Nishimura1, Minje Sung2, Ken Kobayashi2, Kazuhide Nakata2 1Recruit Co., Ltd., 2Tokyo Institute of Technology

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Background : Unbiased Learning-to-Rank • In recommender systems, we want to recommend items that are relevant to users • When utilizing users’ implicit feedback (click, purchase etc.), position biases should be cared about 2 Customer Recommended Items Purchase Relevant BUT not observed Observed Implicit Feedback do not necessarily represent users’ true relevance!

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Unbiased Learning-to-Rank for two-sided markets • Conventional studies mainly deal with one-sided markets (ex. E-Commerce) • We want to extend existing research to “two-sided markets” (ex. Job-matching , Dating apps) 3 Customer Product Click/ Purchase One-sided Proactive side / applicant Reactive side / employer Apply Hire Two-sided Final conversion is determined by reciprocal preferences

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Recommendation for proactive users • Recommend items that satisfy both side’s preferences to proactive users 4 Apply & Hire : satisfies both sides preference Apply : only satisfies proactive side’s preference No action

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice • Assessing the qualities of rankings is difficult since each side’s feedback is biased Challenge in two-sided markets 5 Apply Recommend list 1. Application phase sorted by some Candidate pool Hire 2. Hiring phase sorted by arrival Recommender A Ranking Recommender B Apply & Hire No action 1 2 3 Apply & Hire No action Based on true relevance (unobservable) Based on implicit feedback Apply & Hire No action Recommender A Ranking Recommender B No action 1 2 3 Apply & Hire No action No action Apply & Not Hire Recommender A > Recommender B Recommender A < Recommender B? No action Not Observed Not Observed

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Our Contribution • We proposed an unbiased ranking metric under the condition of two-sided metric • That is an natural extension of conventional studies • We proved that our method gives better performance in semi-synthetic data 6

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Problem Setting 7 AAAJ73icrVY9b9NgEL6WAk35aAsLEksgLWollL6pKkBMlQoSC1Ha0g+pCZXtuIlVJ3YdJ7SY/AFYQR2QECAYKgbY+AEM8AeQ6E9AjEViYeDubCeOncQdcJT49b338dzdc28sm7pWs4U4HBg8MXTy1OnhxMiZs+fOj46NX1itGXVLUVcUQzesdVmqqbpWVVdszdbVddNSpYqsq2vy9gLtrzVUq6YZ1Qf2nqkWKlKpqm1pimSjaLG+OZYSacFXMrrIeIsUeFfOGB/SIQ9FMECBOlRAhSrYuNZBghp+NiADAkyUFcBBmYUrjfdVaMII2tZRS0UNCaXb+FvCpw1PWsVn8lljawWj6Pi10DIJk+K7OBBH4pv4IH6Kvz19OeyDsOzhXXZtVXNz9Oml5T+xVhW821BuW/XFbMMW3GKsGmI3WUJZKK594/H+0fLtpUnnmngrfiH+N+JQfMEMqo3fyvtFdekle7fYRoVHnHGFMVSxxg5Xusl3H5eFK4elVM1J3CmhTMLYZa5zjVGSpoOfGXwiqY3IKMqMZ5fvGc/knhGeJveywFLq3hZ6mYIUygixhnHLKJn2/JGGjXeq0w56sdBD0Ftcng2UKx0xqXcG7PJaZiYUW1VIwgQjmWjVIZkdyf4Q98U7ZMhX8VF8xvsnjlpk7HnuEK13A6jIh8PIJNwn7H5MAWnsLPGY6j/BmSmcN8k61zYyjfKfYjx+hf6Hx+lWhu08Gn0zaTATJJ5Kg7lCO02OVuEZLbNeNx++bhxH6mhd5F7Ncq/I6qEXvZff2YBfPxNircnWtO94lbvSwZu2LVneYZbo3u4SIpN5Nm1Y6MKoMM7jV8e19HfD+HvVxc1a5XmT+NwzQnPk81fGjwN3ObI/WxuR2Sr43O4Tk+qgo4TwFr1TNhrP1XI4XlDXjRjVio9bQi8N75zojuNJTLXCHipet4IMqfLpT1Z6gCc+c6LnYpaxu327jviD3OmFQ/PiRr1pOLH0LeI3zF2FT1Qd51TmWVe8agZPy3yL0+2e5mPx5FDb4P+tKKIc/xf2t8+yRZ1R6fhE/9LNLuzLxnqyOA86z9ucmmrJ/emYjvFCs1piyW4Ih+H1kHJLhvSoihpzgs7D40XQAlzqF8HVi0TAd6FM+M0nulidTWdupOcW51LzOe+taBguw1WsTAZuwjzcwy6tMB+ewXN4kdhJ7CdeJV67qoMDns1F6LgSB/8AOygFVA== u Proactive side Recommend List Vu Apply AAACfHichVHLSgMxFD0d3/XRqgsFN4NVEcSSEV8IguDGpa/6QKXMjLEGpzNDJq1o8Qf8AReuVERE/Qo3/oALP0FcKrgR8XY6ICrqDUlOTu65OUks3xGBYuwhplVV19TW1TfEG5uaWxLJ1ralwCtIm2dsz/HkimUG3BEuzyihHL7iS27mLYcvWzvT5f3lIpeB8NxFtefzjbyZc8WWsE1FVDbZsZotFfR1KXLbypTS29WLB/qkbmSTKZZmYeg/gRGBFKKY9ZIXWMcmPNgoIA8OF4qwAxMBtTUYYPCJ20CJOElIhPscB4iTtkBZnDJMYndozNFqLWJdWpdrBqHaplMc6pKUOnrZPbtkz+yOXbFH9vZrrVJYo+xlj2arouV+NnHYufD6rypPs8L2p+pPzwpbGA+9CvLuh0z5FnZFX9w/el6YmO8t9bFT9kT+T9gDu6UbuMUX+3yOzx8jTh9gfH/un2BpKG2MpkfmhlNT49FX1KML3ein9x7DFGYwi0x47hmucRN713q0AW2wkqrFIk07voQ2+gGEhJK7 Yu!v = 1 AAACe3ichVHLSsNAFD2Nr1pfVUEEN8GiiEiZiNUiCIIbl/VRH7RSkjitwTQJybRSqz/gD7hwZUFE9C/c+AMu+gnisoIbBW/TgKiod5iZM2fuuXNmRnNMwxOM1UNSW3tHZ1e4O9LT29c/EB0c2vLskqvztG6btrujqR43DYunhSFMvuO4XC1qJt/WDlea+9tl7nqGbW2KisP3imrBMvKGrgqictGR3Vy1JGdNnheq69pHcvlUXpKVXDTG4swP+SdQAhBDECk7eo0s9mFDRwlFcFgQhE2o8KhloIDBIW4PVeJcQoa/z3GKCGlLlMUpQyX2kMYCrTIBa9G6WdPz1TqdYlJ3SSljgj2yG9ZgD+yWPbG3X2tV/RpNLxWatZaWO7mBs9GN139VRZoFDj5Vf3oWyCPpezXIu+MzzVvoLX35+Lyxsbg+UZ1kNfZM/i9Znd3TDazyi361xtcvEKEPUL4/90+wNRtX5uOJtbnYcjL4ijDGMI4peu8FLGMVKaTp3BPUcIu70LsUk6almVaqFAo0w/gSUuIDb3+SPg== Yu v = 1 Hire No action AAACfHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuIR8ZXlyrEFGWmZ5QkFhXllyuU1SrYKhjECygb6BmAgQImwxDKUGaAgoB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AzN9EwDTZQdLKBRwcEgzaDEoAEMb3MGBwYPhgCGULC9ixnWMKxl/MekwqTNpAtRysQI1SPMgAKYzACChJK6 Yu!v = 0 AAACfXichVHLSgMxFD0dX7U+WnUjuCmWioKUjKgVQRDduNTWqqBSZsZUh86LmbRQh/oB/oALV4oiIvoTbvwBF36CuFRwI+jtdEBU1BuSnJzcc3OSqI6he4Kxh4jU0trW3hHtjHV19/TGE339a55dcTVe0GzDdjdUxeOGbvGC0IXBNxyXK6Zq8HW1vNjYX69y19Nta1XUHL5tKruWXtI1RRBVTAxumYrYc03fVaxyvehX6qPVseRcUi4mUizDgkj+BHIIUghj2U5cYAs7sKGhAhMcFgRhAwo8apuQweAQtw2fOJeQHuxz1BEjbYWyOGUoxJZp3KXVZshatG7U9AK1RqcY1F1SJpFm9+ySPbM7dsUe2duvtfygRsNLjWa1qeVOMX44mH/9V2XSLLD3qfrTs0AJM4FXnbw7AdO4hdbUV/ePnvOzubQ/wk7ZE/k/YQ/slm5gVV+08xWeO0aMPkD+/tw/wdpERp7OTK1MpuYXwq+IYgjDGKX3zmIeS1hGgc49wBmucRN5l9LSuJRppkqRUDOALyFlPwD1bpNr ranku(v) = 1 Ranking in the recommend list for the user AAAEVXiclVNNaxNBGH7SrLXWj7T1IngphoqnMBG/8NRaxV6UJm3aQC1hdzOJQ/aL2UlqXfIH9Kr00IMoeBB/hhf/gEh/gnis4EXBd2a3LZo0xVl2553nfZ93nnnfHSfyRKwY28uN5a1T46cnzkyePXf+QmFqemYtDrvS5TU39EJZd+yYeyLgNSWUx+uR5LbveHzd6Sxq/3qPy1iEwarajvimb7cD0RKurQiq9BpTRVZiZswOGuXMKCIby+F07g2eoIkQLrrwwRFAke3BRkzPBspgiAjbREKYJEsYP0cfk8TtUhSnCJvQDn3btNrI0IDWOmds2C7t4tEriTmLOfaFfWD77DP7yL6xX8fmSkwOrWWbZifl8qhReHFp5eeJLJ9mhadHrJGaFVq4Y7QK0h4ZRJ/CTfm95zv7K3erc8lV9o59J/1v2R77RCcIej/c9xVe3R2h50CJogroOsSmfjo+IP+WqY5vogLqR0K4JLufWYKyaK7ugKTIraz6x3G9Q67uVOs/mA7hklYp+0C1JCvBvUPf6BzS2J2hOY58J59A97tp5vTP6w94GkN2eIQFVLFqdhi9xzPT3SZ52kO11vGA8MYA/hj3sYiHOj/duPK/92vQWLteKt8q3azcKM4vZXdvApdxBdfoft3GPJawjBrp4HiJV3id/5r/bVnWeBo6lss4F/HXsAp/APer724= v AAACfnichVG7SgNBFD2urxhfURvFJhiMWhgnwReCINpYGjUPMCHsrqMu2Re7k4Auwd4fsLBSEBFBP8LGH7DIJ4ilgo2FN5sFUVHvMDNnztxz58yMYuuaKxirt0itbe0dnaGucHdPb19/ZGAw61oVR+UZ1dItJ6/ILtc1k2eEJnSetx0uG4rOc0p5rbGfq3LH1SxzWxzavGjI+6a2p6myIKoUGSkYsjhwDM+RzXKt5FVqk9WJqehyNFWKxFiC+RH9CZIBiCGIDStyhQJ2YUFFBQY4TAjCOmS41HaQBINNXBEecQ4hzd/nqCFM2gplccqQiS3TuE+rnYA1ad2o6fpqlU7RqTukjGKcPbJr9sIe2A17Yu+/1vL8Gg0vhzQrTS23S/0nw1tv/6oMmgUOPlV/ehbYw6LvVSPvts80bqE29dWj05etpc1xL84u2DP5P2d1dk83MKuv6mWab54hTB+Q/P7cP0E2lUjOJ+bSs7GV1eArQhjFGCbpvRewgnVsIEPnHuMSt7iTIMWlaWmmmSq1BJohfAlp8QNNCJKe ranku(v0) = 2 AAACf3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtIxeQmlmQU5VYXJeZl18ZXl9ZqlKmrayrYKhjHCygb6BmAgQImwxDKUGaAgoB8geUMMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNZuACRoAhenBjMsKM9AzN9EwDTZQdnKBRwcEgzaDEoAEMb3MGBwYPhgCGUKC9DQzLGNYzbGBiZFJn0mMygChlYoTqEWZAAUyWAMuUktA= ranku(v00) = 3 AAAEVniclVNNTxNBGH5KiyB+UPRi4oXYIJyaKfErnlA0cpFAodAESbO7ndZJ9yuz0yJu+geMV/XgwWjigfgzuPAHTOQnGI+YeDHGd2YXiLaUMJvdeed53+edZ953xw5dESnG9jND2dzwuZHR82MXLl66PJ6fuLIWBW3p8IoTuIGs2lbEXeHzihLK5dVQcsuzXb5ut+a1f73DZSQCf1Vth3zTs5q+aAjHUgSVO9O1fIEVmRmTvUYpNQpIx1IwkfmAZ6gjgIM2PHD4UGS7sBDRs4ESGELCNhETJskSxs/RxRhx2xTFKcIitEXfJq02UtSntc4ZGbZDu7j0SmJOYop9ZTvsgO2xL+w7+31irtjk0Fq2abYTLg9r46+urfw6leXRrPD8mDVQs0ID94xWQdpDg+hTOAm/8/Ldwcr98lR8k31iP0j/R7bPdukEfuen83mZl98P0HOoRFEFdB0iUz8d75N/y1THM1E+9SMmXJLdTS1BWTRXd0BS5FZa/ZO47hFXd6pxBqZNuKRVwj5ULcmK8fDINziHNHarb45j3+kn0P2umzn587o9nlqfHZ7iAcpYNTsM3uOF6W6dPM2+Wqt4THitB1/EI8zjic5PN670//3qNdZmi6U7xdvLtwpzC+ndG8V13MAM3a+7mMMCllAhHQ28xhu8zX7L/skN50aS0KFMyrmKf0Yu/xe5OO+f v0 AAAEV3iclVPLbhMxFD1pQgnl0QY2SGwqolJWkVPxqLoqFEQ3oDZp2kilimYmTrAyL3mclDLKDyCxpQsWCCQWiM9gww+wyCcglkVigwTXnmkrSJoKj2Z8fe4918f3ju3QFZFibJCZyObOTJ7Nn5s6f+HipemZwuXNKOhKh9ecwA1k3bYi7gqf15RQLq+Hklue7fItu7Oi/Vs9LiMR+BtqL+Q7ntX2RUs4liKo2pufb8wUWYmZMTtslFOjiHSsBYXMWzxFEwEcdOGBw4ci24WFiJ5tlMEQEraDmDBJljB+jj6miNulKE4RFqEd+rZptZ2iPq11zsiwHdrFpVcScxZz7Cv7yA7YF/aJfWO/TswVmxxayx7NdsLlYWP65dXqz1NZHs0Kz45ZYzUrtLBotArSHhpEn8JJ+L0X+wfVpcpcfIO9Z99J/zs2YJ/pBH7vh/NhnVfejNFzqERRBXQdIlM/He+Tf9dUxzNRPvUjJlyS3U8tQVk0V3dAUuRuWv2TuO4RV3eq9R9Mm3BJq4R9qFqSFeP+kW98Dmnszsgcx77TT6D73TRz8uf1hzyNETs8xj1UsGF2GL/Hc9PdJnnaI7XW8ZDwxhD+BA+wgkc6P9248r/3a9jYXCiV75Rur98qLq+mdy+Pa7iOm3S/7mIZq1hDzeh4hdfYzw6yv3OTuXwSOpFJOVfw18gV/gB67u/Q v00

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Implicit Feedback Mechanism in two-sided markets 8 AAAKMnicrVY9b9NgEL6WAk35aAsLEouhLWollL6pKkBISBUFiYXQD/qBmhDZjptYdezgOGmLyR/gDzAwgUACMcCAxA9ggIkN0bIhJgRbkVgYuDvbiWs3SQccJX597308d/fcGytlQ684Qmx3dR/oOXjocG+i78jRY8f7BwZPLFasqq1qC6plWPayIlc0Qze1BUd3DG25bGtySTG0JWVtmvaXappd0S3ztrNZ1rIluWDqq7oqOyjKDVy9k3OrGVsvFB3Ztq11qVaXrki3UChFpBk1bznSXHwnNzAkkoIvKb5I+Ysh8K8Za7DHgAzkwQIVqlACDUxwcG2ADBX8rEAKBJRRlgUXZTaudN7XoA59aFtFLQ01ZJSu4W8Bn1Z8qYnP5LPC1ipGMfBro6UEI+KzeCl2xEfxSvwQf1v6ctkHYdnEu+LZauVc/8NT8386WpXw7kCxadUWswOrcImx6oi9zBLKQvXsa/cf7cxfnhtxz4mn4ififyK2xXvMwKz9Vp/PanOP2bvNNhqsc8YlxmBijV2udJ3vAS4bVy5LqZojuFNAmYyxi1znCqMkTRc/4/hEUgeRUZRx3y7TMl6Ze0Z46tzLLEupe6voZRSGUEaIdYxbRMmY7480HLxTne6hFxs9hL11yrOGcnVXTOqdBRu8VpgJ+UYVJBhmJMONOkjpvvSWuCmeIUM+iNfiHd7fcNQ8Y89wh2i9EUJFPlxGJuM+YQ9iCkhiZ4nHVP9hzkzlvEm2e+0g0yj/UcYTVOh/eBxrZNjMo9Y2kxozQeaptJgrtFPnaCWe0SLr7eUj0O3EkSpa57lXE9wrsrrrR2/ldyLkN8iEWFtma9p3/cqd2cWbpi1ZXmOWGP7uHCJTeDYdmN6DUVGc+6+OZxnsRvG3qouXtcbzJvO5Z0XmKOCvgh8XrnPkYLZWYrOVDbjdJibVwUAJ4c37p2w8nqflcrywrhcxrtU5bgG91PxzYm8cDzpUK+qh5HcrzBCTT3+yMkI8CZgTPxfTjN3r23nEH+ZOKxy6HzfuTceJpW8ev1HuqnyiGjinCs+66lczfFpmGpxu9jTTEc8Malv8vxVHNMP/he3t02xRZVQGPtG/dH0P9qU7erI5DzrPm5wabciD6Rjr4IVmtcCSjQgOy+8h5SZF9KiKOnOCzsP9RdBDXGoXwdOLRcB3oVT0zSe+WJxIpi4kJ2cnh6Zm/LeiXjgNZ7EyKbgIU3ADu7SAKF/AJ/gCW4m3ia+Jb4nvnmp3l29zEnZdiV//AEihIKw= Yu!v = Ou!v · Ru!v Applied Exposed Relevant AAAKSnicrVY9b9NgEL5CgaZ8tIUFicXQFrUSat9UFSAkpEoFiYXQD/oBTYlsx0msOnFwHLfF5A/wBxiYQDAgBthYkRhgYEWifwAJMYAoEkIwcHe2EydO4g44Svz63vt47u65N1bKhl6xhdjp2be/98DBQ32J/sNHjh4bGBw6vlwxq5aqLammYVqrilzRDL2kLdm6bWirZUuTi4qhrSgbs7S/4mhWRTdLN+3tsrZelPMlPaerso2izODtWxm3KqUNLWfLlmVuSk5Nuix5QkvPFxrStJo1belGVN3bWIhsZAaHxYTgS4oukv5iGPxrzhzqNSANWTBBhSoUQYMS2Lg2QIYKftYgCQLKKFsHF2UWrnTe16AG/WhbRS0NNWSUbuBvHp/WfGkJn8lnha1VjGLg10JLCUbFR/Fc7Ir34oX4Iv529OWyD8KyjXfFs9XKmYEHJxd/xVoV8W5DoWHVFbMNObjIWHXEXmYJZaF69s69h7uLlxZG3bPiifiK+B+LHfEWMyg5P9Vn89rCI/ZusY0Gm5xxkTGUsMYuV7rG9wCXhSuXpVTNUdzJo0zG2AWuc4VRkqaLn0l8IqmNyCjKpG+X7hivzD0jPDXu5TpLqXs59DIGwygjxDrGLaBk3PdHGjbeqU530YuFHsLe4vJ0UK42xaTembDFa4WZkK1XQYIRRjJSr4OU6k99EtfFU2TIO/FSvMb7K46aZexp7hCtt0KoyIfLyGTcJ+xBTAET2FniMdV/hDNTOW+SNa9tZBrlP8Z4ggr9D4/j9QwbeThdM3GYCTJPpclcoZ0aRyvyjBZYr52PQDeOI1W0znKvprhXZHXHj97J71TIb5AJsbbM1rTv+pU73cSbhi1ZXmGWGP7uAiJTeDZtmG3DqFace6+OZxnstuLvVBcva43nTeZzz2yZo4C/Cn5cuMqRg9lai8zWesDtLjGpDgZKCG/WP2Wj8Twtl+OFdb2IUa34uHn04vjnRHsc92Oq1eqh6HcrzJASn/5kZYR4EjAnei6mGLvXt3OIP8ydTjh0P27Um44TS98sflu5q/KJauCcKjzrql/N8GmZrnO60dN0LJ451Db5fyuKaI7/C7vbp9iiyqgMfKJ/6Vob9qViPVmcB53nDU6N1eXBdIzHeKFZzbNkqwWH6feQcpNa9KiKOnOCzsO9RdBDXOoWwdOLRMB3oWTrm090sTw1kTw/MT0/PTwz578V9cEpOIOVScIFmIFr2KUlRPkGPsM3+J74kPiR+J3446nu6/FtTkDT1d/7D8xuKSo= Yu v = Yu!v · Ou v · Ru v Hired Exposed Relevant Applied Assumptions • The reactive side feedback happens only when the corresponding user has already given the feedback • the exposure prob. solely depends on the position of , i.e. • The same holds for the reverse direction θu→v = P(Ou→v = 1) v Ou→v ∼ P( ⋅ ∣ ranku (v)) = = & & &

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Ranking Metric • Two-sided relevance • A successful ranking sorts items in descending order of • The following ranking metrics using true relevances should be optimized 9 AAADO3icnVHLbtNAFL02rxIeTWFTiY1F2ooFisYVUISoFIkNyzYlbaW6iuzJTTLKZGyNJy7Fcj+gP9AFK5AQQnwDKzb8AEhds+KxK4gNC64d8ygVIHGt8dw5954zZ2aCSIrYMLZv2ceOnzh5auJ05czZc+cnq1MXVuNwpDm2eChDvR74MUqhsGWEkbgeafSHgcS1YHAnr68lqGMRqntmO8LNod9Toiu4bwhqV9802+koyZxFxwuwJ1TKSSzOKk4Z83OewfsmFd2dmbzV8bTo9Y2vdbjlJNmiO+P4qvO9JrH7aym76nk/lNz/V2KHlVipFJo+6i0RY1Yvah6qTum/Xa2xOivCOZq4ZVKDMpbC6lPwoAMhcBjBEBAUGMol+BDTtwEuMIgI24SUME2ZKOoIGVSIO6IupA6f0AH9e7TaKFFF61wzLticdpE0NDEdmGWv2TN2wF6x5+wd+/pHrbTQyL1s0xyMuRi1J3enV778kzWk2UD/J+uvng104WbhVZD3qEDyU/AxP3mwd7ByqzmbzrHH7AP5f8T22Us6gUo+8SfL2HwIFXoA9/frPpqsztfdG/Xry9dqjUb5FBNwCS7DFbrvBWjAXViCFnDrthVYA0vaL+y39nv747jVtkrORTgU9udvG1DT/A== Ruv = 8 > < > : 2 if Ru!v = 1 and Ru v = 1, 1 if Ru!v = 1 and Ru v = 0, 0 otherwise. AAAE8HiclVPLbtNAFL0uAUp4NIUNEpsRUaNEQtEE8VIlpEBBsAG1aZJG6gTLdiaJFT/C2E4prveIHwCJBSoSC+Az2PQHWPQTEMsisWHB9di0KklTMZYz554759w744w+tEzPp3RXmTmROXnq9OyZ7Nlz5y/M5eYvNj03EAZvGK7lipauedwyHd7wTd/iraHgmq1bfE0fLMX5tREXnuk6dX9zyNu21nPMrmloPlJqbqfAbM3vG5oV1iJm8a5fZEJzBmoYRMVR6RqpIUKGjKIEE2bFARNmr++XCHNcJ7B1LghjLFsg7HmgdRbvEtYVmhFWolB6bpEGSRRbEfMCW/qYDmlEJAlHMmyqQRT723pHO9QG1jE6rk96xaSF8X5Kai5Py1QOMg4qKchDOpbdeeUVMOiACwYEYAMHB3zEFmjg4bMOFaAwRK4NIXICkSnzHCLIojbAVRxXaMgO8LeH0XrKOhjHnp5UG1jFwlegksAC/UY/0T26Q7/Q7/T3kV6h9Ih72cRZT7R8qM69vrz661iVjbMP/QPV1J596MId2auJvQ8lE+/CSPSjl2/2VhdrC2GBfqA/sP9tuku/4g6c0U/j4wqvvUP32N9B1Ybcry07cPCEQ+QF4ihFJlbto398pgJXbqTneZTW2tfGZ9/9D6WOvMAoUf89EYEohPv7uekeQuLBRI+D3PE7iL9gR87Jfykay6gTKjyBe1CDuqwwvcYL+b06mOlN7LUFD5FXx/in8ACW4FHsj3eo8u+NGQfN6+XKrfLNlRv5ajW9TbNwBa5CEW/MbajCY1iGBhhKWakrbeVZRmTeZt5ntpOlM0qquQSHRubzHxUuLfY= R (ranku(v), Ru!v, Ru v) := 1 |U| X u2U X v2Vu (ranku(v)) · g(Ru!v, Ru v) AAAElHiclVPNaxNBFH8bV631o6mCCF4WQyVBCLPFL0QhGouCWNq0aQPdsuxuZpMlsx/MTlLrsnfxH/DgScGD+Gd48Sx46J8gHivowYNvJ2tLTZrSWXbnze+93++9ebNjR8yLBSE7SuGEevLU6akz02fPnb8wU5y9uBaHfe7QphOykLdsK6bMC2hTeILRVsSp5duMrtu9euZfH1Aee2GwKrYjuulbncBzPccSCJlFZjAMblsGo64oG9wKembST8uDisG9TldUHmiGyy0nMXxLdG030dMDURoStWeVNDFY2DHnR2W0G5qeS6VmsUSqRA5t1NBzowT5WApnlVdgQBtCcKAPPlAIQKDNwIIYnw3QgUCE2CYkiHG0POmnkMI0cvsYRTHCQrSH3w6uNnI0wHWmGUu2g1kYvhyZGsyRb+Qj2SVfyCfynfw5VCuRGlkt2zjbQy6NzJnXV1Z+HcnycRbQ3WdNrFmAC3dlrR7WHkkk24Uz5A9evtldudeYS66T9+QH1v+O7JDPuINg8NP5sEwbb1E90w+QtSX368sKAuxwgjhHO80tD7N2UT/rKcfIrbyfh3HZHjfrvXsMpo04x9WQ/a8jHK0EHu35JmtwaffGauz7jt5BdoJtOQ//pXTEY47J8BweQgNWZYbJOV7I82qjpzO21hYsIG6O4IvwGOrwJNPHO6T/f2NGjbX5qn67emv5ZqlWy2/TFFyFa1DGG3MHavAUlqCJdXyF34qiFNTL6n21ri4MQwtKzrkEB4a6+Be+3gxV (ranku(v)) = 1(ranku(v)  K) log2 (ranku(v) + 1) AAAEU3iclVNNb9NAEH35AEppaQoXJC4WUasiINpUfAkJKVAQXEBt2rSR2mLZzia14tjW2kkplu+IGycOnEDigPgBXEFc+AM99CcgjkXiAAdm16YVTZqKteydebPvzeys1/QdOwgZ28lkc/ljx0+MnBw9NTZ+eqIweWY58LrC4jXLczxRN42AO7bLa6EdOrzuC250TIevmO05GV/pcRHYnrsUbvl8vWO0XLtpW0ZIkF641Jqp6lFXWxOG1osva4njSOeidlubfRJJpBfH2pWyXiiyElND6zfKqVFEOua9ycxzrKEBDxa66IDDRUi2AwMBPasog8EnbB0RYYIsW8U5YowSt0urOK0wCG3Tt0Xeaoq65EvNQLEtyuLQK4ipYYpts/dsl31lH9g39vtQrUhpyFq2aDYTLvf1iRfnFn8eyerQHGJjnzW05hBN3FS12lS7rxC5Cyvh95692l28VZ2Kptlb9p3qf8N22Bfagdv7Yb1b4NXXpC71XWJtqv12VAUudTgiXJAdp5ZNWTdIX/ZU0MrNtJ+HcZ09rux98z+YJuGCvIT9tyOCrAh392LDNYSy2wM19mNH70CeYEPNyb8U90X0ARke4Q6qWFIZhud4qs6rQZHWwFrruE+43oc/xj3M4YHUpztUPnhj+o3l2VL5eunawtVipZLephGcxwXM0I25gQoeYh41quMlPuITPue2c7/y2Xw+WZrNpJyz+Gfkx/8AYqP0kQ== g(Ru!v, Ru v) = 2Ruv 1 AAAEWniclVPLbtNAFD1JDfQBtAUWSGwqoiJW0QTxEqvSgmADatMmjVQqy3YmYYhfGk9SipUfYMUOAQsEEgvEZ7DhB1hU4gcQyyKxYcGdsdsKkqZiLHvunHvPnTP3etzYF4libKdQHLOOHT8xPjE5dfLU6emZ2TP1JOpKj9e8yI9kw3US7ouQ15RQPm/EkjuB6/N1t7Ok/es9LhMRhWtqO+abgdMORUt4jiKoXrXTbq9vz5RYmZkxN2hUcqOEfCxHs4W3eIQmInjoIgBHCEW2DwcJPRuogCEmbBMpYZIsYfwcfUwSt0tRnCIcQjv0bdNqI0dDWuuciWF7tItPryTmHObZV/aR7bIv7BP7zn4fmis1ObSWbZrdjMtje/r5+dVfR7ICmhUeH7BGalZo4abRKkh7bBB9Ci/j95693F29VZ1PL7H37Afpf8d22Gc6Qdj76X1Y4dU3I/TsKVFUAV2HxNRPx4fk3zLVCUxUSP1ICZdk93NLUBbN1R2QFLmVV/8wrr/P1Z1q/QfTJVzSKmPvqZZkpVjc943OIY3dGZrjwHf0CXS/m2bO/rz+gMcessMD3EYVa2aH0Xs8Nd1tkqc9VGsDdwm3B/CHuIMl3NP56cZV/r1fg0b9SrlyvXxt5WppYTG/e+O4gIu4TPfrBhZwH8uokY4neIFXeD32zSpaE9ZUFlos5Jyz+GtY5/4A9FjwuQ== Ruv Equivalent to AAAEYniclVNNTxNBGH4Kq2L9gMLFRA/EBuOpmRq/wqlSDCYGA4VCE8Bmdzstm+5XZqdF3PQP+Ac08WAg8UD8GV78Ax64ezEeMfHiwXdmF4i2lDib3Xnned/nnWfed8cKXSeSjB1mRkaNCxcvjV3OXrl67fr4RG5yLQo6wuZVO3ADUbPMiLuOz6vSkS6vhYKbnuXydatdVv71LheRE/ircjfkW57Z8p2mY5uSoJebnim3hRfPlxd6pef1iTwrMD2m+41iauSRjqUgl/mATTQQwEYHHjh8SLJdmIjo2UARDCFhW4gJE2Q52s/RQ5a4HYriFGES2qZvi1YbKerTWuWMNNumXVx6BTGnMcO+sgN2xL6wT+w7+31mrljnUFp2abYSLg/r429urPw6l+XRLLF9yhqqWaKJx1qrQ9pDjahT2Am/+/rt0cpsZSa+w/bZD9K/xw7ZZzqB3/1pf1zmlfdD9BwrkVQBVYdI10/F++Tf0dXxdJRP/YgJF2T3UsuhLIqrOiAociet/llc94SrOtX8D6ZFuKBVwj5WLciKMXfiG55DaLs9MMep7/wTqH439Jz8eb0+T33ADot4ggpW9Q7D93ilu9sgT2ug1hqeEl7vw19gHmUsqPx044r/3q9+Y+1eofiw8GD5fr40l969MdzEbdyl+/UIJTzDEqqkQ+Ad9rA/+s3IGjljKgkdyaScKfw1jFt/AK6286s= DCG@K

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Example • A naive plug-in estimator doesn’t work • Example : 10 AAAEMXiclVPPTxNBGH1lVRB/AHox8dLYYDw1U6NiPCFo5KKBQqFJIZvd7bRMur8yuy3gpnfj3XggwWjiwXj0T/DiP+ChN+PNcMTEiwe/mV0h2lLCbLbz5n3z3vfNtx07dEUUM9bLjRhnzp4bHTs/fuHipcsTk1NXVqOgLR1ecQI3kFXbirgrfF6JRezyaii55dkuX7Nb8yq+1uEyEoG/Eu+EfMOzmr5oCMeKiaqVzaSdX5dWvtM1JwusyPTI94NSBgrIxmIwlXuBddQRwEEbHjh8xIRdWIjoqaEEhpC4DSTESUJCxzm6GCdtm3Zx2mER26LfJq1qGevTWnlGWu1QFpdeSco8ptlX9oEdsC/sI/vBfh/rlWgPVcsOzXaq5aE58fLa8q8TVR7NMTaPVENrjtHAfV2roNpDzahTOKm+8/z1wfKD8nRyk71j+1T/W9Zjn+kEfuen836Jl3fJXfn7pNrS5/V0BT51OCFeEu5mSFDWTfJXPZW0cyvr53Fa91Cret84hdImXtIqVf/tiCSUYO4wNtxDatwa6HEUO/kE6gvW9Zz+l7p9EXNAhqd4iDJWdIbhObb196pTpDmw1ioeE2/28c/wCPN4ovzpDpX+vzH9YPV2sXSveHfpTmF2IbtNY7iOG7hFN2YGs1jAIiq6wlfYwxvjk9Ezvhnf060juUxzFf8MY/8P7RHsDQ== Ru!v AAAEMXiclVNNTxNBGH7K+gH4AeiFxEtjg/HUTA0I8YSikYsGCoWaQja722mZdPYjs9sibno33o0HE40mHoxHf4IX/4CH3ow3wxETLx58d3aFaEuJs9nOM887z/O+827HDqQII8Z6uRHj1OkzZ0fHxs+dv3BxYnLq0kbot5XDK44vfVW1rZBL4fFKJCLJq4HilmtLvmm3lpL4ZoerUPjeerQX8G3XanqiIRwrIqr2yIzb+S1p5Ttdc7LAikyPfD8oZaCAbKz4U7mn2EIdPhy04YLDQ0RYwkJITw0lMATEbSMmThESOs7RxThp27SL0w6L2Bb9NmlVy1iP1olnqNUOZZH0KlLmMcO+sPfsgH1mH9h39utYr1h7JLXs0WynWh6YE8+m136eqHJpjrBzpBpac4QGFnStgmoPNJOcwkn1nScvDtZulWfia+wt26f637Ae+0Qn8Do/nHervPyS3BN/j1S7+ryursCjDsfEK8LdDAnKukP+SU8V7dzN+nmcVh5qk943/kNpE69olar/dEQRinHnMDbcQ2ncGuhxFDv5BMkXrOs5/S91+yLmgAwPcBtlrOsMw3M81t+rTpHmwFqruEe82cc/xF0s4X7iT3eo9O+N6QcbN4qlm8W51dnC4nJ2m0ZxBVdxnW7MPBaxjBVUdIXP8QqvjY9Gz/hqfEu3juQyzWX8NYz93/EC7A4= Yu v AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEMniclVO7bhNBFD3O8gjhkQQaJJoIKyg01hjxElUgINKAEidOLMXRanc9dkbenV3Nrh3Cyj3iA6Cg4CFRIFr+gIYfoAgdokLQBYmGgjuzSyKw44hZrefMuXPOvXPX40a+iBPGtgsj1qHDR46OHhs7fuLkqfGJydMrcdhRHq96oR+qmuvE3BeSVxOR+LwWKe4Ers9X3facjq92uYpFKJeTrYivB05LiqbwnISoel05sm2nnd5M96I9UWQlZsZUPyjnoIh8LISThUeoo4EQHjoIwCGREPbhIKZnDWUwRMStIyVOERImztHDGGk7tIvTDofYNv22aLWWs5LW2jM2ao+y+PQqUk5hmn1kb9gO+8Desq/s175eqfHQtWzR7GZaHtnjj88u/TxQFdCcYGNPNbTmBE1cN7UKqj0yjD6Fl+m7D5/uLN2oTKcX2Cv2jep/ybbZezqB7P7wXi/yyjNy1/6SVJvmvIGpQFKHU+IV4V6OBGXdIH/dU0U7N/N+7qf1d7W6983/ULrEK1pl6j8dUYRS3NqNDfdQBrcHeuzFDj6B/oINM2f/pV5fxB6Q4R5uooJlk2F4jgfmezUo0hpYaw13iLf7+Pu4jTnc1f50h8r/3ph+sHKpVL5aurJ4uTg7n9+mUZzDeczQjbmGWcxjAVWqI8ITPMcL6531yfpsfcm2jhRyzRn8NazvvwFXWuyv ranku(v) AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVM7bxNBGBzneITwSAINEk2EFURlrSNeogoERBpQ7MSJpRBZd+e1c/K9tHd2CCf3CFqiFFQgUUT8DBo6KhApKRFlkGgomFsficCOI/Z03tn5dub79juvFbpOFAuxmxsxjh0/cXL01NjpM2fPjU9Mnl+OgrayZcUO3EBVLTOSruPLSuzErqyGSpqe5coVqzWXxlc6UkVO4C/Fm6Fc88ym7zQc24xJlWZqE3lREHpM9YNiBvLIxkIwmXuGx6gjgI02PEj4iIldmIj4rKIIgZDcGhJyisjRcYkuxqhtc5fkDpNsi79NrlYz1uc69Yy02mYWl6+icgrT4pPYEXvig3gnvolfh3ol2iOtZZOz1dPKsDb+/OLizyNVHucY6weqoTXHaOCWrtVh7aFm0lPYPX3n6fbe4u3ydHJFvBHfWf9rsSve8wR+54f9tiTLr+ie+vtUbejzeroCnx1OyCviboYcZl2nf9pTxZ0bWT8P07r72rT3jf9QWuQVVz31n44oogR392PDPZTGrYEeB7GjT5B+wbqee/+lbl+kNiDDQ9xBGUs6w/AcT/T3qjPSHFhrFffJ1/r4R7iHOTxI/XmHiv/emH6wPFMo3ihcL13Lz85nt2kUl3AZV3ljbmIW81hAhXVIvMBLbBk7xkfjs/Glt3Ukl2ku4K9hfP0N9XPn2A== 2 AAAEJ3iclVM7bxNBGBzneITwSAINEk2EFURlrXkkiCoQEGlAsRMnlkJk3Z3Xzsn30t7ZIZzcI2hBFFQgUUT8DBo6KhApKRFlkGgomFsficCOI/Z03tn5dub79juvFbpOFAuxkxsxjhw9dnz0xNjJU6fPjE9Mnl2JgrayZcUO3EBVLTOSruPLSuzErqyGSpqe5cpVqzWfxlc7UkVO4C/HW6Fc98ym7zQc24xJla7WJvKiIPSY6gfFDOSRjcVgMvcED1FHABtteJDwERO7MBHxWUMRAiG5dSTkFJGj4xJdjFHb5i7JHSbZFn+bXK1lrM916hlptc0sLl9F5RSmxSexLXbFB/FOfBO/DvRKtEdayxZnq6eVYW386fmln4eqPM4xNvZVQ2uO0cANXavD2kPNpKewe/rO45e7SzfL08kl8UZ8Z/2vxY54zxP4nR/225Isv6J76u9TtanP6+kKfHY4Ia+IuxlymHWD/mlPFXduZv08SOvuadPeN/5DaZFXXPXUfzqiiBLc3osN91AatwZ67McOP0H6Bet67v2Xun2R2oAM93ELZSzrDMNzPNLfq85Ic2CtVdwlX+vjH+AO5nEv9ecdKv57Y/rBypVCcaZwvXQtP7eQ3aZRXMBFXOaNmcUcFrCICuuQeIbneGFsGx+Nz8aX3taRXKY5h7+G8fU3+Tbn2Q== 3 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEMXiclVNNTxNBGH7K+gH4AeiFxEtjg/HUTA0I8YSikYsKhUKTQja722mZdL8yuy3ipnfj3Xgw0WjiwXj0J3jxD3jozXgzHDHx4sF3ZleItpQ4m+0887zzPO8773bs0BVRzFgvN2KcOn3m7OjY+LnzFy5OTE5d2oiCtnR4xQncQFZtK+Ku8HklFrHLq6Hklme7fNNuLan4ZofLSAT+erwX8m3PavqiIRwrJqr2yEza+S1p5Ttdc7LAikyPfD8oZaCAbKwEU7mn2EIdARy04YHDR0zYhYWInhpKYAiJ20ZCnCQkdJyji3HStmkXpx0WsS36bdKqlrE+rZVnpNUOZXHplaTMY4Z9Ye/ZAfvMPrDv7NexXon2ULXs0WynWh6aE8+m136eqPJojrFzpBpac4wGFnStgmoPNaNO4aT6zpMXB2u3yjPJNfaW7VP9b1iPfaIT+J0fzrtVXn5J7srfJ9WuPq+nK/CpwwnxknA3Q4Ky7pC/6qmknbtZP4/Tuoda1fvGfyht4iWtUvWfjkhCCe4cxoZ7SI1bAz2OYiefQH3Bup7T/1K3L2IOyPAAt1HGus4wPMdj/b3qFGkOrLWKe8SbffxD3MUS7it/ukOlf29MP9i4USzdLM6tzhYWl7PbNIoruIrrdGPmsYhlrKCiK3yOV3htfDR6xlfjW7p1JJdpLuOvYez/BuGq7Ao= Ou!v AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEMXiclVNNTxNBGH7K+gH4AeiFxEtjg/HUTA0I8YSikYsGCoWaQja722mZdL8yuy3ipnfj3Xgw0WjiwXj0J3jxD3jozXgzHDHx4sF3ZleItpQ4m+0887zzPO8773bs0BVRzFgvN2KcOn3m7OjY+LnzFy5OTE5d2oiCtnR4xQncQFZtK+Ku8HklFrHLq6Hklme7fNNuLan4ZofLSAT+erwX8m3PavqiIRwrJqr2yEza+S1p5Ttdc7LAikyPfD8oZaCAbKwEU7mn2EIdARy04YHDR0zYhYWInhpKYAiJ20ZCnCQkdJyji3HStmkXpx0WsS36bdKqlrE+rZVnpNUOZXHplaTMY4Z9Ye/ZAfvMPrDv7NexXon2ULXs0WynWh6aE8+m136eqPJojrFzpBpac4wGFnStgmoPNaNO4aT6zpMXB2u3yjPJNfaW7VP9b1iPfaIT+J0fzrtVXn5J7srfJ9WuPq+nK/CpwwnxknA3Q4Ky7pC/6qmknbtZP4/Tuoda1fvGfyht4iWtUvWfjkhCCe4cxoZ7SI1bAz2OYiefQH3Bup7T/1K3L2IOyPAAt1HGus4wPMdj/b3qFGkOrLWKe8SbffxD3MUS7it/ukOlf29MP9i4USzdLM6tzhYWl7PbNIoruIrrdGPmsYhlrKCiK3yOV3htfDR6xlfjW7p1JJdpLuOvYez/Bge77BQ= Yu!v AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAEMXiclVPPTxNBGH1lVRB/AHox8dLYYDw1U6NiPCFo5KKBQqFJIZvd7bRsOvsjs9sCbno33o0HEowmHoxH/wQv/gMeejPeDEdMvHjwm9kVoi0lzGY7b943733ffNuxQ+FGMWO93Ihx5uy50bHz4xcuXro8MTl1ZTUK2tLhFScQgazaVsSF6/NK7MaCV0PJLc8WfM1uzav4WofLyA38lXgn5Bue1fTdhutYMVG1spm08+vCyne65mSBFZke+X5QykAB2VgMpnIvsI46AjhowwOHj5iwgIWInhpKYAiJ20BCnCTk6jhHF+OkbdMuTjssYlv026RVLWN9WivPSKsdyiLolaTMY5p9ZR/YAfvCPrIf7PexXon2ULXs0GynWh6aEy+vLf86UeXRHGPzSDW05hgN3Ne1ulR7qBl1CifVd56/Plh+UJ5ObrJ3bJ/qf8t67DOdwO/8dN4v8fIuuSt/n1Rb+ryersCnDifES8LdDLmUdZP8VU8l7dzK+nmcVhxqVe8bp1DaxEtapeq/HZGEEswdxoZ7SI1bAz2OYiefQH3Bup7T/1K3L2IOyPAUD1HGis4wPMe2/l51ijQH1lrFY+LNPv4ZHmEeT5Q/3aHS/zemH6zeLpbuFe8u3SnMLmS3aQzXcQO36MbMYBYLWERFV/gKe3hjfDJ6xjfje7p1JJdpruKfYez/AdZn7Ac= Ru v AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEMXiclVNNTxNBGH7K+gH4AeiFxEtjg/HUTA0I8YSikYsKhUKTQja722nZdPYjs9sibno33o0HE40mHoxHf4IX/4CH3ow3wxETLx58Z3aFaEuJs9nOM887z/O+827HDoUbxYz1ciPGqdNnzo6OjZ87f+HixOTUpY0oaEuHV5xABLJqWxEXrs8rsRsLXg0ltzxb8E27taTimx0uIzfw1+O9kG97VtN3G65jxUTVHplJO78lrHyna04WWJHpke8HpQwUkI2VYCr3FFuoI4CDNjxw+IgJC1iI6KmhBIaQuG0kxElCro5zdDFO2jbt4rTDIrZFv01a1TLWp7XyjLTaoSyCXknKPGbYF/aeHbDP7AP7zn4d65VoD1XLHs12quWhOfFseu3niSqP5hg7R6qhNcdoYEHX6lLtoWbUKZxU33ny4mDtVnkmucbesn2q/w3rsU90Ar/zw3m3yssvyV35+6Ta1ef1dAU+dTghXhLuZsilrDvkr3oqaedu1s/jtOJQq3rf+A+lTbykVar+0xFJKMGdw9hwD6lxa6DHUezkE6gvWNdz+l/q9kXMARke4DbKWNcZhud4rL9XnSLNgbVWcY94s49/iLtYwn3lT3eo9O+N6QcbN4qlm8W51dnC4nJ2m0ZxBVdxnW7MPBaxjBVUdIXP8QqvjY9Gz/hqfEu3juQyzWX8NYz938sA7AQ= Ou v AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEFAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB57NpBUlT9Szn3r3v3vu++5yzQteJYiF2ciPGseMnRsdOjp86febsxOTUuZUoaCtbVuzADVTVMiPpOr6sxE7symqopOlZrly1WvNJfLUjVeQE/nK8Fcp1z2z6TsOxzZhUSdQm86Ig9JjuB8UM5JGNxWAq9xSPUEcAG214kPARE7swEfFZQxECIbl1dMkpIkfHJXoYp7bNXZI7TLIt/ja5WstYn+vEM9Jqm1lcvorKacyIz2Jb7IqP4r34Ln4f6NXVHkktW5ytVCvD2sSzC0u/DlV5nGNs7KuG1hyjgZu6Voe1h5pJTmGn+s6TV7tLt8oz3cvirfjB+t+IHfGBJ/A7P+13JVl+TffE36dqU5/X0xX47HCXvCLuZchh1g36Jz1V3LmZ9fMgrbunTXrfOILSIq+4StV/O6KIurizFxvuoTRuDfTYjx1+guQL1vWc/pd6fZHagAwPcBtlLOsMw3M81t+rzkhzYK1V3CNf6+Mf4i7mcT/x5x0q/n9j+sHK1ULxRuF66Vp+biG7TWO4iEu4whszizksYBEV1iHxHC/w0tg2PhlfjK/p1pFcpjmPf4bx7Q/t7efW 0 AAAErniclVPLbtNAFL0uBkp5NC0bpLKwiIpSCUUTqACxCqSo3YDatElT1ZFlO5PYil+MJynF8hYhfoAFK5BYID6DDT/Aop+AYFckNiy4HpsWcJqIsZw5c+6cc+9cZ4zAsUNOyIE0dUo+febs9LmZ8xcuXpotzM03Q3/ATNowfcdnLUMPqWN7tMFt7tBWwKjuGg7dNvq1JL49pCy0fW+L7we07eo9z+7aps6R0grPVVfnFnOjldpqXL2lOrTLSyrTvb4WDeLScOmGUkeEjDKMU6yoTrJQmd2z+JKievSJMtll5w+XnZyLViiSMhFDyYNKBoqQjXV/TnoBKnTABxMG4AIFDzhiB3QI8dmFChAIkGtDhBxDZIs4hRhmUDvAXRR36Mj28beHq92M9XCdeIZCbWIWB1+GSgUWyWfynhyST+QD+UJ+nugVCY+kln2cjVRLA2325ZXNHxNVLs4crGPV2Jo5dOGuqNXG2gPBJKcwU/3w2avDzXv1xeg6eUu+Yv1vyAH5iCfwht/Ndxu0/hrdE38PVXvivK6owMMOR8gzxHGGbMxqoX/SU4Y797J+nqR1jrRJ77v/oTSQZ7hK1b87whBF8OAoNt6DCdwf6XEcm3yC5At2xJz+l+JcRBuR4RHchzpsiQzjczwV36uDkd7IWlvwEHktxz+GFajBauKPd6jy743Jg+bNcuV2eXljuVhdy27TNCzANSjhjbkDVViDdWhgHd+keWlBuioTuSm3ZS3dOiVlmsvw15CtX4FxFRM= DCG@3 (ranku(v), Ru!v, Ru v) 6= DCG@3 (ranku(v), Yu!v, Yu v) AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1 AAAEJ3iclVO/b9NAGH2pgZbyoy0sSCwVURFTdEZAEVOhILqAmrRpI5Uqsp1LasW/dHZSSpQdwQpiYAKJoeLPYGFjAtGRETEWiYWB54tpBUlT9Szn3r3v3vu++5yzI8+NEyF2ciPGseMnRsdOjp86febsxOTUuZU4bClHlp3QC1XFtmLpuYEsJ27iyUqkpOXbnly1m/NpfLUtVeyGwXKyFcl132oEbt11rIRU0axO5kVB6DHdD8wM5JGNxXAq9xSPUEMIBy34kAiQEHuwEPNZgwmBiNw6OuQUkavjEl2MU9viLskdFtkmfxtcrWVswHXqGWu1wyweX0XlNGbEZ7EtdsVH8V58F78P9Opoj7SWLc52Tyuj6sSzC0u/DlX5nBNs7KuG1pygjpu6Vpe1R5pJT+H09O0nr3aXbpVmOpfFW/GD9b8RO+IDTxC0fzrvirL0mu6pf0DVpj6vrysI2OEOeUXczZDLrBv0T3uquHMz6+dBWm9Pm/a+fgSlTV5x1VP/7Ygi6uDOXmy4h9K4OdBjP3b4CdIvWNNz77/U7YtUB2R4gNsoYVlnGJ7jsf5eNUYaA2ut4B75ah//EHcxj/upP++Q+f+N6QcrVwvmjcL14rX83EJ2m8ZwEZdwhTdmFnNYwCLKrEPiOV7gpbFtfDK+GF97W0dymeY8/hnGtz/xsOfX 1

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice • Consider a surrogate metric • It holds How a naive estimator fails 11 It indicates we should consider both side’s propensity scores! AAAF1niclVPNT9RAFH+7uor4AagHEy8TCQQikCnxKyQmKBC5qLB8raHYTLuz3YZuW2ZnF7ApN6Px5M2DJ008GONf4cV/wAN/gvGIiRcPvk4Lu1A+4jTtvPm99/u9N286ZuA6dUnpdi5/6nThzNmOc53nL1y81NXdc3mx7jeExRcs3/VFyWR17joeX5COdHkpEJzVTJcvmasTsX+pyUXd8b15uRnwlRqzPafiWEwiZPTkvvbrNSarphlORUb41AgbRBeMNKMhkizceBHpLq/IZTL6PHzWComIbpV92UKTWDJMNIxw7KpcIbrnS2YTXe/s19carEzukwENI3RZ5ZK1tAbJzQymkh6MVjkS9UFMXGwrZy8X2U2WfLPKGUGiN7wyF6ZgFg/3y2IBN0mxFToYYaPsgWJ7q7ba/RiuqSYMxYjJ3ZCvjeGhWNJp8mG+EQzbzPGitp7EheuB8APpE0IOaLdLG929dISqQbKGlhq9kI4Zvyf3GnQogw8WNKAGHDyQaLvAoI7PMmhAIUBsBULEBFqO8nOIoBO5DYziGMEQXcWvjavlFPVwHWvWFdvCLC6+ApkE+ugP+pnu0O/0C/1J/x6pFSqNuJZNnM2EywOj6821uT8nsmo4S6i2WMfWLKEC91StDtYeKCTehZXwmy/e7cyNFfvCfvqR/sL6P9Bt+g134DV/W59mefE9qsf6HrLW1X5rqgIPOxwiLtCOUsvBrFXUj3sqMHI97edRXHePG/e+8h9ME3GBq4S92xGBVggP93zHawhlrx6q0fKdvIP4BMtqTv6lKOMxDsnwGB5AEeZVhuNzbKjzKqPHPrTWEkwhbmTwJzAJE/Ao1sc7pB28MVljcXREuzNye/ZW7/h0eps64DrcgAG8MXdhHKZhBhbAyl/Nj+Un8pOFUmGr8LLwKgnN51LOFdg3Cm//AdSueQ8= E Ou!v,Ou v ⇥ 2Yu!v · 2Yu v 1 ⇤ = (1 ✓ u!v ) + ✓ u!v (1 ✓ u v ) 2Ru!v + ✓ u!v ✓ u v 2Ru!v(1+Ru v) | {z } g(Ru!v, Ru v)+1 1, 6/ g(R u!v , R u v ) a plug-in estimator of AAAErXiclVNLb9NAEB4XA6U8msAF0cuKqCgRallXvMSpUCq4gNq0SVPVrWU7m8SKX1o/SnF9REL8AQ6cQOKA+Blc+AMc+hNQj0XiwoHx2rSCpKlYy9mZb+b7ZnacNXzbCkJK96SxU/LpM2fHz02cv3Dx0mSpfLkZeBE3WcP0bI+3DD1gtuWyRmiFNmv5nOmOYbM1o7+QxddixgPLc1fDHZ9tOnrXtTqWqYcIaaVXaofrZqKkiWqzTrhLGkTlVrcX7qZqEDlaEhHVckkjJbkbC7epRQjYWKatV1Wuu31MTKtxrUZUs+2FRIhV57aSdaHAdRKn5CZZFwJ25qVkhihFrZpWqtBZKhYZNJTCqECxlryy9BpUaIMHJkTgAAMXQrRt0CHAZwMUoOAjtgkJYhwtS8QZpDCB3AizGGboiPbxt4veRoG66GeagWCbWMXGlyOTwDT9Rj/RA/qVfqbf6a9jtRKhkfWyg7uRc5mvTb65uvLzRJaDewi9I9bInkPowH3Rq4W9+wLJTmHm/Pjl24OVB/Xp5Ab9QPex//d0j37BE7jxD/PjMqu/Q/VM30XWtjivIzpwccIJ4hzttLAsrNpD/WymHDO3i3kex7UPudnsO//BNBDn6OXsPxPhaCXw6DA2WoMLuz9U4yh28gmyL9gWe/5fSgci2pAKz+Ah1GFVVBhd44X4Xm2MdIf22oJFxLUB/Dk8hgV4kunjHVL+vTGDRnNuVrk7e2f5dmV+sbhN4zAF16GKN+YezMNTWIIG9rEvlaVr0pR8S27IqryVp45JBecK/LXk7m/6CBRb 1 |U| X u2U X v2Vu (ranku(v)) · 2Yu!v+Yv v 1 AAAEQ3iclVPLThRBFD1D+0B8MMjGhE3HCQQTM6kxvsIKRYIbDQwMTAKk091TM3SmX6nuGcTOLNwZf8CFK0yMMW70G4yJP+CCTzDsgMSNJt6qbiE6wxCq0133nlvn3Fu3uqzQdaKYsZ3cgHbm7LnzgxeGLl66fGU4P3J1OQpawuYVO3ADUbXMiLuOzyuxE7u8GgpuepbLV6zmjIyvtLmInMBfirdCvu6ZDd+pO7YZE2TkxxqTZSNp6WvC1Nudm3rquNK5YeQLrMjU0LuNUmYUkI35YCT3EmuoIYCNFjxw+IjJdmEiomcVJTCEhK0jIUyQ5ag4RwdDxG3RKk4rTEKb9G2Qt5qhPvlSM1Jsm7K49Api6hhn39kHts++sY/sB/t1rFaiNGQtWzRbKZeHxvCra4s/T2R5NMfYOGL1rTlGHfdVrQ7VHipE7sJO+e3nr/cXp8rjyQR7y3ap/m22w77QDvz2gf1ugZffkLrU94m1qfbrqQp86nBCuCC7k1kOZd0gfdlTQSs3s34ex3UPubL39VMwLcIFeSn7b0cEWQkeHsb6awhlN3tqHMVO3oE8wZqa03+p0xUxemR4ggcoY0ll6J/jmTqvGkUaPWutYpZwowt/ikeYwZzUpztU+v/GdBvLt4qlu8U7C7cL07PZbRrEGK5jkm7MPUzjMeZRoTpe4D0+4bP2VdvV9rSDdOlALuOM4p+h/f4DVVzx3A== g(Ru!v, Ru v)

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Proposed Unbiased Estimator • We propose an unbiased estimator: 12 AAALFXicrVZLj9tUFD5pCy3m0U7ZILG5MA8laju9GVUFISFVKkhlQTSdMp2i8TSynZvEqh+p46TTuv4D8AO6YAWCBWIBrJBArEDAjhUS/QmIZZHYdNFzju3EsZO4i9pK7vW55/Gdc757bXPg2MNQyoe1I0ePPff88RMvaC++9PIrJ0+tnL4+9EeBpXYt3/GDG6YxVI7tqd3QDh11YxAowzUdtWfeukzre2MVDG3f+yi8O1AHrtHz7K5tGSGK2iu1tX470l0j7Adu9MH2XhzXP25HI6EHdq8fGkHg3xHj+KxIhI7qTmQNsfGu0LuBYUXNONLDvgqNsqXILeStY36qb92M5oQrLs4YinOimag30kHonu+NXFMFQte1Df32yOiIM9XYYrEURC5Q+9Sq3JR8ifKkmU5WIb22/ZVjDujQAR8sGIELCjwIce6AAUO896EJEgYoO4AIZQHObF5XEIOGtiPUUqhhoPQW/vfwaT+VevhMPodsbWEUB38BWgpYl3/Jr+Uj+Yf8Rv4jHy/0FbEPwnIXRzOxVYP2yU9eu/Z/pZWLYwj9qdVSzCF04W3GaiP2AUsoCyuxH9978OjaOzvr0Yb8Qv6L+D+XD+UvmIE3/s/66qra+Yy9B2yj4A5n7DIGD2sccaVjHjNcAc4illI113GlhzIDY/e5zkNGSZoR3ufxiaQhIqMo51M7fWG8AfeM8MTcywOWUve66KUOqygjxDbG7aOkkfojjRBHqtNt9BKgh7y3qjzHKLdmYlLvfDjkuclM6EyqIGCNkaxN6iBaWutv+aH8Ehnyu/xW/oDjdxy1w9h17hDND3OoyEfEyAxcJ+xZTAmb2FniMdV/jTOzOG+Szc5DZBrlX2c8WYWehcfGJMNpHuOlmYyZCQbvSp+5QisxR3N5j/ZZb56PTLeKIyO07nCvtrhXZHUzjb7I71bOb5YJsXbA1rQepZV7Y4Y3U1uyfI9Z4qSrO4jM5L0ZwuU5jCrifPrqJJbZahH/orokWSvebwafe35hH2X8NfGO4H2OnO2t/dLeOsi4vSQm1cFBCeHtpKdsOV6iFXG8vG4SsaxVHbeHXsbpOTEfx/2KahU9uGm38gzx+PQnKyfHk4w55XOxxdiTvp1F/HnuLMJhp3HL3mzcsfTr4K/IXYtPVAf3qcl73UqrmT8t9Qmnpz3VK/Fso7bP760yom1+Fy63b7HFiFE5+ERv6XgO+1qVngLOg87zKafqE3m2OxoVXmiv9lhyWMDhpz2k3ERBj6poMyfoPHy6CHaOS8siJHqlCPgt1Cx++ZQn17c2mxc3L1y9sHrpSvpVdAJehzexMk14Cy7BFezSLli1T2vf136s/aQ90H7WftV+S1SP1FKbV2Hm0v58AmmKc1A= hIPW(Yu!v, Yu v) = 1 ✓u!v✓u v 2Yu!v 2Yu v 1 + 1 ✓u!v 2Yu!v 1 Aaa Theorem AAALBnicrVZLj9tUFD5pS2kNtB0QEhIbwzyUSNX0ZlQVVKlSpYJUFkTTKZkpGg/R9SOJNX6kju1O8XiP+AMsWFHBAhUBG9QtEgv4A0j0JyCWRWLDgnOO7cRj59EFthLfe+55fOec715bHzn2OBTiaePU6TMvnH3x3HnlpZdfuXDx0sqru2M/Cgyra/iOH9zT5dhybM/qhnboWPdGgSVd3bH29MNbtL4XW8HY9r2Pwocj68CVA8/u24YMUdRbaby+oQ1lmGiuDIeGdJKdNO1ls8BNPtjeS1PNsfphsxAF0jtEjShtxq3L6sc40gJ7MAxlEPgP1DjNZCobFbJMo6V5figHqqYpG9r9SJrq9Ruq1g+kkbTThA2O1a6aKR+n2jhy2ZPtqd1UzaYxT3d7EQoczNKUM4G1VM0w/VAdVlJpZtiWAm71Lq2KTcGXWh+088Eq5Ne2v3LGAQ1M8MGACFywwIMQxw5IGOO9D20QMELZASQoC3Bk87oFKShoG6GWhRoSpYf4P8DZfi71cE4+x2xtYBQHfwFaqrAufhffimfiN/FY/Cn+nesrYR+E5SE+9czWGvUufv7G3X+WWrn4DGE4tVqIOYQ+vMtYbcQ+YgllYWT28adfPLt7fWc92RCPxF+I/yvxVPyCGXjx38Y3d6ydL9l7wDYWPOCMXcbgYY0TrnTKzwJXgKOEpVTNdVwZoExi7CHXecwoSTPB+wrOSBoiMopyJbfT5sYbcc8IT8q9PGApda+PXpqwijJCbGPcIUpauT/SCPFJdbqPXgL0UPa2LM8Y5caJmNQ7H454rDMTzEkVVFhjJGuTOqgdpfOH+FB8jQz5VfwgnuDzR45qMnaNO0TjoxIq8pEwMonrhL2IKWATO0s8pvqvcWYG502yk+MQmUb5NxlPUaH/w2NrkuE0j3hhJjEzQfKu9JkrtJJyNJf36JD1ZvkodJdxJEJrk3u1xb0iq0/y6PP8bpX8FpkQa0dsTetJXrm3TvBmakuW7zFLnHx1B5HpvDdDuDWDUVWcz1+dzLJYreKfV5csa4v3m+Rzz6/so4K/Ot4JvM+Ri721X9tbBwW3F8SkOjgoIbxmfsrW42VaCccr62YR61rL4w7QS5yfE7NxHC+pVtWDm3erzBCPT3+ycko8KZhTPxc7jD3r22XEX+bOPBx2HrfuzcYdSz8Tf1XuGnyiOrhPdd7rRl7N8mmpTTg97am2FM82avv83qoj2uZ34WL7DltEjMrBGb2l0xns6yz1FHAedJ5POdWcyIvd0VrihfbqgCVHFRx+3kPKTa3oURVt5gSdh88XwS5xaVGETK8WAb+F2tUvn/pgd2uzfW3z6p2rqzdv519F5+BNeBsr04Z34Cbcxi51wWgcNx41Hje+Uz5Tvld+Up5kqqcauc1rcOJSfv4PVVNuWg== ˆ RIPW (ranku(v), Yu!v, Yu v) := 1 |U| X u2U X v2Vu (ranku(v)) · hIPW(Yu!v, Yu v) AAAE9XiclVPLbtNAFL0uAUp4NIUNEhuLqFEiVdGEt5CQCqWiLIA0bZpUcWTZziSx4kcYTxKK5T3iB1ggFiBAQiz4CDZsWLLoJyCWRWLDguux27RNmoqxnDn3cc69c53Ru5bpcUK2pKljieMnTk6fSp4+c/bcTGr2/Lrn9phBy4Zruayqax61TIeWucktWu0yqtm6RSt6ZzGMV/qUeabrrPHNLq3bWssxm6ahcXSpqe8ZxdZ4W9flJdV/ovo9hWlyP5iXBbZCHCgWbfKa0ta4L5INzfJLQaBGFrP9h8VKEGSR6XSQFWT7uXl5A5G8IxYZQi2nMLPV5nVZcVyutRQlKYcrozztaQ35jryngqh7QLa0p8XSUDUSzampNMkTseRRUIhBGuJVdGelF6BAA1wwoAc2UHCAI7ZAAw+fGhSAQBd9dfDRxxCZIk4hgCRye5hFMUNDbwd/W2jVYq+DdqjpCbaBVSx8GTJlmCM/yCeyTb6Rz+Qn+Xuoli80wl42cdcjLu2qMy8vrv45kmXjzqE9ZE3smUMTboleTey9KzzhKYyI33/+anv1dmnOz5B35Bf2/5Zska94Aqf/2/iwQkuvUT3Ud5A1EOe1RQcOTthHP0McxMjEqm3UD2fKMHMQz/MwrrXLDWff/A+mjn6GVsTemQhD5MO93dhkDSZwZ6zGMHb0CcIv2BB79F8KRiLqmAqP4C6UYE1UmFzjmfheDYy0xvZahSX0qyP+x3AfFuFBqI93qHDwxoyC9Sv5wo389ZVr6YXl+DZNwyW4DFm8MTdhAZahCGUwpKvShqRLRmKQeJN4n/gYpU5JMecC7FuJL/8A6fswnw== E Ou!v,Ou v h ˆ RIPW(rank u (v), Y u!v , Y u v ) i = R (rank u (v), R u!v , R u v ) is an unbiased estimator if the gain function is a linear or exponential function. AAAESHiclVNNb9NAEH2JC5Ty0RQuSFwioiJO0QbxJU6FgigHUJo2TaSmsmxnk1jxl9ablGL5ihB/gAMnkDgguMIf4IK4c+hPQFyQilQhcWC8Nq0gaSrWsnfmzb43s7NeM3DsUDK2nctrU0eOHps+PnPi5KnTs4W5M2uhPxAWr1u+44umaYTcsT1el7Z0eDMQ3HBNhzfM/mISbwy5CG3fW5VbAd9wja5nd2zLkATphWKrZ8io5RqyZxlOVItjPfWEG92vNsgtlFiZqVEcNSqZUUI2qv5c7ilaaMOHhQFccHiQZDswENKzjgoYAsI2EBEmyLJVnCPGDHEHtIrTCoPQPn275K1nqEd+ohkqtkVZHHoFMYuYZ1/YG7bDPrG37Cv7daBWpDSSWrZoNlMuD/TZZ+dWdg9luTRL9PZZE2uW6OCGqtWm2gOFJLuwUv7w8fOdlZu1+egie8W+Uf0v2Tb7SDvwhj+s18u89oLUE32PWJtqv66qwKMOR4QLsuPMsilrj/STngpauZn18yCus8dNet/5D6ZJuCAvZf/piCArwu292GQNoez+WI392OE7SE6wreb0X4pHIvqYDA9wCzWsqgyTczxS59WmSHdsrU3cJVwfwR/iDhZxL9GnO1T598aMGmuXy5Vr5avLV0oLS9ltmsZ5XMAlujHXsYAlVFGnOp7gHd7jg/ZZ+67taj/TpflcxjmLv8ZU/jc2h/VD ˆ RIPW where Inverse Propensity Score

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Experimental Setting • Dataset • Social networking data with ground truth relevances collected by [Su WWW’22] • Manually added exposure labels and implicit feedback labels • Model • Embedding-based model • Compare three types of model : Naive, one-sided IPW, two-sided IPW (Proposed Method) • Evaluation • Evaluate on the test data DCG@K 13

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Experiment Result 14 • bold: best score, underlined: worst score • methods: • Baseline : naive estimator • IPW1 : one-sided IPW • IPW2 : Proposed estimator • Result • Proposed method (IPW2) gives the best result overall • Conventional IPW method (IPW1) gives the worst result

KDD 2023 Workshop - Causal Inference and Machine Learning in Practice Summary • Contributions • Indicated that conventional IPW methods fail under the situation of two-sided markets • Proposed an unbiased estimator, which works in two-sided settings • Verified it outperforms existing methods through the experiments • Future Work • Create large dataset • There are few publicly available data on two-sided markets 15 Thank you for listening!