Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Slide 1

Slide 1 text

Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD2022) Yuta Saito (w/ Thorsten Joachims)

Slide 2

Slide 2 text

Outline ● Existing “Exposure-based” Fairness ○ intro to fair-ranking and the exposure-based framework ○ remaining issues/unfairness in the exposure-based framework ● Our “Impact-based” Axioms and Fair Ranking Method ○ redefine fairness in ranking via some “impact-based” axioms ○ also describe a simple and efficient method to satisfy our axioms SIGIR’18 KDD’18 Seminal Papers

Slide 3

Slide 3 text

Ranking in Online Platform / Marketplaces There are many use cases of ranking in real-world applications -> typically, rankings optimize only the user satisfaction Amazon Booking.com ICML2021

Slide 4

Slide 4 text

Fairness Considerations in Ranking ● Utility to Users: Travelers want to find hotels relevant to their query (location, budget) quickly ● Impact on Items: Hotels earn money from the ranking Ex1: Hotels Search Booking.com

Slide 5

Slide 5 text

● Utility to Users: Conference attendees want to find accepted papers relevant to their expertise ● Impact on Items: Authors of the accepted papers want to get as much attention as possible Fairness Considerations in Ranking Ex2: Academic Conference ICML2021

Slide 6

Slide 6 text

● Typically, ranking systems maximize the user utility by ordering items by “relevance” ● but it often produces unfair rankings wrt “exposure” ● How should we define and ensure fairness in ranking? High-Level Motivation of “Fair Ranking” Position Items Relevance Exposure 1 hotel1 0.95 1 2 hotel2 0.94 0.5 ... ... ... ... 100 hotel100 0.90 0.01 A ranking of items shown to a user very critical in Airbnb/YouTube/Amazon/Spotify/Booking.com/Linkedin, etc..

Slide 7

Slide 7 text

Basic Setup and Notation Position Items Relevance Exposure k=1 i1 r(u,i1) e(1)=1.0 k=2 i2 r(u,i2) e(2)=0.5 ... ... ... ... A ranking of items/products shown to a user : items : users : exposure probability of the “k”-th position (known) : relevance (known)

Slide 8

Slide 8 text

We formulate Ranking as Resource Allocation users (u) positions (k) u1 1 2 3 u2 1 2 3 items resources to allocate to the items (higher positions are more valuable)

Slide 9

Slide 9 text

Allocation of Positions as Valuable Resources users (u) positions (k) u1 1 2 3 u2 1 2 3 we want to optimize this “ranking rule” Items

Slide 10

Slide 10 text

“Doubly” Stochastic Matrix k=1 k=2 k=3 i1 0.6 0.3 0.1 1 i2 0.3 0.3 0.4 1 i3 0.1 0.4 0.5 1 1 1 1 For each user “u”, we obtain a doubly stochastic matrix like the following

Slide 11

Slide 11 text

Utility to the users (typical performance metric in ranking) Utility to the users under ranking “X” click probability based on position-based model (can be conversion, dwell time, DCG, etc..) Typical ranking aims at maximizing the user-utility total number of clicks given by ranking "X" stochastic ranking an ideal ranking under the typical formulation

Slide 12

Slide 12 text

Impact on the items Impact the ranking "X" has on each item “i” We also want to fairly distribute impact to the items total clicks each item click probability based on position-based model (can be conversion, dwell time, DCG, etc..) total number of clicks given to each item under ranking "X"

Slide 13

Slide 13 text

Well-Established Framework Let’s allocate exposure proportional to item merit ● Expo-Fair Ranking -- Maximizes the user utility under the “exposure-fairness” constraint SIGIR’18 KDD’18 definition of fairness in ranking

Slide 14

Slide 14 text

Merit and Exposure in the Exposure-Fair Framework ● Merit -- global/average popularity of item “i” in the entire platform independent of the ranking ● Exposure -- total exposure that is allocated to item “i” under ranking “X” dependent on the ranking

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Toy Example: Setup ● only two users (“u_1” and “u_2”) and items (“i_1” and “i_2”) ● exposure prob of position 1 is “1” while that of position 2 is “0” (it is meaningless for each item to be presented at position 2) : exposure probability of the “k”-th position

Slide 17

Slide 17 text

Toy Example: True Relevance Table relevance (r(u,i)) 0.8 0.3 0.5 0.4 1.3 0.7 For each user, which item should be ranked at the top position in a ranking with what probability? more popular less popular

Slide 18

Slide 18 text

Max Ranking (Conventional) relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 Allocates all exposure to the first item (because it is more relevant)

Slide 19

Slide 19 text

Very unfair to the items.. relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 1.3 0.0

Slide 20

Slide 20 text

Expo-Fair Ranking relevance/exposure 0.8 / ??? 0.3 / ??? 0.5 / ??? 0.4 / ??? 1.3 0.7 1.3 0.7 allocate exposure proportional to item merit so to be fair to the items

Slide 21

Slide 21 text

Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / ??? 0.5 / 0.3 0.4 / ??? 1.3 0.7 1.3 0.7 Then, let’s maximize the utility under the exposure-fairness constraint

Slide 22

Slide 22 text

Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 Satisfy the exposure fairness constraint (the “perfectly fair ranking”)

Slide 23

Slide 23 text

“Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28

Slide 24

Slide 24 text

“Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 0.95 0.28 Do you think this perfectly fair ranking is really fair?

Slide 25

Slide 25 text

Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 The “baseline” ranking we can achieve without any optimization

Slide 26

Slide 26 text

Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35

Slide 27

Slide 27 text

The expo-fair even fails to dominate the uniform baseline the first item receives a better impact than under the uniform ranking at the cost of impact on the second item... Exposure-Fair Ranking Uniform Random Baseline the exposure-fairness produces a large and obvious disparate impact Exposure-Fair (“perfectly fair”) Uniform Baseline Impact on “Item 1” 0.95 (+46%) 0.65 Impact on “Item 2” 0.28 (-20%) 0.35

Slide 28

Slide 28 text

Let’s swap the ranking of the two items.. relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28

Slide 29

Slide 29 text

Impact on the second item increases relevance/exposure 0.8 / 0.0 0.3 / 1.0 0.5 / 0.7 0.4 / 0.3 1.3 0.7 0.7 1.3 0.35 (↓) 0.42 (↑)

Slide 30

Slide 30 text

The expo-fair ranking is also inefficient, producing “envy” the first item prefers the current ranking to that of the second item the second item prefers the ranking of the first item rather than its own (i_2 envies i_1) Exposure-Fair (“perfectly fair”) Swap each other’s ranking Impact on “Item 1” 0.95 0.35 (-63%) Impact on “Item 2” 0.28 0.42 (+50%) the exposure-fairness allows to produce some envy among items

Slide 31

Slide 31 text

Remaining “Unfairness” in the Exposure-Fair Framework It seems that we should “redefine” fairness in ranking.. suggested by our toy example We’ve seen that the perfectly fair ranking in the exposure-fair framework ● produces some “envy” among items ● fails to guarantee the number of clicks under uniform ranking

Slide 32

Slide 32 text

Our Impact-based Axioms for Fair Ranking 1. envy-freeness -- there should not be any item who envies some other items in the platform (seems very difficult to achieve and guarantee..) 2. dominance over uniform ranking -- every item should get better impact than under the uniform ranking

Slide 33

Slide 33 text

Need for a New Fair Ranking Algorithm We’ve seen that the perfectly fair ranking in the exposure-fair framework ● produces some “envy” among items (violates envy-freeness) ● fails to guarantee the number of clicks under uniform ranking (violates dominance over uniform ranking) suggested by our toy example How can we guarantee our impact-based axioms in a computationally efficient way? Our algorithmic question:

Slide 34

Slide 34 text

The “Fair Division” Problem in Classical Game Theory Fair Division: How should we fairly divide/allocate fruits? Apple Banana Orange Father 0.8 0.1 0.1 Mother 0.2 0.0 0.9 Child 0.3 0.0 0.8 agents (divisible) fruits Preference Table

Slide 35

Slide 35 text

How about maximizing the “total family welfare”? Standard Approach would be to maximize the total family welfare (seems difficult to come up with another way) where is utility of each family member fruit allocation

Slide 36

Slide 36 text

How about maximizing the “total family welfare”? Maximizing the sum can lead to an “abusive” allocation, giving nothing to the child Allocation “X” Apple Banana Orange Utility Father 1.0 1.0 0.0 0.9 Mother 0.0 0.0 1.0 0.9 Child 0.0 0.0 0.0 0.0 sum is maximized indeed

Slide 37

Slide 37 text

Great Idea in Fair Division: “Nash Social Welfare” Alternative approach is to maximize Nash Social Welfare The NSW doesn’t allow to have a zero utility agent, possibly producing a more equitable allocation In contrast, maximizng the sum allows zero utility agents.. Actually, maximizing the NSW guarantees envy-freeness and dominance over uniform allocation in the fruit allocation problem

Slide 38

Slide 38 text

Great Idea in Fair Division: “Nash Social Welfare” Allocation “X” Apple Banana Orange Utility Father 1.0 1.0 0.0 0.90 Mother 0.0 0.0 0.5 0.45 Child 0.0 0.0 0.5 0.40 product is now maximized Alternative approach is to maximize Nash Social Welfare

Slide 39

Slide 39 text

Fair Division to Fair Ranking The “equitable” NSW objective The “unfair” sum objective “Default Objective” in Ranking (Even the Exposure-Fair maximizes this one) ？？？ total clicks each item

Slide 40

Slide 40 text

Nash Social Welfare Ranking (our main proposal) Then, let’s maximize the “product” of impact on the items guaranteed to dominate the uniform ranking and to be envy-free (at least approximately)

Slide 41

Slide 41 text

Nash Social Welfare Ranking (our main proposal) Then, let’s maximize the “product” of impact on item convex program, efficiently solvable

Slide 42

Slide 42 text

Toy Example: True Relevance Table relevance (r(u,i)) 0.8 0.3 0.5 0.4 1.3 0.7 more popular less popular

Slide 43

Slide 43 text

Expo-Fair Ranking (Existing) relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28

Slide 44

Slide 44 text

Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35

Slide 45

Slide 45 text

NSW Ranking (Our Proposal) relevance/exposure 0.8 / 1.0 0.3/ 0.0 0.5 / 0.0 0.4 / 1.0 1.3 0.7 1.0 1.0 0.80 0.40

Slide 46

Slide 46 text

There actually is no envy in the platform relevance/exposure 0.8 / 0.0 0.3 / 1.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 1.0 1.0 0.5 (↓) 0.3 (↓)

Slide 47

Slide 47 text

Summary of the Toy Example User Utility Imp_i1 Imp_i2 Max 1.30 1.30 0.00 Expo-Fair 1.23 0.95 0.28 NSW (ours) 1.20 0.80 0.40 Uniform 1.00 0.65 0.35 dominate the uniform (& envy-free)

Slide 48

Slide 48 text

Synthetic Experiment Setup , evaluate the fairness-utility trade-off ● Mean-Max Envy (smaller is better) ● 10% Better-Off Pct (larger is better) ● 10% Worse-Off Pct (smaller is better) Mean of maximum envy for each item has Prop of items who improve their utility over 10% from unif Prop of items who degrade their utility over 10% from unif

Slide 49

Slide 49 text

Experiment Results Storonger popularity bias means there is a fewer number of very popular items, producing a more difficult situation to pursue fairness-utility at the same time

Slide 50

Slide 50 text

Experiment Results ● NSW is the most equitable, especially in the presence of popularity bias ● Max and Exposure-Based produce some envy and disparate impact ● However, there is a difficult tradeoff between fairness and utility especially when popularity bias is strong ours existing conventional

Slide 51

Slide 51 text

A Simple Extension: Controlling Utility-Fairness Trade-off alpha-NSW Allocation (\alpha=zero leads to the original NSW) A larger “alpha” leads to a higher utility, while a smaller value prioritizes fairness of impact

Slide 52

Slide 52 text

Weighted Envy-Freeness and Dominance 1. weighted envy-freeness 2. weighted dominance over the uniform random allocation

Slide 53

Slide 53 text

Steerable Trade-off via the “alpha-NSW” objective ● compare several different values of \alpha (0.0, 0.5, 1.0, 2.0) ● we can choose a variety of fairness-utility trade-off via tuning \alpha ● In particular, 1-NSW achieves the user utility similar to Exposure-based while leading to a much more equitable impact distribution

Slide 54

Slide 54 text

Overall Summary ● Current exposure-based fair ranking produces envy and fails to ensure the impact guaranteed under the uniform random baseline ● We redefine fairness in ranking via envy-freeness and dominance over uniform ranking to resolve the remaining unfairness ● By maximizing the product of the impacts or the NSW, we guarantee envy-freeness and dominance over uniform ranking in an efficient way ● We also extend the NSW to have a steerable fairness-utility trade-off

Slide 55

Slide 55 text

More About The Paper ● Contact: ys552@cornell.edu ● arXiv: https://arxiv.org/abs/2202.06317 ● Experiment code: https://github.com/usaito/kdd2022-fair-ranking-nsw Thank you!