Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Rankings have become the primary interface of many two-sided markets. Many have noted that the rankings not only affect the satisfaction of the users (e.g., customers, listeners, employers, travelers), but that the position in the ranking allocates exposure – and thus economic opportunity – to the ranked items (e.g., articles, products, songs, job seekers, restaurants, hotels). This has raised questions of fairness to the items, and most existing works have addressed fairness by explicitly linking item exposure to item relevance. However, we argue that any particular choice of such a link function may be difficult to defend, and we show that the resulting rankings can still be unfair. To avoid these shortcomings, we develop a new axiomatic approach that is rooted in principles of fair division. This not only avoids the need to choose a link function, but also more meaningfully quantifies the impact on the items beyond exposure. Our axioms of envy-freeness and dominance over uniform ranking postulate that in fair rankings every item should prefer their own rank allocation over that of any other item, and that no item should be actively disadvantaged by the rankings. To compute ranking policies that are fair according to these axioms, we propose a new ranking objective related to the Nash Social Welfare. We show that the solution has guarantees regarding its envy-freeness, its dominance over uniform rankings for every item, and its Pareto optimality. In contrast, we show that conventional exposure-based fairness can produce large amounts of envy and have a highly disparate impact on the items. Beyond these theoretical results, we illustrate empirically how our framework controls the trade-off between impact-based individual item fairness and user utility.

Transcript

1. Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD2022) Yuta Saito (w/ Thorsten Joachims)
2. Outline • Existing “Exposure-based” Fairness ◦ intro to fair-ranking and the exposure-based framework ◦ remaining issues/unfairness in the exposure-based framework • Our “Impact-based” Axioms and Fair Ranking Method ◦ redefine fairness in ranking via some “impact-based” axioms ◦ also describe a simple and efficient method to satisfy our axioms SIGIR’18 KDD’18 Seminal Papers
3. Ranking in Online Platform / Marketplaces There are many use cases of ranking in real-world applications -> typically, rankings optimize only the user satisfaction Amazon Booking.com ICML2021
4. Fairness Considerations in Ranking • Utility to Users: Travelers want to find hotels relevant to their query (location, budget) quickly • Impact on Items: Hotels earn money from the ranking Ex1: Hotels Search Booking.com
5. • Utility to Users: Conference attendees want to find accepted papers relevant to their expertise • Impact on Items: Authors of the accepted papers want to get as much attention as possible Fairness Considerations in Ranking Ex2: Academic Conference ICML2021
6. • Typically, ranking systems maximize the user utility by ordering items by “relevance” • but it often produces unfair rankings wrt “exposure” • How should we define and ensure fairness in ranking? High-Level Motivation of “Fair Ranking” Position Items Relevance Exposure 1 hotel1 0.95 1 2 hotel2 0.94 0.5 ... ... ... ... 100 hotel100 0.90 0.01 A ranking of items shown to a user very critical in Airbnb/YouTube/Amazon/Spotify/Booking.com/Linkedin, etc..
7. Basic Setup and Notation Position Items Relevance Exposure k=1 i1 r(u,i1) e(1)=1.0 k=2 i2 r(u,i2) e(2)=0.5 ... ... ... ... A ranking of items/products shown to a user : items : users : exposure probability of the “k”-th position (known) : relevance (known)
8. We formulate Ranking as Resource Allocation users (u) positions (k) u1 1 2 3 u2 1 2 3 items resources to allocate to the items (higher positions are more valuable)
9. Allocation of Positions as Valuable Resources users (u) positions (k) u1 1 2 3 u2 1 2 3 we want to optimize this “ranking rule” Items
10. “Doubly” Stochastic Matrix k=1 k=2 k=3 i1 0.6 0.3 0.1 1 i2 0.3 0.3 0.4 1 i3 0.1 0.4 0.5 1 1 1 1 For each user “u”, we obtain a doubly stochastic matrix like the following
11. Utility to the users (typical performance metric in ranking) Utility to the users under ranking “X” click probability based on position-based model (can be conversion, dwell time, DCG, etc..) Typical ranking aims at maximizing the user-utility total number of clicks given by ranking "X" stochastic ranking an ideal ranking under the typical formulation
12. Impact on the items Impact the ranking "X" has on each item “i” We also want to fairly distribute impact to the items total clicks each item click probability based on position-based model (can be conversion, dwell time, DCG, etc..) total number of clicks given to each item under ranking "X"
13. Well-Established Framework Let’s allocate exposure proportional to item merit • Expo-Fair Ranking -- Maximizes the user utility under the “exposure-fairness” constraint SIGIR’18 KDD’18 definition of fairness in ranking
14. Merit and Exposure in the Exposure-Fair Framework • Merit -- global/average popularity of item “i” in the entire platform independent of the ranking • Exposure -- total exposure that is allocated to item “i” under ranking “X” dependent on the ranking
15. Well-Established Framework Let’s allocate exposure proportional to item merit • Expo-Fair Ranking -- Maximizes the user utility under the “exposure-fairness” constraint SIGIR’18 KDD’18 items having similar merit should get similar exposure
16. Toy Example: Setup • only two users (“u_1” and “u_2”) and items (“i_1” and “i_2”) • exposure prob of position 1 is “1” while that of position 2 is “0” (it is meaningless for each item to be presented at position 2) : exposure probability of the “k”-th position
17. Toy Example: True Relevance Table relevance (r(u,i)) 0.8 0.3 0.5 0.4 1.3 0.7 For each user, which item should be ranked at the top position in a ranking with what probability? more popular less popular
18. Max Ranking (Conventional) relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 Allocates all exposure to the first item (because it is more relevant)
19. Very unfair to the items.. relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 1.3 0.0
20. Expo-Fair Ranking relevance/exposure 0.8 / ??? 0.3 / ??? 0.5 / ??? 0.4 / ??? 1.3 0.7 1.3 0.7 allocate exposure proportional to item merit so to be fair to the items
21. Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / ??? 0.5 / 0.3 0.4 / ??? 1.3 0.7 1.3 0.7 Then, let’s maximize the utility under the exposure-fairness constraint
22. Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 Satisfy the exposure fairness constraint (the “perfectly fair ranking”)
23. “Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
24. “Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 0.95 0.28 Do you think this perfectly fair ranking is really fair?
25. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 The “baseline” ranking we can achieve without any optimization
26. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35
27. The expo-fair even fails to dominate the uniform baseline the first item receives a better impact than under the uniform ranking at the cost of impact on the second item... Exposure-Fair Ranking Uniform Random Baseline the exposure-fairness produces a large and obvious disparate impact Exposure-Fair (“perfectly fair”) Uniform Baseline Impact on “Item 1” 0.95 (+46%) 0.65 Impact on “Item 2” 0.28 (-20%) 0.35
28. Let’s swap the ranking of the two items.. relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
29. Impact on the second item increases relevance/exposure 0.8 / 0.0 0.3 / 1.0 0.5 / 0.7 0.4 / 0.3 1.3 0.7 0.7 1.3 0.35 (↓) 0.42 (↑)
30. The expo-fair ranking is also inefficient, producing “envy” the first item prefers the current ranking to that of the second item the second item prefers the ranking of the first item rather than its own (i_2 envies i_1) Exposure-Fair (“perfectly fair”) Swap each other’s ranking Impact on “Item 1” 0.95 0.35 (-63%) Impact on “Item 2” 0.28 0.42 (+50%) the exposure-fairness allows to produce some envy among items
31. Remaining “Unfairness” in the Exposure-Fair Framework It seems that we should “redefine” fairness in ranking.. suggested by our toy example We’ve seen that the perfectly fair ranking in the exposure-fair framework • produces some “envy” among items • fails to guarantee the number of clicks under uniform ranking
32. Our Impact-based Axioms for Fair Ranking 1. envy-freeness -- there should not be any item who envies some other items in the platform (seems very difficult to achieve and guarantee..) 2. dominance over uniform ranking -- every item should get better impact than under the uniform ranking
33. Need for a New Fair Ranking Algorithm We’ve seen that the perfectly fair ranking in the exposure-fair framework • produces some “envy” among items (violates envy-freeness) • fails to guarantee the number of clicks under uniform ranking (violates dominance over uniform ranking) suggested by our toy example How can we guarantee our impact-based axioms in a computationally efficient way? Our algorithmic question:
34. The “Fair Division” Problem in Classical Game Theory Fair Division: How should we fairly divide/allocate fruits? Apple Banana Orange Father 0.8 0.1 0.1 Mother 0.2 0.0 0.9 Child 0.3 0.0 0.8 agents (divisible) fruits Preference Table
35. How about maximizing the “total family welfare”? Standard Approach would be to maximize the total family welfare (seems difficult to come up with another way) where is utility of each family member fruit allocation
36. How about maximizing the “total family welfare”? Maximizing the sum can lead to an “abusive” allocation, giving nothing to the child Allocation “X” Apple Banana Orange Utility Father 1.0 1.0 0.0 0.9 Mother 0.0 0.0 1.0 0.9 Child 0.0 0.0 0.0 0.0 sum is maximized indeed
37. Great Idea in Fair Division: “Nash Social Welfare” Alternative approach is to maximize Nash Social Welfare The NSW doesn’t allow to have a zero utility agent, possibly producing a more equitable allocation In contrast, maximizng the sum allows zero utility agents.. Actually, maximizing the NSW guarantees envy-freeness and dominance over uniform allocation in the fruit allocation problem
38. Great Idea in Fair Division: “Nash Social Welfare” Allocation “X” Apple Banana Orange Utility Father 1.0 1.0 0.0 0.90 Mother 0.0 0.0 0.5 0.45 Child 0.0 0.0 0.5 0.40 product is now maximized Alternative approach is to maximize Nash Social Welfare
39. Fair Division to Fair Ranking The “equitable” NSW objective The “unfair” sum objective “Default Objective” in Ranking (Even the Exposure-Fair maximizes this one) ？？？ total clicks each item
40. Nash Social Welfare Ranking (our main proposal) Then, let’s maximize the “product” of impact on the items guaranteed to dominate the uniform ranking and to be envy-free (at least approximately)
41. Nash Social Welfare Ranking (our main proposal) Then, let’s maximize the “product” of impact on item convex program, efficiently solvable
42. Toy Example: True Relevance Table relevance (r(u,i)) 0.8 0.3 0.5 0.4 1.3 0.7 more popular less popular
43. Expo-Fair Ranking (Existing) relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
44. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5 0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35
45. NSW Ranking (Our Proposal) relevance/exposure 0.8 / 1.0 0.3/ 0.0 0.5 / 0.0 0.4 / 1.0 1.3 0.7 1.0 1.0 0.80 0.40
46. There actually is no envy in the platform relevance/exposure 0.8 / 0.0 0.3 / 1.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 1.0 1.0 0.5 (↓) 0.3 (↓)
47. Summary of the Toy Example User Utility Imp_i1 Imp_i2 Max 1.30 1.30 0.00 Expo-Fair 1.23 0.95 0.28 NSW (ours) 1.20 0.80 0.40 Uniform 1.00 0.65 0.35 dominate the uniform (& envy-free)
48. Synthetic Experiment Setup , evaluate the fairness-utility trade-off • Mean-Max Envy (smaller is better) • 10% Better-Off Pct (larger is better) • 10% Worse-Off Pct (smaller is better) Mean of maximum envy for each item has Prop of items who improve their utility over 10% from unif Prop of items who degrade their utility over 10% from unif
49. Experiment Results Storonger popularity bias means there is a fewer number of very popular items, producing a more difficult situation to pursue fairness-utility at the same time
50. Experiment Results • NSW is the most equitable, especially in the presence of popularity bias • Max and Exposure-Based produce some envy and disparate impact • However, there is a difficult tradeoff between fairness and utility especially when popularity bias is strong ours existing conventional
51. A Simple Extension: Controlling Utility-Fairness Trade-off alpha-NSW Allocation (\alpha=zero leads to the original NSW) A larger “alpha” leads to a higher utility, while a smaller value prioritizes fairness of impact
52. Weighted Envy-Freeness and Dominance 1. weighted envy-freeness 2. weighted dominance over the uniform random allocation
53. Steerable Trade-off via the “alpha-NSW” objective • compare several different values of \alpha (0.0, 0.5, 1.0, 2.0) • we can choose a variety of fairness-utility trade-off via tuning \alpha • In particular, 1-NSW achieves the user utility similar to Exposure-based while leading to a much more equitable impact distribution
54. Overall Summary • Current exposure-based fair ranking produces envy and fails to ensure the impact guaranteed under the uniform random baseline • We redefine fairness in ranking via envy-freeness and dominance over uniform ranking to resolve the remaining unfairness • By maximizing the product of the impacts or the NSW, we guarantee envy-freeness and dominance over uniform ranking in an efficient way • We also extend the NSW to have a steerable fairness-utility trade-off
55. More About The Paper • Contact: ys552@cornell.edu • arXiv: https://arxiv.org/abs/2202.06317 • Experiment code: https://github.com/usaito/kdd2022-fair-ranking-nsw Thank you!