Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Fair Ranking as Fair Division: Impact-Based Ind...

usaito
July 04, 2022

Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Rankings have become the primary interface of many two-sided markets. Many have noted that the rankings not only affect the satisfaction of the users (e.g., customers, listeners, employers, travelers), but that the position in the ranking allocates exposure – and thus economic opportunity – to the ranked items (e.g., articles, products, songs, job seekers, restaurants, hotels). This has raised questions of fairness to the items, and most existing works have addressed fairness by explicitly linking item exposure to item relevance. However, we argue that any particular choice of such a link function may be difficult to defend, and we show that the resulting rankings can still be unfair. To avoid these shortcomings, we develop a new axiomatic approach that is rooted in principles of fair division. This not only avoids the need to choose a link function, but also more meaningfully quantifies the impact on the items beyond exposure. Our axioms of envy-freeness and dominance over uniform ranking postulate that in fair rankings every item should prefer their own rank allocation over that of any other item, and that no item should be actively disadvantaged by the rankings. To compute ranking policies that are fair according to these axioms, we propose a new ranking objective related to the Nash Social Welfare. We show that the solution has guarantees regarding its envy-freeness, its dominance over uniform rankings for every item, and its Pareto optimality. In contrast, we show that conventional exposure-based fairness can produce large amounts of envy and have a highly disparate impact on the items. Beyond these theoretical results, we illustrate empirically how our framework controls the trade-off between impact-based individual item fairness and user utility.

usaito

July 04, 2022
Tweet

More Decks by usaito

Other Decks in Research

Transcript

  1. Outline • Existing “Exposure-based” Fairness ◦ intro to fair-ranking and

    the exposure-based framework ◦ remaining issues/unfairness in the exposure-based framework • Our “Impact-based” Axioms and Fair Ranking Method ◦ redefine fairness in ranking via some “impact-based” axioms ◦ also describe a simple and efficient method to satisfy our axioms SIGIR’18 KDD’18 Seminal Papers
  2. Ranking in Online Platform / Marketplaces There are many use

    cases of ranking in real-world applications -> typically, rankings optimize only the user satisfaction Amazon Booking.com ICML2021
  3. Fairness Considerations in Ranking • Utility to Users: Travelers want

    to find hotels relevant to their query (location, budget) quickly • Impact on Items: Hotels earn money from the ranking Ex1: Hotels Search Booking.com
  4. • Utility to Users: Conference attendees want to find accepted

    papers relevant to their expertise • Impact on Items: Authors of the accepted papers want to get as much attention as possible Fairness Considerations in Ranking Ex2: Academic Conference ICML2021
  5. • Typically, ranking systems maximize the user utility by ordering

    items by “relevance” • but it often produces unfair rankings wrt “exposure” • How should we define and ensure fairness in ranking? High-Level Motivation of “Fair Ranking” Position Items Relevance Exposure 1 hotel1 0.95 1 2 hotel2 0.94 0.5 ... ... ... ... 100 hotel100 0.90 0.01 A ranking of items shown to a user very critical in Airbnb/YouTube/Amazon/Spotify/Booking.com/Linkedin, etc..
  6. Basic Setup and Notation Position Items Relevance Exposure k=1 i1

    r(u,i1) e(1)=1.0 k=2 i2 r(u,i2) e(2)=0.5 ... ... ... ... A ranking of items/products shown to a user : items : users : exposure probability of the “k”-th position (known) : relevance (known)
  7. We formulate Ranking as Resource Allocation users (u) positions (k)

    u1 1 2 3 u2 1 2 3 items resources to allocate to the items (higher positions are more valuable)
  8. Allocation of Positions as Valuable Resources users (u) positions (k)

    u1 1 2 3 u2 1 2 3 we want to optimize this “ranking rule” Items
  9. “Doubly” Stochastic Matrix k=1 k=2 k=3 i1 0.6 0.3 0.1

    1 i2 0.3 0.3 0.4 1 i3 0.1 0.4 0.5 1 1 1 1 For each user “u”, we obtain a doubly stochastic matrix like the following
  10. Utility to the users (typical performance metric in ranking) Utility

    to the users under ranking “X” click probability based on position-based model (can be conversion, dwell time, DCG, etc..) Typical ranking aims at maximizing the user-utility total number of clicks given by ranking "X" stochastic ranking an ideal ranking under the typical formulation
  11. Impact on the items Impact the ranking "X" has on

    each item “i” We also want to fairly distribute impact to the items total clicks each item click probability based on position-based model (can be conversion, dwell time, DCG, etc..) total number of clicks given to each item under ranking "X"
  12. Well-Established Framework Let’s allocate exposure proportional to item merit •

    Expo-Fair Ranking -- Maximizes the user utility under the “exposure-fairness” constraint SIGIR’18 KDD’18 definition of fairness in ranking
  13. Merit and Exposure in the Exposure-Fair Framework • Merit --

    global/average popularity of item “i” in the entire platform independent of the ranking • Exposure -- total exposure that is allocated to item “i” under ranking “X” dependent on the ranking
  14. Well-Established Framework Let’s allocate exposure proportional to item merit •

    Expo-Fair Ranking -- Maximizes the user utility under the “exposure-fairness” constraint SIGIR’18 KDD’18 items having similar merit should get similar exposure
  15. Toy Example: Setup • only two users (“u_1” and “u_2”)

    and items (“i_1” and “i_2”) • exposure prob of position 1 is “1” while that of position 2 is “0” (it is meaningless for each item to be presented at position 2) : exposure probability of the “k”-th position
  16. Toy Example: True Relevance Table relevance (r(u,i)) 0.8 0.3 0.5

    0.4 1.3 0.7 For each user, which item should be ranked at the top position in a ranking with what probability? more popular less popular
  17. Max Ranking (Conventional) relevance/exposure 0.8 / 1.0 0.3 / 0.0

    0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 Allocates all exposure to the first item (because it is more relevant)
  18. Very unfair to the items.. relevance/exposure 0.8 / 1.0 0.3

    / 0.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 2.0 0.0 1.3 0.0
  19. Expo-Fair Ranking relevance/exposure 0.8 / ??? 0.3 / ??? 0.5

    / ??? 0.4 / ??? 1.3 0.7 1.3 0.7 allocate exposure proportional to item merit so to be fair to the items
  20. Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / ??? 0.5

    / 0.3 0.4 / ??? 1.3 0.7 1.3 0.7 Then, let’s maximize the utility under the exposure-fairness constraint
  21. Expo-Fair Ranking relevance/exposure 0.8 / 1.0 0.3 / 0.0 0.5

    / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 Satisfy the exposure fairness constraint (the “perfectly fair ranking”)
  22. “Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0

    0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
  23. “Perfectly Fair Ranking” relevance/exposure 0.8 / 1.0 0.3 / 0.0

    0.5 / 0.3 0.4 / 0.7 0.95 0.28 Do you think this perfectly fair ranking is really fair?
  24. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5

    0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 The “baseline” ranking we can achieve without any optimization
  25. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5

    0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35
  26. The expo-fair even fails to dominate the uniform baseline the

    first item receives a better impact than under the uniform ranking at the cost of impact on the second item... Exposure-Fair Ranking Uniform Random Baseline the exposure-fairness produces a large and obvious disparate impact Exposure-Fair (“perfectly fair”) Uniform Baseline Impact on “Item 1” 0.95 (+46%) 0.65 Impact on “Item 2” 0.28 (-20%) 0.35
  27. Let’s swap the ranking of the two items.. relevance/exposure 0.8

    / 1.0 0.3 / 0.0 0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
  28. Impact on the second item increases relevance/exposure 0.8 / 0.0

    0.3 / 1.0 0.5 / 0.7 0.4 / 0.3 1.3 0.7 0.7 1.3 0.35 (↓) 0.42 (↑)
  29. The expo-fair ranking is also inefficient, producing “envy” the first

    item prefers the current ranking to that of the second item the second item prefers the ranking of the first item rather than its own (i_2 envies i_1) Exposure-Fair (“perfectly fair”) Swap each other’s ranking Impact on “Item 1” 0.95 0.35 (-63%) Impact on “Item 2” 0.28 0.42 (+50%) the exposure-fairness allows to produce some envy among items
  30. Remaining “Unfairness” in the Exposure-Fair Framework It seems that we

    should “redefine” fairness in ranking.. suggested by our toy example We’ve seen that the perfectly fair ranking in the exposure-fair framework • produces some “envy” among items • fails to guarantee the number of clicks under uniform ranking
  31. Our Impact-based Axioms for Fair Ranking 1. envy-freeness -- there

    should not be any item who envies some other items in the platform (seems very difficult to achieve and guarantee..) 2. dominance over uniform ranking -- every item should get better impact than under the uniform ranking
  32. Need for a New Fair Ranking Algorithm We’ve seen that

    the perfectly fair ranking in the exposure-fair framework • produces some “envy” among items (violates envy-freeness) • fails to guarantee the number of clicks under uniform ranking (violates dominance over uniform ranking) suggested by our toy example How can we guarantee our impact-based axioms in a computationally efficient way? Our algorithmic question:
  33. The “Fair Division” Problem in Classical Game Theory Fair Division:

    How should we fairly divide/allocate fruits? Apple Banana Orange Father 0.8 0.1 0.1 Mother 0.2 0.0 0.9 Child 0.3 0.0 0.8 agents (divisible) fruits Preference Table
  34. How about maximizing the “total family welfare”? Standard Approach would

    be to maximize the total family welfare (seems difficult to come up with another way) where is utility of each family member fruit allocation
  35. How about maximizing the “total family welfare”? Maximizing the sum

    can lead to an “abusive” allocation, giving nothing to the child Allocation “X” Apple Banana Orange Utility Father 1.0 1.0 0.0 0.9 Mother 0.0 0.0 1.0 0.9 Child 0.0 0.0 0.0 0.0 sum is maximized indeed
  36. Great Idea in Fair Division: “Nash Social Welfare” Alternative approach

    is to maximize Nash Social Welfare The NSW doesn’t allow to have a zero utility agent, possibly producing a more equitable allocation In contrast, maximizng the sum allows zero utility agents.. Actually, maximizing the NSW guarantees envy-freeness and dominance over uniform allocation in the fruit allocation problem
  37. Great Idea in Fair Division: “Nash Social Welfare” Allocation “X”

    Apple Banana Orange Utility Father 1.0 1.0 0.0 0.90 Mother 0.0 0.0 0.5 0.45 Child 0.0 0.0 0.5 0.40 product is now maximized Alternative approach is to maximize Nash Social Welfare
  38. Fair Division to Fair Ranking The “equitable” NSW objective The

    “unfair” sum objective “Default Objective” in Ranking (Even the Exposure-Fair maximizes this one) ??? total clicks each item
  39. Nash Social Welfare Ranking (our main proposal) Then, let’s maximize

    the “product” of impact on the items guaranteed to dominate the uniform ranking and to be envy-free (at least approximately)
  40. Nash Social Welfare Ranking (our main proposal) Then, let’s maximize

    the “product” of impact on item convex program, efficiently solvable
  41. Expo-Fair Ranking (Existing) relevance/exposure 0.8 / 1.0 0.3 / 0.0

    0.5 / 0.3 0.4 / 0.7 1.3 0.7 1.3 0.7 0.95 0.28
  42. Uniform Random Ranking relevance/exposure 0.8 / 0.5 0.3 / 0.5

    0.5 / 0.5 0.4 / 0.5 1.3 0.7 1.0 1.0 0.65 0.35
  43. NSW Ranking (Our Proposal) relevance/exposure 0.8 / 1.0 0.3/ 0.0

    0.5 / 0.0 0.4 / 1.0 1.3 0.7 1.0 1.0 0.80 0.40
  44. There actually is no envy in the platform relevance/exposure 0.8

    / 0.0 0.3 / 1.0 0.5 / 1.0 0.4 / 0.0 1.3 0.7 1.0 1.0 0.5 (↓) 0.3 (↓)
  45. Summary of the Toy Example User Utility Imp_i1 Imp_i2 Max

    1.30 1.30 0.00 Expo-Fair 1.23 0.95 0.28 NSW (ours) 1.20 0.80 0.40 Uniform 1.00 0.65 0.35 dominate the uniform (& envy-free)
  46. Synthetic Experiment Setup , evaluate the fairness-utility trade-off • Mean-Max

    Envy (smaller is better) • 10% Better-Off Pct (larger is better) • 10% Worse-Off Pct (smaller is better) Mean of maximum envy for each item has Prop of items who improve their utility over 10% from unif Prop of items who degrade their utility over 10% from unif
  47. Experiment Results Storonger popularity bias means there is a fewer

    number of very popular items, producing a more difficult situation to pursue fairness-utility at the same time
  48. Experiment Results • NSW is the most equitable, especially in

    the presence of popularity bias • Max and Exposure-Based produce some envy and disparate impact • However, there is a difficult tradeoff between fairness and utility especially when popularity bias is strong ours existing conventional
  49. A Simple Extension: Controlling Utility-Fairness Trade-off alpha-NSW Allocation (\alpha=zero leads

    to the original NSW) A larger “alpha” leads to a higher utility, while a smaller value prioritizes fairness of impact
  50. Steerable Trade-off via the “alpha-NSW” objective • compare several different

    values of \alpha (0.0, 0.5, 1.0, 2.0) • we can choose a variety of fairness-utility trade-off via tuning \alpha • In particular, 1-NSW achieves the user utility similar to Exposure-based while leading to a much more equitable impact distribution
  51. Overall Summary • Current exposure-based fair ranking produces envy and

    fails to ensure the impact guaranteed under the uniform random baseline • We redefine fairness in ranking via envy-freeness and dominance over uniform ranking to resolve the remaining unfairness • By maximizing the product of the impacts or the NSW, we guarantee envy-freeness and dominance over uniform ranking in an efficient way • We also extend the NSW to have a steerable fairness-utility trade-off
  52. More About The Paper • Contact: [email protected] • arXiv: https://arxiv.org/abs/2202.06317

    • Experiment code: https://github.com/usaito/kdd2022-fair-ranking-nsw Thank you!