Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Rankings have become the primary interface of many two-sided markets. Many have noted that the rankings not only affect the satisfaction of the users (e.g., customers, listeners, employers, travelers), but that the position in the ranking allocates exposure – and thus economic opportunity – to the ranked items (e.g., articles, products, songs, job seekers, restaurants, hotels). This has raised questions of fairness to the items, and most existing works have addressed fairness by explicitly linking item exposure to item relevance. However, we argue that any particular choice of such a link function may be difficult to defend, and we show that the resulting rankings can still be unfair. To avoid these shortcomings, we develop a new axiomatic approach that is rooted in principles of fair division. This not only avoids the need to choose a link function, but also more meaningfully quantifies the impact on the items beyond exposure. Our axioms of envy-freeness and dominance over uniform ranking postulate that in fair rankings every item should prefer their own rank allocation over that of any other item, and that no item should be actively disadvantaged by the rankings. To compute ranking policies that are fair according to these axioms, we propose a new ranking objective related to the Nash Social Welfare. We show that the solution has guarantees regarding its envy-freeness, its dominance over uniform rankings for every item, and its Pareto optimality. In contrast, we show that conventional exposure-based fairness can produce large amounts of envy and have a highly disparate impact on the items. Beyond these theoretical results, we illustrate empirically how our framework controls the trade-off between impact-based individual item fairness and user utility.

Transcript

1. Fair Ranking as Fair Division:
   Impact-Based Individual Fairness in Ranking
   (KDD2022)
   Yuta Saito (w/ Thorsten Joachims)
   

   View Slide

2. Outline
   ● Existing “Exposure-based” Fairness
   ○ intro to fair-ranking and the exposure-based framework
   ○ remaining issues/unfairness in the exposure-based framework
   ● Our “Impact-based” Axioms and Fair Ranking Method
   ○ redefine fairness in ranking via some “impact-based” axioms
   ○ also describe a simple and efficient method to satisfy our axioms
   SIGIR’18
   KDD’18
   Seminal Papers
   

   View Slide

3. Ranking in Online Platform / Marketplaces
   There are many use cases of ranking in real-world applications
   -> typically, rankings optimize only the user satisfaction
   Amazon Booking.com ICML2021
   

   View Slide

4. Fairness Considerations in Ranking
   ● Utility to Users:
   Travelers want to find hotels relevant to
   their query (location, budget) quickly
   ● Impact on Items:
   Hotels earn money from the ranking
   Ex1: Hotels Search
   Booking.com
   

   View Slide

5. ● Utility to Users:
   Conference attendees want to find
   accepted papers relevant to their
   expertise
   ● Impact on Items:
   Authors of the accepted papers want to
   get as much attention as possible
   Fairness Considerations in Ranking
   Ex2: Academic Conference ICML2021
   

   View Slide

6. ● Typically, ranking systems
   maximize the user utility by
   ordering items by “relevance”
   ● but it often produces unfair
   rankings wrt “exposure”
   ● How should we define and
   ensure fairness in ranking?
   High-Level Motivation of “Fair Ranking”
   Position Items Relevance Exposure
   1 hotel1 0.95 1
   2 hotel2 0.94 0.5
   ... ... ... ...
   100 hotel100 0.90 0.01
   A ranking of items shown to a user
   very critical in Airbnb/YouTube/Amazon/Spotify/Booking.com/Linkedin, etc..
   

   View Slide

7. Basic Setup and Notation
   Position Items Relevance Exposure
   k=1 i1 r(u,i1) e(1)=1.0
   k=2 i2 r(u,i2) e(2)=0.5
   ... ... ... ...
   A ranking of items/products shown to a user
   : items
   : users
   : exposure probability of the “k”-th position (known)
   : relevance
   (known)
   

   View Slide

8. We formulate Ranking as Resource Allocation
   users (u) positions (k)
   u1
   1
   2
   3
   u2
   1
   2
   3
   items
   resources to allocate
   to the items
   (higher positions are
   more valuable)
   

   View Slide

9. Allocation of Positions as Valuable Resources
   users (u) positions (k)
   u1
   1
   2
   3
   u2
   1
   2
   3
   we want to optimize this
   “ranking rule”
   Items
   

   View Slide

10. “Doubly” Stochastic Matrix
    k=1 k=2 k=3
    i1 0.6 0.3 0.1 1
    i2 0.3 0.3 0.4 1
    i3 0.1 0.4 0.5 1
    1 1 1
    For each user “u”, we obtain a doubly stochastic matrix like the following
    

    View Slide

11. Utility to the users (typical performance metric in ranking)
    Utility to the users under ranking “X”
    click probability based on position-based model
    (can be conversion, dwell time, DCG, etc..)
    Typical ranking aims at
    maximizing the user-utility
    total number of clicks
    given by ranking "X"
    stochastic ranking
    an ideal ranking under the typical formulation
    

    View Slide

12. Impact on the items
    Impact the ranking "X" has on each item “i”
    We also want to fairly distribute impact to the items
    total clicks each item
    click probability based on position-based model
    (can be conversion, dwell time, DCG, etc..)
    total number of clicks
    given to each item
    under ranking "X"
    

    View Slide

13. Well-Established Framework
    Let’s allocate exposure proportional to item merit
    ● Expo-Fair Ranking
    -- Maximizes the user utility under the “exposure-fairness” constraint
    SIGIR’18
    KDD’18
    definition of fairness in ranking
    

    View Slide

14. Merit and Exposure in the Exposure-Fair Framework
    ● Merit -- global/average popularity of item “i” in the entire platform
    independent of the ranking
    ● Exposure -- total exposure that is allocated to item “i” under ranking “X”
    dependent on the ranking
    

    View Slide

15. Well-Established Framework
    Let’s allocate exposure proportional to item merit
    ● Expo-Fair Ranking
    -- Maximizes the user utility under the “exposure-fairness” constraint
    SIGIR’18
    KDD’18
    items having similar merit
    should get similar exposure
    

    View Slide

16. Toy Example: Setup
    ● only two users (“u_1” and “u_2”) and items (“i_1” and “i_2”)
    ● exposure prob of position 1 is “1” while that of position 2 is “0”
    (it is meaningless for each item to be presented at position 2)
    : exposure probability of the “k”-th position
    

    View Slide

17. Toy Example: True Relevance Table
    relevance (r(u,i))
    0.8 0.3
    0.5 0.4
    1.3 0.7
    For each user, which item should be ranked
    at the top position in a ranking with what probability?
    more popular less popular
    

    View Slide

18. Max Ranking (Conventional)
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    2.0 0.0
    Allocates all exposure to the first item (because it is more relevant)
    

    View Slide

19. Very unfair to the items..
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    2.0 0.0
    1.3 0.0
    

    View Slide

20. Expo-Fair Ranking
    relevance/exposure
    0.8 / ??? 0.3 / ???
    0.5 / ??? 0.4 / ???
    1.3 0.7
    1.3 0.7
    allocate exposure proportional to
    item merit so to be fair to the items
    

    View Slide

21. Expo-Fair Ranking
    relevance/exposure
    0.8 / 1.0 0.3 / ???
    0.5 / 0.3 0.4 / ???
    1.3 0.7
    1.3 0.7
    Then, let’s maximize the utility under
    the exposure-fairness constraint
    

    View Slide

22. Expo-Fair Ranking
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    Satisfy the exposure fairness constraint (the “perfectly fair ranking”)
    

    View Slide

23. “Perfectly Fair Ranking”
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28
    

    View Slide

24. “Perfectly Fair Ranking”
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    0.95 0.28
    Do you think this perfectly fair ranking is really fair?
    

    View Slide

25. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    The “baseline” ranking we can achieve without any optimization
    

    View Slide

26. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    0.65 0.35
    

    View Slide

27. The expo-fair even fails to dominate the uniform baseline
    the first item receives
    a better impact than under
    the uniform ranking
    at the cost of impact on
    the second item...
    Exposure-Fair Ranking Uniform Random Baseline
    the exposure-fairness produces a large and obvious disparate impact
    Exposure-Fair
    (“perfectly fair”)
    Uniform
    Baseline
    Impact on
    “Item 1”
    0.95 (+46%) 0.65
    Impact on
    “Item 2”
    0.28 (-20%) 0.35
    

    View Slide

28. Let’s swap the ranking of the two items..
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28
    

    View Slide

29. Impact on the second item increases
    relevance/exposure
    0.8 / 0.0 0.3 / 1.0
    0.5 / 0.7 0.4 / 0.3
    1.3 0.7
    0.7 1.3
    0.35 (↓) 0.42 (↑)
    

    View Slide

30. The expo-fair ranking is also inefficient, producing “envy”
    the first item prefers the
    current ranking to that of the
    second item
    the second item prefers the
    ranking of the first item rather
    than its own (i_2 envies i_1)
    Exposure-Fair
    (“perfectly fair”)
    Swap each
    other’s ranking
    Impact on
    “Item 1”
    0.95 0.35 (-63%)
    Impact on
    “Item 2”
    0.28 0.42 (+50%)
    the exposure-fairness allows to produce some envy among items
    

    View Slide

31. Remaining “Unfairness” in the Exposure-Fair Framework
    It seems that we should “redefine” fairness in ranking..
    suggested by our toy example
    We’ve seen that the perfectly fair ranking in the exposure-fair framework
    ● produces some “envy” among items
    ● fails to guarantee the number of clicks under uniform ranking
    

    View Slide

32. Our Impact-based Axioms for Fair Ranking
    1. envy-freeness -- there should not be any item who envies some other
    items in the platform (seems very difficult to achieve and guarantee..)
    2. dominance over uniform ranking --
    every item should get better impact than under the uniform ranking
    

    View Slide

33. Need for a New Fair Ranking Algorithm
    We’ve seen that the perfectly fair ranking in the exposure-fair framework
    ● produces some “envy” among items (violates envy-freeness)
    ● fails to guarantee the number of clicks under uniform ranking
    (violates dominance over uniform ranking)
    suggested by our toy example
    How can we guarantee our impact-based axioms
    in a computationally efficient way?
    Our algorithmic question:
    

    View Slide

34. The “Fair Division” Problem in Classical Game Theory
    Fair Division: How should we fairly divide/allocate fruits?
    Apple Banana Orange
    Father 0.8 0.1 0.1
    Mother 0.2 0.0 0.9
    Child 0.3 0.0 0.8
    agents
    (divisible) fruits
    Preference Table
    

    View Slide

35. How about maximizing the “total family welfare”?
    Standard Approach would be to
    maximize the total family welfare
    (seems difficult to come up with another way)
    where
    is utility of each family member
    fruit
    allocation
    

    View Slide

36. How about maximizing the “total family welfare”?
    Maximizing the sum can lead to an
    “abusive” allocation, giving nothing to the child
    Allocation “X” Apple Banana Orange Utility
    Father 1.0 1.0 0.0 0.9
    Mother 0.0 0.0 1.0 0.9
    Child 0.0 0.0 0.0 0.0
    sum is maximized indeed
    

    View Slide

37. Great Idea in Fair Division: “Nash Social Welfare”
    Alternative approach is
    to maximize Nash Social Welfare
    The NSW doesn’t allow to have a zero utility agent,
    possibly producing a more equitable allocation
    In contrast, maximizng the sum allows zero utility agents..
    Actually, maximizing the NSW guarantees envy-freeness
    and dominance over uniform allocation in the fruit allocation problem
    

    View Slide

38. Great Idea in Fair Division: “Nash Social Welfare”
    Allocation “X” Apple Banana Orange Utility
    Father 1.0 1.0 0.0 0.90
    Mother 0.0 0.0 0.5 0.45
    Child 0.0 0.0 0.5 0.40
    product is now maximized
    Alternative approach is
    to maximize Nash Social Welfare
    

    View Slide

39. Fair Division to Fair Ranking
    The “equitable” NSW objective
    The “unfair” sum objective
    “Default Objective” in Ranking
    (Even the Exposure-Fair maximizes this one)
    ？？？
    total clicks each item
    

    View Slide

40. Nash Social Welfare Ranking (our main proposal)
    Then, let’s maximize the “product” of impact on the items
    guaranteed to dominate the uniform ranking and to be envy-free
    (at least approximately)
    

    View Slide

41. Nash Social Welfare Ranking (our main proposal)
    Then, let’s maximize the “product” of impact on item
    convex program, efficiently solvable
    

    View Slide

42. Toy Example: True Relevance Table
    relevance (r(u,i))
    0.8 0.3
    0.5 0.4
    1.3 0.7
    more popular less popular
    

    View Slide

43. Expo-Fair Ranking (Existing)
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28
    

    View Slide

44. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    0.65 0.35
    

    View Slide

45. NSW Ranking (Our Proposal)
    relevance/exposure
    0.8 / 1.0 0.3/ 0.0
    0.5 / 0.0 0.4 / 1.0
    1.3 0.7
    1.0 1.0
    0.80 0.40
    

    View Slide

46. There actually is no envy in the platform
    relevance/exposure
    0.8 / 0.0 0.3 / 1.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    1.0 1.0
    0.5 (↓) 0.3 (↓)
    

    View Slide

47. Summary of the Toy Example
    User Utility Imp_i1 Imp_i2
    Max 1.30 1.30 0.00
    Expo-Fair 1.23 0.95 0.28
    NSW (ours) 1.20 0.80 0.40
    Uniform 1.00 0.65 0.35
    dominate
    the uniform
    (& envy-free)
    

    View Slide

48. Synthetic Experiment Setup
    , evaluate the fairness-utility trade-off
    ● Mean-Max Envy
    (smaller is better)
    ● 10% Better-Off Pct
    (larger is better)
    ● 10% Worse-Off Pct
    (smaller is better)
    Mean of maximum envy
    for each item has
    Prop of items who improve
    their utility over 10% from unif
    Prop of items who degrade
    their utility over 10% from unif
    

    View Slide

49. Experiment Results
    Storonger popularity bias means there is a fewer number of very popular items,
    producing a more difficult situation to pursue fairness-utility at the same time
    

    View Slide

50. Experiment Results
    ● NSW is the most equitable, especially in the presence of popularity bias
    ● Max and Exposure-Based produce some envy and disparate impact
    ● However, there is a difficult tradeoff between fairness and utility
    especially when popularity bias is strong
    ours
    existing
    conventional
    

    View Slide

51. A Simple Extension: Controlling Utility-Fairness Trade-off
    alpha-NSW Allocation (\alpha=zero leads to the original NSW)
    A larger “alpha” leads to a higher utility,
    while a smaller value prioritizes fairness of impact
    

    View Slide

52. Weighted Envy-Freeness and Dominance
    1. weighted envy-freeness
    2. weighted dominance over the uniform random allocation
    

    View Slide

53. Steerable Trade-off via the “alpha-NSW” objective
    ● compare several different values of \alpha (0.0, 0.5, 1.0, 2.0)
    ● we can choose a variety of fairness-utility trade-off via tuning \alpha
    ● In particular, 1-NSW achieves the user utility similar to Exposure-based
    while leading to a much more equitable impact distribution
    

    View Slide

54. Overall Summary
    ● Current exposure-based fair ranking produces envy and fails to
    ensure the impact guaranteed under the uniform random baseline
    ● We redefine fairness in ranking via envy-freeness and dominance over
    uniform ranking to resolve the remaining unfairness
    ● By maximizing the product of the impacts or the NSW, we guarantee
    envy-freeness and dominance over uniform ranking in an efficient way
    ● We also extend the NSW to have a steerable fairness-utility trade-off
    

    View Slide

55. More About The Paper
    ● Contact: ys552@cornell.edu
    ● arXiv: https://arxiv.org/abs/2202.06317
    ● Experiment code:
    https://github.com/usaito/kdd2022-fair-ranking-nsw
    Thank you!
    

    View Slide