Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking (KDD'22)

Rankings have become the primary interface of many two-sided markets. Many have noted that the rankings not only affect the satisfaction of the users (e.g., customers, listeners, employers, travelers), but that the position in the ranking allocates exposure – and thus economic opportunity – to the ranked items (e.g., articles, products, songs, job seekers, restaurants, hotels). This has raised questions of fairness to the items, and most existing works have addressed fairness by explicitly linking item exposure to item relevance. However, we argue that any particular choice of such a link function may be difficult to defend, and we show that the resulting rankings can still be unfair. To avoid these shortcomings, we develop a new axiomatic approach that is rooted in principles of fair division. This not only avoids the need to choose a link function, but also more meaningfully quantifies the impact on the items beyond exposure. Our axioms of envy-freeness and dominance over uniform ranking postulate that in fair rankings every item should prefer their own rank allocation over that of any other item, and that no item should be actively disadvantaged by the rankings. To compute ranking policies that are fair according to these axioms, we propose a new ranking objective related to the Nash Social Welfare. We show that the solution has guarantees regarding its envy-freeness, its dominance over uniform rankings for every item, and its Pareto optimality. In contrast, we show that conventional exposure-based fairness can produce large amounts of envy and have a highly disparate impact on the items. Beyond these theoretical results, we illustrate empirically how our framework controls the trade-off between impact-based individual item fairness and user utility.

usaito
PRO

July 04, 2022
Tweet

More Decks by usaito

Other Decks in Research

Transcript

  1. Fair Ranking as Fair Division:
    Impact-Based Individual Fairness in Ranking
    (KDD2022)
    Yuta Saito (w/ Thorsten Joachims)

    View Slide

  2. Outline
    ● Existing “Exposure-based” Fairness
    ○ intro to fair-ranking and the exposure-based framework
    ○ remaining issues/unfairness in the exposure-based framework
    ● Our “Impact-based” Axioms and Fair Ranking Method
    ○ redefine fairness in ranking via some “impact-based” axioms
    ○ also describe a simple and efficient method to satisfy our axioms
    SIGIR’18
    KDD’18
    Seminal Papers

    View Slide

  3. Ranking in Online Platform / Marketplaces
    There are many use cases of ranking in real-world applications
    -> typically, rankings optimize only the user satisfaction
    Amazon Booking.com ICML2021

    View Slide

  4. Fairness Considerations in Ranking
    ● Utility to Users:
    Travelers want to find hotels relevant to
    their query (location, budget) quickly
    ● Impact on Items:
    Hotels earn money from the ranking
    Ex1: Hotels Search
    Booking.com

    View Slide

  5. ● Utility to Users:
    Conference attendees want to find
    accepted papers relevant to their
    expertise
    ● Impact on Items:
    Authors of the accepted papers want to
    get as much attention as possible
    Fairness Considerations in Ranking
    Ex2: Academic Conference ICML2021

    View Slide

  6. ● Typically, ranking systems
    maximize the user utility by
    ordering items by “relevance”
    ● but it often produces unfair
    rankings wrt “exposure”
    ● How should we define and
    ensure fairness in ranking?
    High-Level Motivation of “Fair Ranking”
    Position Items Relevance Exposure
    1 hotel1 0.95 1
    2 hotel2 0.94 0.5
    ... ... ... ...
    100 hotel100 0.90 0.01
    A ranking of items shown to a user
    very critical in Airbnb/YouTube/Amazon/Spotify/Booking.com/Linkedin, etc..

    View Slide

  7. Basic Setup and Notation
    Position Items Relevance Exposure
    k=1 i1 r(u,i1) e(1)=1.0
    k=2 i2 r(u,i2) e(2)=0.5
    ... ... ... ...
    A ranking of items/products shown to a user
    : items
    : users
    : exposure probability of the “k”-th position (known)
    : relevance
    (known)

    View Slide

  8. We formulate Ranking as Resource Allocation
    users (u) positions (k)
    u1
    1
    2
    3
    u2
    1
    2
    3
    items
    resources to allocate
    to the items
    (higher positions are
    more valuable)

    View Slide

  9. Allocation of Positions as Valuable Resources
    users (u) positions (k)
    u1
    1
    2
    3
    u2
    1
    2
    3
    we want to optimize this
    “ranking rule”
    Items

    View Slide

  10. “Doubly” Stochastic Matrix
    k=1 k=2 k=3
    i1 0.6 0.3 0.1 1
    i2 0.3 0.3 0.4 1
    i3 0.1 0.4 0.5 1
    1 1 1
    For each user “u”, we obtain a doubly stochastic matrix like the following

    View Slide

  11. Utility to the users (typical performance metric in ranking)
    Utility to the users under ranking “X”
    click probability based on position-based model
    (can be conversion, dwell time, DCG, etc..)
    Typical ranking aims at
    maximizing the user-utility
    total number of clicks
    given by ranking "X"
    stochastic ranking
    an ideal ranking under the typical formulation

    View Slide

  12. Impact on the items
    Impact the ranking "X" has on each item “i”
    We also want to fairly distribute impact to the items
    total clicks each item
    click probability based on position-based model
    (can be conversion, dwell time, DCG, etc..)
    total number of clicks
    given to each item
    under ranking "X"

    View Slide

  13. Well-Established Framework
    Let’s allocate exposure proportional to item merit
    ● Expo-Fair Ranking
    -- Maximizes the user utility under the “exposure-fairness” constraint
    SIGIR’18
    KDD’18
    definition of fairness in ranking

    View Slide

  14. Merit and Exposure in the Exposure-Fair Framework
    ● Merit -- global/average popularity of item “i” in the entire platform
    independent of the ranking
    ● Exposure -- total exposure that is allocated to item “i” under ranking “X”
    dependent on the ranking

    View Slide

  15. Well-Established Framework
    Let’s allocate exposure proportional to item merit
    ● Expo-Fair Ranking
    -- Maximizes the user utility under the “exposure-fairness” constraint
    SIGIR’18
    KDD’18
    items having similar merit
    should get similar exposure

    View Slide

  16. Toy Example: Setup
    ● only two users (“u_1” and “u_2”) and items (“i_1” and “i_2”)
    ● exposure prob of position 1 is “1” while that of position 2 is “0”
    (it is meaningless for each item to be presented at position 2)
    : exposure probability of the “k”-th position

    View Slide

  17. Toy Example: True Relevance Table
    relevance (r(u,i))
    0.8 0.3
    0.5 0.4
    1.3 0.7
    For each user, which item should be ranked
    at the top position in a ranking with what probability?
    more popular less popular

    View Slide

  18. Max Ranking (Conventional)
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    2.0 0.0
    Allocates all exposure to the first item (because it is more relevant)

    View Slide

  19. Very unfair to the items..
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    2.0 0.0
    1.3 0.0

    View Slide

  20. Expo-Fair Ranking
    relevance/exposure
    0.8 / ??? 0.3 / ???
    0.5 / ??? 0.4 / ???
    1.3 0.7
    1.3 0.7
    allocate exposure proportional to
    item merit so to be fair to the items

    View Slide

  21. Expo-Fair Ranking
    relevance/exposure
    0.8 / 1.0 0.3 / ???
    0.5 / 0.3 0.4 / ???
    1.3 0.7
    1.3 0.7
    Then, let’s maximize the utility under
    the exposure-fairness constraint

    View Slide

  22. Expo-Fair Ranking
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    Satisfy the exposure fairness constraint (the “perfectly fair ranking”)

    View Slide

  23. “Perfectly Fair Ranking”
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28

    View Slide

  24. “Perfectly Fair Ranking”
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    0.95 0.28
    Do you think this perfectly fair ranking is really fair?

    View Slide

  25. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    The “baseline” ranking we can achieve without any optimization

    View Slide

  26. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    0.65 0.35

    View Slide

  27. The expo-fair even fails to dominate the uniform baseline
    the first item receives
    a better impact than under
    the uniform ranking
    at the cost of impact on
    the second item...
    Exposure-Fair Ranking Uniform Random Baseline
    the exposure-fairness produces a large and obvious disparate impact
    Exposure-Fair
    (“perfectly fair”)
    Uniform
    Baseline
    Impact on
    “Item 1”
    0.95 (+46%) 0.65
    Impact on
    “Item 2”
    0.28 (-20%) 0.35

    View Slide

  28. Let’s swap the ranking of the two items..
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28

    View Slide

  29. Impact on the second item increases
    relevance/exposure
    0.8 / 0.0 0.3 / 1.0
    0.5 / 0.7 0.4 / 0.3
    1.3 0.7
    0.7 1.3
    0.35 (↓) 0.42 (↑)

    View Slide

  30. The expo-fair ranking is also inefficient, producing “envy”
    the first item prefers the
    current ranking to that of the
    second item
    the second item prefers the
    ranking of the first item rather
    than its own (i_2 envies i_1)
    Exposure-Fair
    (“perfectly fair”)
    Swap each
    other’s ranking
    Impact on
    “Item 1”
    0.95 0.35 (-63%)
    Impact on
    “Item 2”
    0.28 0.42 (+50%)
    the exposure-fairness allows to produce some envy among items

    View Slide

  31. Remaining “Unfairness” in the Exposure-Fair Framework
    It seems that we should “redefine” fairness in ranking..
    suggested by our toy example
    We’ve seen that the perfectly fair ranking in the exposure-fair framework
    ● produces some “envy” among items
    ● fails to guarantee the number of clicks under uniform ranking

    View Slide

  32. Our Impact-based Axioms for Fair Ranking
    1. envy-freeness -- there should not be any item who envies some other
    items in the platform (seems very difficult to achieve and guarantee..)
    2. dominance over uniform ranking --
    every item should get better impact than under the uniform ranking

    View Slide

  33. Need for a New Fair Ranking Algorithm
    We’ve seen that the perfectly fair ranking in the exposure-fair framework
    ● produces some “envy” among items (violates envy-freeness)
    ● fails to guarantee the number of clicks under uniform ranking
    (violates dominance over uniform ranking)
    suggested by our toy example
    How can we guarantee our impact-based axioms
    in a computationally efficient way?
    Our algorithmic question:

    View Slide

  34. The “Fair Division” Problem in Classical Game Theory
    Fair Division: How should we fairly divide/allocate fruits?
    Apple Banana Orange
    Father 0.8 0.1 0.1
    Mother 0.2 0.0 0.9
    Child 0.3 0.0 0.8
    agents
    (divisible) fruits
    Preference Table

    View Slide

  35. How about maximizing the “total family welfare”?
    Standard Approach would be to
    maximize the total family welfare
    (seems difficult to come up with another way)
    where
    is utility of each family member
    fruit
    allocation

    View Slide

  36. How about maximizing the “total family welfare”?
    Maximizing the sum can lead to an
    “abusive” allocation, giving nothing to the child
    Allocation “X” Apple Banana Orange Utility
    Father 1.0 1.0 0.0 0.9
    Mother 0.0 0.0 1.0 0.9
    Child 0.0 0.0 0.0 0.0
    sum is maximized indeed

    View Slide

  37. Great Idea in Fair Division: “Nash Social Welfare”
    Alternative approach is
    to maximize Nash Social Welfare
    The NSW doesn’t allow to have a zero utility agent,
    possibly producing a more equitable allocation
    In contrast, maximizng the sum allows zero utility agents..
    Actually, maximizing the NSW guarantees envy-freeness
    and dominance over uniform allocation in the fruit allocation problem

    View Slide

  38. Great Idea in Fair Division: “Nash Social Welfare”
    Allocation “X” Apple Banana Orange Utility
    Father 1.0 1.0 0.0 0.90
    Mother 0.0 0.0 0.5 0.45
    Child 0.0 0.0 0.5 0.40
    product is now maximized
    Alternative approach is
    to maximize Nash Social Welfare

    View Slide

  39. Fair Division to Fair Ranking
    The “equitable” NSW objective
    The “unfair” sum objective
    “Default Objective” in Ranking
    (Even the Exposure-Fair maximizes this one)
    ???
    total clicks each item

    View Slide

  40. Nash Social Welfare Ranking (our main proposal)
    Then, let’s maximize the “product” of impact on the items
    guaranteed to dominate the uniform ranking and to be envy-free
    (at least approximately)

    View Slide

  41. Nash Social Welfare Ranking (our main proposal)
    Then, let’s maximize the “product” of impact on item
    convex program, efficiently solvable

    View Slide

  42. Toy Example: True Relevance Table
    relevance (r(u,i))
    0.8 0.3
    0.5 0.4
    1.3 0.7
    more popular less popular

    View Slide

  43. Expo-Fair Ranking (Existing)
    relevance/exposure
    0.8 / 1.0 0.3 / 0.0
    0.5 / 0.3 0.4 / 0.7
    1.3 0.7
    1.3 0.7
    0.95 0.28

    View Slide

  44. Uniform Random Ranking
    relevance/exposure
    0.8 / 0.5 0.3 / 0.5
    0.5 / 0.5 0.4 / 0.5
    1.3 0.7
    1.0 1.0
    0.65 0.35

    View Slide

  45. NSW Ranking (Our Proposal)
    relevance/exposure
    0.8 / 1.0 0.3/ 0.0
    0.5 / 0.0 0.4 / 1.0
    1.3 0.7
    1.0 1.0
    0.80 0.40

    View Slide

  46. There actually is no envy in the platform
    relevance/exposure
    0.8 / 0.0 0.3 / 1.0
    0.5 / 1.0 0.4 / 0.0
    1.3 0.7
    1.0 1.0
    0.5 (↓) 0.3 (↓)

    View Slide

  47. Summary of the Toy Example
    User Utility Imp_i1 Imp_i2
    Max 1.30 1.30 0.00
    Expo-Fair 1.23 0.95 0.28
    NSW (ours) 1.20 0.80 0.40
    Uniform 1.00 0.65 0.35
    dominate
    the uniform
    (& envy-free)

    View Slide

  48. Synthetic Experiment Setup
    , evaluate the fairness-utility trade-off
    ● Mean-Max Envy
    (smaller is better)
    ● 10% Better-Off Pct
    (larger is better)
    ● 10% Worse-Off Pct
    (smaller is better)
    Mean of maximum envy
    for each item has
    Prop of items who improve
    their utility over 10% from unif
    Prop of items who degrade
    their utility over 10% from unif

    View Slide

  49. Experiment Results
    Storonger popularity bias means there is a fewer number of very popular items,
    producing a more difficult situation to pursue fairness-utility at the same time

    View Slide

  50. Experiment Results
    ● NSW is the most equitable, especially in the presence of popularity bias
    ● Max and Exposure-Based produce some envy and disparate impact
    ● However, there is a difficult tradeoff between fairness and utility
    especially when popularity bias is strong
    ours
    existing
    conventional

    View Slide

  51. A Simple Extension: Controlling Utility-Fairness Trade-off
    alpha-NSW Allocation (\alpha=zero leads to the original NSW)
    A larger “alpha” leads to a higher utility,
    while a smaller value prioritizes fairness of impact

    View Slide

  52. Weighted Envy-Freeness and Dominance
    1. weighted envy-freeness
    2. weighted dominance over the uniform random allocation

    View Slide

  53. Steerable Trade-off via the “alpha-NSW” objective
    ● compare several different values of \alpha (0.0, 0.5, 1.0, 2.0)
    ● we can choose a variety of fairness-utility trade-off via tuning \alpha
    ● In particular, 1-NSW achieves the user utility similar to Exposure-based
    while leading to a much more equitable impact distribution

    View Slide

  54. Overall Summary
    ● Current exposure-based fair ranking produces envy and fails to
    ensure the impact guaranteed under the uniform random baseline
    ● We redefine fairness in ranking via envy-freeness and dominance over
    uniform ranking to resolve the remaining unfairness
    ● By maximizing the product of the impacts or the NSW, we guarantee
    envy-freeness and dominance over uniform ranking in an efficient way
    ● We also extend the NSW to have a steerable fairness-utility trade-off

    View Slide

  55. More About The Paper
    ● Contact: [email protected]
    ● arXiv: https://arxiv.org/abs/2202.06317
    ● Experiment code:
    https://github.com/usaito/kdd2022-fair-ranking-nsw
    Thank you!

    View Slide