Save 37% off PRO during our Black Friday Sale! »

The Case for Public Service Recommender Algorithms

77cacd503a0b6641e3b7a2e26dbbbaa1?s=47 Ben Fields
October 06, 2018

The Case for Public Service Recommender Algorithms

In this position paper we lay out the role for public service organisations within the fairness, accountability, and transparency discourse. We explore the idea of public service algorithms and what role they might play, especially with recommender systems. We then describe a research agenda for public service recommendation systems.

(As given at FATRec2018 https://piret.gitlab.io/fatrec2018/program/ )

77cacd503a0b6641e3b7a2e26dbbbaa1?s=128

Ben Fields

October 06, 2018
Tweet

Transcript

  1. The Case for Public Service Recommender Algorithms Ben Fields (presenting)

    Rhianne Jones Tim Cowlishaw BBC
  2. Motivations and Aims The Case for Public Service Recommender Algorithms

    | FATrec18 | 6 October 2018
  3. Motivations and Aims Data to deliver personalised services has become

    a key strategic priority for Public Service Media organisations across Europe The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  4. Motivations and Aims Concerns about potential risk for PSM values:

    • potential to undermine shared and collective media experiences • reinforce audiences’ preexisting preferences • (Public Service) Media becoming more like a goldfish bowl, rather than a window to the world • Lots of examples in recent work: Pasquale 2015, Bulck and Moe 2017, Bennet 2018, Lots 2018, Sørensen and Hutchinson 2018 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  5. Motivations and Aims For traditional commercial applications the goal is

    a straightforward extension of an organisation’s overall commercial aims https://www.flickr.com/photos/24354425@N03/16593266327 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  6. “inform, educate, and entertain” The Case for Public Service Recommender

    Algorithms | FATrec18 | 6 October 2018
  7. “inform, educate, and entertain” Fairness, accountability, and transparency <=?=> The

    Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  8. A Research Agenda We join wider calls for PSM to

    do personalisation differently (Bennet 2018, Helberger 2015) We do this from a specific PSM context but with wider relevance in mind The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  9. A Research Agenda Thus far guidance on how PSM should

    approach personalisation has been vague and not sufficient to drive implementation The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  10. Our position paper asks - how can/should public service algorithms

    that enshrine the principles of Fairness, Accountability and Transparency (FAT) lead to novel ways to design recommender algorithms? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  11. How do we operationalise public service media (PSM) values as

    tangible concepts in specific PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <1>
  12. operationalise PSM • Caveat: Notions of public service inevitably vary

    across different geo-political and cultural contexts (Helberger 2015) • This is principally about core aspects to a PSM approach/ remits have implications for how we design and evaluate recommenders • Key challenges: operationalising concepts like diversity, surprise, shared experience, etc. The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  13. operationalise PSM “any initiative to promote diversity exposure will first

    have to deal with ‘the question of what exposure diversity actually is’ as well as how to measure it” (Helberger et al. 2018) The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  14. What are useful metrics for which to optimise (e.g. diversity

    or serendipity), how should the importance of different metrics be balanced in different PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <2>
  15. Optimise the metrics Can we select our metrics to explicitly

    address civic goals: • to counter filter bubble effects (Bozdag and van den Hoven 2015)? • for sociocultural diversity (Sheth et al 2011)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  16. Optimise the metrics How can we broaden the scope of

    these metrics? e.g serendipity, self-actualisation
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  17. Optimise the metrics “even if an algorithm is designed with

    the goal of stimulating ‘diversity’ an assessment of its performance by other measures nullifies these good intentions” (van Es 2017)
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  18. What data (metadata/audience data) should algorithms work on, what are

    the limits of this data in its current form and how might awareness of this inform new approaches? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <3>
  19. Data selection • Content vs. metadata vs. behavioural? • Privacy/ethical

    concerns • Regulatory requirements
 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  20. How much accuracy loss is acceptable in pursuit of new

    metrics, e.g. diversity? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <4>
  21. At what (accuracy) cost? • Tradeoffs must be made explicitly

    to minimise unexpected and undesirable outcomes • Can this question be answered through other research practises? (e.g. audience research, UX methodologies)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  22. How should transparency work - when and to whom is

    it useful, e.g. regulators? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <5>
  23. Transparency for whom? Principles of transparency key to the mission

    of PSM: They provide the mechanism by which PSM are regulated and held accountable The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  24. Transparency for whom? • Transparency for stakeholders/audiences or regulators? •

    When we say transparency, when do we mean disclosure? • In a user-facing system this is an issue of enabling consent to be meaningful • However in a stakeholder arrangement this is about disclosure The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  25. To what extent should we be transparent about how we

    are resolving metric and optimisation complexity (the trade-offs we are making)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <6>
  26. Transparency, how much? • What is the effective and maximally

    useful fidelity of transparency? • Is it optimal/necessary/ideal to exposure users to metric tradeoffs? • How transparent to be about accuracy v. diversity? • Third parties/party platforms/challenges? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  27. How do we design for interpretability and explainability to enable

    appropriate oversight of how recommenders are making decisions and ensure due accountability? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <7>
  28. Design for accountability? • Accountability vital to PSM/BBC • Transparency/full

    disclosure vs meaningful explanations for a user • Does the desire for transparency push algorithmic design in particular directions? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  29. Design for accountability? • How can complex algorithmic systems designed

    to be intelligible/interpretable • What types of explanations can a system generate/what’s possible? • What explanations will be sufficient for oversight? • How will explanation needs vary e.g regulators/ stakeholders/editorial/public? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  30. What do emerging approaches in algorithmic auditing offer us in

    terms of scrutinising recommender systems in the real world? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <8>
  31. Algorithmic auditing for recsys? • Does auditing an algorithmic system

    change the system and by extension the user (experience) • How can we observe and monitor impacts of algorithmic systems? • Is auditing a useful approach to assess bias, unfairness, diversity etc? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  32. What type/level of explanation will be most useful? Will explanations

    produced for editorial need to vary from the type of explanations PSM may provide to audiences? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <9>
  33. Explanation, how much? For whom? • Are we explaining algorithmic

    decision making to users or to experts (principally: editorial teams)? • What is the distance between these two groups? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  34. How will we determine the value of different potential approaches?

    How might new methodologies, e.g. multi-method, comparative, or longitudinal research, explore cumulative effects? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <10>
  35. Determining value • What does testing look like in a

    public service context? • Should we consult public on value of different approaches in PSM contexts • Do public service organisations have an obligation to explore longer view and cumulative impacts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  36. Next steps The Case for Public Service Recommender Algorithms |

    FATrec18 | 6 October 2018
  37. better align recommender systems in public service contexts with their

    underlying value frameworks The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018
  38. The Case for Public Service Recommender Algorithms | FATrec18 |

    6 October 2018
  39. Thanks! Let’s have some questions! Ben Fields @alsothings ben.fields@bbc.co.uk http://bit.ly/bbcfatrec18

    The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018