Share
Embed
The Case for Public Service Recommender Algorithms
In this position paper we lay out the role for public service organisations within the fairness, accountability, and transparency discourse. We explore the idea of public service algorithms and what role they might play, especially with recommender systems. We then describe a research agenda for public service recommendation systems.
(As given at FATRec2018 https://piret.gitlab.io/fatrec2018/program/ )
Ben Fields
October 06, 2018
Transcript
39 slides
-
The Case for Public Service Recommender Algorithms Ben Fields (presenting)
Rhianne Jones Tim Cowlishaw BBC -
Motivations and Aims The Case for Public Service Recommender Algorithms
| FATrec18 | 6 October 2018 -
Motivations and Aims Data to deliver personalised services has become
a key strategic priority for Public Service Media organisations across Europe The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Motivations and Aims Concerns about potential risk for PSM values:
• potential to undermine shared and collective media experiences • reinforce audiences’ preexisting preferences • (Public Service) Media becoming more like a goldfish bowl, rather than a window to the world • Lots of examples in recent work: Pasquale 2015, Bulck and Moe 2017, Bennet 2018, Lots 2018, Sørensen and Hutchinson 2018 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Motivations and Aims For traditional commercial applications the goal is
a straightforward extension of an organisation’s overall commercial aims https://www.flickr.com/photos/24354425@N03/16593266327 The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
“inform, educate, and entertain” The Case for Public Service Recommender
Algorithms | FATrec18 | 6 October 2018 -
“inform, educate, and entertain” Fairness, accountability, and transparency <=?=> The
Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
A Research Agenda We join wider calls for PSM to
do personalisation differently (Bennet 2018, Helberger 2015) We do this from a specific PSM context but with wider relevance in mind The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
A Research Agenda Thus far guidance on how PSM should
approach personalisation has been vague and not sufficient to drive implementation The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Our position paper asks - how can/should public service algorithms
that enshrine the principles of Fairness, Accountability and Transparency (FAT) lead to novel ways to design recommender algorithms? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
How do we operationalise public service media (PSM) values as
tangible concepts in specific PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <1> -
operationalise PSM • Caveat: Notions of public service inevitably vary
across different geo-political and cultural contexts (Helberger 2015) • This is principally about core aspects to a PSM approach/ remits have implications for how we design and evaluate recommenders • Key challenges: operationalising concepts like diversity, surprise, shared experience, etc. The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
operationalise PSM “any initiative to promote diversity exposure will first
have to deal with ‘the question of what exposure diversity actually is’ as well as how to measure it” (Helberger et al. 2018) The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
What are useful metrics for which to optimise (e.g. diversity
or serendipity), how should the importance of different metrics be balanced in different PSM contexts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <2> -
Optimise the metrics Can we select our metrics to explicitly
address civic goals: • to counter filter bubble effects (Bozdag and van den Hoven 2015)? • for sociocultural diversity (Sheth et al 2011)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Optimise the metrics How can we broaden the scope of
these metrics? e.g serendipity, self-actualisation The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Optimise the metrics “even if an algorithm is designed with
the goal of stimulating ‘diversity’ an assessment of its performance by other measures nullifies these good intentions” (van Es 2017) The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
What data (metadata/audience data) should algorithms work on, what are
the limits of this data in its current form and how might awareness of this inform new approaches? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <3> -
Data selection • Content vs. metadata vs. behavioural? • Privacy/ethical
concerns • Regulatory requirements The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
How much accuracy loss is acceptable in pursuit of new
metrics, e.g. diversity? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <4> -
At what (accuracy) cost? • Tradeoffs must be made explicitly
to minimise unexpected and undesirable outcomes • Can this question be answered through other research practises? (e.g. audience research, UX methodologies)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
How should transparency work - when and to whom is
it useful, e.g. regulators? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <5> -
Transparency for whom? Principles of transparency key to the mission
of PSM: They provide the mechanism by which PSM are regulated and held accountable The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Transparency for whom? • Transparency for stakeholders/audiences or regulators? •
When we say transparency, when do we mean disclosure? • In a user-facing system this is an issue of enabling consent to be meaningful • However in a stakeholder arrangement this is about disclosure The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
To what extent should we be transparent about how we
are resolving metric and optimisation complexity (the trade-offs we are making)? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <6> -
Transparency, how much? • What is the effective and maximally
useful fidelity of transparency? • Is it optimal/necessary/ideal to exposure users to metric tradeoffs? • How transparent to be about accuracy v. diversity? • Third parties/party platforms/challenges? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
How do we design for interpretability and explainability to enable
appropriate oversight of how recommenders are making decisions and ensure due accountability? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <7> -
Design for accountability? • Accountability vital to PSM/BBC • Transparency/full
disclosure vs meaningful explanations for a user • Does the desire for transparency push algorithmic design in particular directions? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Design for accountability? • How can complex algorithmic systems designed
to be intelligible/interpretable • What types of explanations can a system generate/what’s possible? • What explanations will be sufficient for oversight? • How will explanation needs vary e.g regulators/ stakeholders/editorial/public? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
What do emerging approaches in algorithmic auditing offer us in
terms of scrutinising recommender systems in the real world? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <8> -
Algorithmic auditing for recsys? • Does auditing an algorithmic system
change the system and by extension the user (experience) • How can we observe and monitor impacts of algorithmic systems? • Is auditing a useful approach to assess bias, unfairness, diversity etc? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
What type/level of explanation will be most useful? Will explanations
produced for editorial need to vary from the type of explanations PSM may provide to audiences? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <9> -
Explanation, how much? For whom? • Are we explaining algorithmic
decision making to users or to experts (principally: editorial teams)? • What is the distance between these two groups? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
How will we determine the value of different potential approaches?
How might new methodologies, e.g. multi-method, comparative, or longitudinal research, explore cumulative effects? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 <10> -
Determining value • What does testing look like in a
public service context? • Should we consult public on value of different approaches in PSM contexts • Do public service organisations have an obligation to explore longer view and cumulative impacts? The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Next steps The Case for Public Service Recommender Algorithms |
FATrec18 | 6 October 2018 -
better align recommender systems in public service contexts with their
underlying value frameworks The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018 -
Thanks! Let’s have some questions! Ben Fields @alsothings ben.fi[email protected] http://bit.ly/bbcfatrec18
The Case for Public Service Recommender Algorithms | FATrec18 | 6 October 2018