Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intrinsic social motivation via causal influence in multi-agent RL

Intrinsic social motivation via causal influence in multi-agent RL

Presentation of a paper by Jacques et al. (2018) at the Mila Learning Agents RG: https://arxiv.org/abs/1810.08647

An important feature of human decision-making is our ability to predict each others’ behavior. Through social interaction, we learn a model of others’ internal states, which helps us to anticipate future actions, plan and collaborate. Recent deep learning models have been compared to idiot savants - capable of performing highly specialized tasks but lacking what social psychology calls a “theory of mind”. In this research, Jaques et al. study the conditions for a theory of mind to emerge in multi-agent RL, and discover an interesting connection to causal inference.

The authors began by exploring a novel reward structure based on “social influence”, observing a rudimentary form of communication emerged between agents. Then, by providing an explicit communication channel, they observed agents could achieve better collective outcomes. Finally, using tools from causal inference, they endowed each agent with a model of other agents (MOA) network, allowing them to predict others’ actions without direct access to the counterpart’s reward function. In doing so, agents exhibited intrinsic motivation and the researchers were able to remove the external reward mechanism altogether.

In this talk, we will discuss a few important ideas from Causal Inference, such as counterfactual reasoning, the MOA framework and the use of mutual information as a mechanism for designing social rewards. No prior background in causal modeling is required or expected.

Breandan Considine

July 18, 2023
Tweet

More Decks by Breandan Considine

Other Decks in Research

Transcript

  1. Intrinsic social motivation
    via causal influence
    in multi-agent RL
    Natasha Jaques, Angeliki Lazaridou,
    Edward Hughes, Çaglar Gulçehre et al.
    Presented by
    Breandan Considine

    View full-size slide

  2. http://www.cs.cornell.edu/~helou/IMRL.pdf
    Intrinsically motivated Reinforcement Learning
    2005

    View full-size slide

  3. Intrinsic motivation
    Novelty/Surprise
    1. Barto, A. et al. 2013. Novelty or Surprise?
    Frontiers in Psychology.
    2. Houthooft, R. et al. 2016. Information
    Maximizing Exploration. Advances in Neural
    Information Processing Systems.
    3. Itti, L. and Baldi, P. Bayesian surprise
    attracts human attention. Vision Research,
    49(10):1295–1306, 2009.
    4. Conti, et al. Improving Exploration in
    Evolution Strategies for Deep Reinforcement
    Learning via a Population of Novelty-Seeking
    Agents. CoRR, abs/1712.06560.
    Empowerment
    1. Klyubin, A.S., et al. 2005. All else being
    equal be empowered. In European Conference
    on Artificial Life (pp. 744–753).
    2. Mnih, V., et al., 2015. Human-level control
    through deep reinforcement learning.
    Nature, 518(7540), p.529.
    3. Klyubin, A.S., et al. 2008. Keep your
    options open: An information-based driving
    principle for sensorimotor systems.
    4. Jung, T., et al., 2011. Empowerment for
    continuous agent — environment systems.
    Adaptive Behavior, 19(1), pp.16–39.

    View full-size slide

  4. https://www.youtube.com/embed/0yI2wJ6F8r0?enablejsapi=1
    Unifying count-based exploration and intrinsic motivation

    View full-size slide

  5. Surprise-based intrinsic motivation
    for deep reinforcement learning
    2017

    View full-size slide

  6. Sequential social dilemmas

    View full-size slide

  7. Notation

    The actions of all N agents are combined to form a joint action
    Discounted future rewards
    Extrinsic and Intrinsic rewards
    Trajectories

    View full-size slide

  8. How would B’s action change if
    I had acted differently in this situation?

    View full-size slide

  9. Averaging over multiple counterfactuals
    Averaging over multiple counterfactuals

    View full-size slide

  10. Mutual information and empowerment

    View full-size slide

  11. Sequential social dilemmas

    View full-size slide

  12. Model of other agents

    View full-size slide

  13. https://www.youtube.com/embed/iH_V5WKQxmo?enablejsapi=1

    View full-size slide

  14. References
    Multi-agent RL in Sequential Social Dilemmas
    Emergent Communication through Negotiation
    Unifying Count-Based Exploration and Intrinsic
    Motivation
    Surprise-Based Intrinsic Motivation for Deep RL
    How can we define intrinsic motivation?
    Intrinsically Motivated Reinforcement Learning
    Formal Theory of Creativity, Fun, and Intrinsic
    Motivation

    View full-size slide