Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intrinsic social motivation via causal influence in multi-agent RL

Intrinsic social motivation via causal influence in multi-agent RL

Presentation of a paper by Jacques et al. (2018) at the Mila Learning Agents RG: https://arxiv.org/abs/1810.08647

An important feature of human decision-making is our ability to predict each others’ behavior. Through social interaction, we learn a model of others’ internal states, which helps us to anticipate future actions, plan and collaborate. Recent deep learning models have been compared to idiot savants - capable of performing highly specialized tasks but lacking what social psychology calls a “theory of mind”. In this research, Jaques et al. study the conditions for a theory of mind to emerge in multi-agent RL, and discover an interesting connection to causal inference.

The authors began by exploring a novel reward structure based on “social influence”, observing a rudimentary form of communication emerged between agents. Then, by providing an explicit communication channel, they observed agents could achieve better collective outcomes. Finally, using tools from causal inference, they endowed each agent with a model of other agents (MOA) network, allowing them to predict others’ actions without direct access to the counterpart’s reward function. In doing so, agents exhibited intrinsic motivation and the researchers were able to remove the external reward mechanism altogether.

In this talk, we will discuss a few important ideas from Causal Inference, such as counterfactual reasoning, the MOA framework and the use of mutual information as a mechanism for designing social rewards. No prior background in causal modeling is required or expected.

Breandan Considine

July 18, 2023

More Decks by Breandan Considine

Other Decks in Research


  1. Intrinsic social motivation via causal influence in multi-agent RL Natasha

    Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Gulçehre et al. Presented by Breandan Considine
  2. Intrinsic motivation Novelty/Surprise 1. Barto, A. et al. 2013. Novelty

    or Surprise? Frontiers in Psychology. 2. Houthooft, R. et al. 2016. Information Maximizing Exploration. Advances in Neural Information Processing Systems. 3. Itti, L. and Baldi, P. Bayesian surprise attracts human attention. Vision Research, 49(10):1295–1306, 2009. 4. Conti, et al. Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents. CoRR, abs/1712.06560. Empowerment 1. Klyubin, A.S., et al. 2005. All else being equal be empowered. In European Conference on Artificial Life (pp. 744–753). 2. Mnih, V., et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), p.529. 3. Klyubin, A.S., et al. 2008. Keep your options open: An information-based driving principle for sensorimotor systems. 4. Jung, T., et al., 2011. Empowerment for continuous agent — environment systems. Adaptive Behavior, 19(1), pp.16–39.
  3. Notation <State, Transition, Action, reward> The actions of all N

    agents are combined to form a joint action Discounted future rewards Extrinsic and Intrinsic rewards Trajectories
  4. References Multi-agent RL in Sequential Social Dilemmas Emergent Communication through

    Negotiation Unifying Count-Based Exploration and Intrinsic Motivation Surprise-Based Intrinsic Motivation for Deep RL How can we define intrinsic motivation? Intrinsically Motivated Reinforcement Learning Formal Theory of Creativity, Fun, and Intrinsic Motivation