Intrinsic social motivation via causal influence in multi-agent RL

Presentation of a paper by Jacques et al. (2018) at the Mila Learning Agents RG: https://arxiv.org/abs/1810.08647

An important feature of human decision-making is our ability to predict each others’ behavior. Through social interaction, we learn a model of others’ internal states, which helps us to anticipate future actions, plan and collaborate. Recent deep learning models have been compared to idiot savants - capable of performing highly specialized tasks but lacking what social psychology calls a “theory of mind”. In this research, Jaques et al. study the conditions for a theory of mind to emerge in multi-agent RL, and discover an interesting connection to causal inference.

The authors began by exploring a novel reward structure based on “social influence”, observing a rudimentary form of communication emerged between agents. Then, by providing an explicit communication channel, they observed agents could achieve better collective outcomes. Finally, using tools from causal inference, they endowed each agent with a model of other agents (MOA) network, allowing them to predict others’ actions without direct access to the counterpart’s reward function. In doing so, agents exhibited intrinsic motivation and the researchers were able to remove the external reward mechanism altogether.

In this talk, we will discuss a few important ideas from Causal Inference, such as counterfactual reasoning, the MOA framework and the use of mutual information as a mechanism for designing social rewards. No prior background in causal modeling is required or expected.

Transcript

1. Intrinsic social motivation via causal influence in multi-agent RL Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Gulçehre et al. Presented by Breandan Considine
4. http://www.cs.cornell.edu/~helou/IMRL.pdf Intrinsically motivated Reinforcement Learning 2005
6. Intrinsic motivation Novelty/Surprise 1. Barto, A. et al. 2013. Novelty or Surprise? Frontiers in Psychology. 2. Houthooft, R. et al. 2016. Information Maximizing Exploration. Advances in Neural Information Processing Systems. 3. Itti, L. and Baldi, P. Bayesian surprise attracts human attention. Vision Research, 49(10):1295–1306, 2009. 4. Conti, et al. Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents. CoRR, abs/1712.06560. Empowerment 1. Klyubin, A.S., et al. 2005. All else being equal be empowered. In European Conference on Artificial Life (pp. 744–753). 2. Mnih, V., et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540), p.529. 3. Klyubin, A.S., et al. 2008. Keep your options open: An information-based driving principle for sensorimotor systems. 4. Jung, T., et al., 2011. Empowerment for continuous agent — environment systems. Adaptive Behavior, 19(1), pp.16–39.
8. https://www.youtube.com/embed/0yI2wJ6F8r0?enablejsapi=1 Unifying count-based exploration and intrinsic motivation
10. Surprise-based intrinsic motivation for deep reinforcement learning 2017
14. Sequential social dilemmas
18. Notation <State, Transition, Action, reward> The actions of all N agents are combined to form a joint action Discounted future rewards Extrinsic and Intrinsic rewards Trajectories
19. How would B’s action change if I had acted differently in this situation?
20. Averaging over multiple counterfactuals Averaging over multiple counterfactuals
21. Mutual information and empowerment
22. Sequential social dilemmas
25. Model of other agents
26. https://www.youtube.com/embed/iH_V5WKQxmo?enablejsapi=1
28. References Multi-agent RL in Sequential Social Dilemmas Emergent Communication through Negotiation Unifying Count-Based Exploration and Intrinsic Motivation Surprise-Based Intrinsic Motivation for Deep RL How can we define intrinsic motivation? Intrinsically Motivated Reinforcement Learning Formal Theory of Creativity, Fun, and Intrinsic Motivation