Collective Predictive Coding Hypothesis and Beyond (@Japanese Association for Philosophy of Science, 26th October 2024)

Collective Predictive Coding Hypothesis and Beyond

Tadairo Taniguchi

Graduate School of Informatics, Kyoto University
Research Organization of Science and Technology, Ritsumeikan University

Humanity has accumulated and passed down various knowledge and cultures within and across communities through the formation of language and communication. How to computationally express such phenomena has been an important question in research on language evolution, symbol emergence, and emergent communication. The author has been proposing a systemic view called symbol emergence systems and has taken a constructive approach to studying symbol emergence, known as symbol emergence in robotics (Taniguchi et al. 2016, 2019). Based on these constructive studies, the Collective Predictive Coding (CPC) hypothesis was proposed (Taniguchi 2024). This CPC hypothesis is based on a model that can express the formation of internal representations as representation learning through individual predictive coding. It proposes that symbol emergence, as the formation of external representations, can be expressed as collective predictive coding among multiple agents.

The CPC hypothesis aims to provide a unified computational framework for understanding how symbolic communication, particularly language, emerges in human societies, and how internal representations are formed in human cognitive systems. The CPC hypothesis argues that symbol emergence is viewed as social representation learning, which acts as distributed Bayesian inference. This distributed inference is embodied through language games, whose representative model is the Metropolis-Hastings naming game (Taniguchi et al. 2023), where each agent makes autonomous decisions to reject or adopt signs referring to their respective beliefs. The idea has been tested in an experimental semiotics study (Okumura et al. 2023). In essence, the CPC hypothesis proposes that language emerges as a collective effort to predict and encode the sensory experiences of all members of a society. It extends the concept of predictive coding from individual brains to the societal level, suggesting that symbol systems like language arise from a decentralized process of minimizing prediction errors across a population of agents interacting with their environment and each other.

After the proposal of the CPC hypothesis, we are gradually realizing that the total structure of the CPC seems to be relevant to scientific activity. The CPC model internally embraces not only the bottom-up formation of symbol systems reflecting the world structure based on observations but also top-down constraints given to the agents who are participating in communication using symbol systems. Also, the language game, including propose and acceptance/reject decisions referring to their own beliefs, is analogous to scientific communications, e.g., discussion and submitting and reviewing papers. Such systematic correspondence between symbol emergence and scientific activities leads us to the application of CPC to scientific activities in society, i.e., CPC as a model of science (CPC-MS) (Taniguchi et al. 2024).

This presentation will introduce the CPC hypothesis and then the basics of CPC-MS as an extension of the CPC hypothesis. Also, some additional implications will be presented.

References

1. Okumura, R., Taniguchi, T., Hagiwara, Y., & Taniguchi, A. (2023). Metropolis-Hastings algorithm in joint-attention naming game: experimental semiotics study. Frontiers in Artificial Intelligence, 6, 1235231.
2. Taniguchi, T. (2024). Collective predictive coding hypothesis: Symbol emergence as decentralized bayesian inference. Frontiers in Robotics and AI, 11, 1353870.
3. Taniguchi, T., Nagai, T., Nakamura, T., Iwahashi, N., Ogata, T., & Asoh, H. (2016). Symbol emergence in robotics: a survey. Advanced Robotics, 30(11-12), 706-728.
4. Taniguchi, T., Takagi, S., Otsuka, J., Hayashi, Y., & Hamada, T. (submitted). Collective Predictive Coding as Model of Science: Formalizing Scientific Activities towards Generative Science. (arXiv preprint arXiv:2409.00102, 2024)
5. Taniguchi, T., Ugur, E., Hoffmann, M., Jamone, L., Nagai, T., Rosman, B., Matsuka, T., Iwahashi, N., Oztop, E., Piater, J., & Wörgötter, F. (2019). Symbol Emergence in Cognitive Developmental Systems: A Survey. IEEE Transactions on Cognitive and Developmental Systems, 11(4), 494-516.
6. Taniguchi, T., Yoshida, Y., Matsui, Y., Le Hoang, N., Taniguchi, A., & Hagiwara, Y. (2023). Emergent communication through metropolis-hastings naming game with deep generative models. Advanced Robotics, 37(19), 1266-1282.

Transcript

1. Tadahiro Taniguchi 1) Professor, Graduate School of Informatics, Kyoto University 2) Visiting Professor, Research Organization of Science and Technology, Ritsumeikan University 3) Senior Technical Advisor, Panasonic Holdings Corporation Talk at Japanese Association for Philosophy of Science 26th October 2024 Collective Predictive Coding Hypothesis and Beyond
2. Collective Predictive Coding as Model of Science: Formalizing Scientific Activities Towards Generative Science [Taniguchi+ 2024]  Collective predictive coding, which encodes a model of the world as explicit knowledge through distributed interactions, seems to be an appropriate model for scientific activity.  CPC-MS = Collective Predictive Coding as Model of Science 2 Taniguchi, T., Takagi, S., Otsuka, J., Hayashi, Y., & Hamada, H. T. (2024). Collective Predictive Coding as Model of Science: Formalizing Scientific Activities Towards Generative Science. arXiv preprint arXiv:2409.00102. Scientific knowledge
3. Symbol Emergence Systems [Taniguchi 2010, and so on] 谷口忠大 「コミュニケーションするロボットは創れるか」（NTT出版）2010 谷口忠大「心を知るための人工知能」共立出版、2020
4. Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Representation learning Symbol emergence systems [Taniguchi+ 2016] 記号創発創発システム
5. Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Symbol emergence systems [Taniguchi+ 2016] 記号創発創発システム
6. Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Symbol emergence systems [Taniguchi+ 2016] 記号創発創発システム
7. Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Symbol emergence systems [Taniguchi+ 2016] 記号創発創発システム
8. Emergent symbol system Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Symbol emergence systems [Taniguchi+ 2016] 記号創発創発システム
9.  Human children models their subjective worlds through their own physical experiences and the integration of sensory-motor information, and they acquire language, enabling communication.  We also create and update symbol systems including language to understand the world and cooperate adaptively and autonomously.  SER aims to realize a developmental robot that acquires, invents, and shares language based on real-world physical and social experiences to understand the emergence of symbol systems in human society. Constructive approach to Symbol Emergence Systems 谷口忠大「記号創発ロボティクス」（講談社メチエ）2014 Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. Tadahiro Taniguchi, Emre Ugur, Matej Hoffmann, Lorenzo Jamone, Takayuki Nagai, Benjamin Rosman, Toshihiko Matsuka, Naoto Iwahashi, Erhan Oztop, Justus Piater, Florentin Wörgötter, Symbol Emergence in Cognitive Developmental Systems: A Survey, IEEE Transactions on Cognitive and Developmental Systems, 11(4), pp.494-516, 2019. Symbol emergence in robotics (SER)
10. Multimodal Categorization and Lexical Acquisition by Cognitive Robots [Nakamura+ 2009-] 2024/10/26 10 Takaya Araki, Tomoaki Nakamura, Takayuki Nagai, Shogo Nagasaka, Tadahiro Taniguchi, Naoto Iwahashi. Online Learning of Concepts and Words Using Multimodal LDA and Hierarchical Pitman-Yor Language Model. IEEE/RSJ International Conference on Intelligent Robots and Systems 2012 (IROS 2012), 1623-1630 .(2012)
11. Integration of Multimodal LDA with Nonparametric Bayesian Unsupervised Word Segmentation for Multimodal Object Categorization and Lexical Acquisition [Nakamura+ 2014] T. Nakamura, T. Nagai, K. Funakoshi, S. Nagasaka, T. Taniguchi and N. Iwahashi, "Mutual learning of an object concept and language model based on MLDA and NPYLM," 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014, pp. 600-607, doi: 10.1109/IROS.2014.6942621. Observations Visual Auditory Haptic Speech By integrating multimodal sensory information and audio signals, the robot was able to form categories, discover words from the sound signals, and obtain a grounded dictionary.
12. Symbol emergence systems [Taniguchi+ 2016] 記号創発システム Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh, Symbol Emergence in Robotics: A Survey, Advanced Robotics, 30(11-12) pp.706-728, 2016. DOI:10.1080/01691864.2016.1164622 Representation learning by individuals Symbol emergence by groups 12
13. Metropolis-Hastings naming game [Taniguchi+ 2023] Outline 1. Perception: Speaker and Listener agents (Sp and Li) observe the d-th object and infer their internal representations (assuming joint attention). 2. Communication: Speaker tells the name of the object probabilistically. The Listener determines if it accepts the naming with a certain probability depending on its belief state. 3. Learning: After the communication is performed, the Listener updates its internal parameters for representation learning and naming. 4. Turn taking: Speaker and Listener alternate their roles and go back to 1. Observation o u Semiotic Communication Representation learning Object Agent A Agent B Internal representations Speaker utters a sign as sampling Listener judges if it accepts the sign Observation Sign  Taniguchi, T., Yoshida, Y., Matsui, Y., Le Hoang, N., Taniguchi, A., & Hagiwara, Y.. Emergent communication through Metropolis-Hastings naming game with deep generative models. Advanced Robotics, 37(19), 1266-1282. (2023)  Yoshinobu Hagiwara , Hiroyoshi Kobayashi, Akira Taniguchi and Tadahiro Taniguchi, Symbol Emergence as an Interpersonal Multimodal Categorization, Frontiers in Robotics and AI, 6(134), pp.1-17, 2019 The ratio of how much the other person's name and the name you assumed yourself match your own beliefs.
14. MH Naming Game is a Decentralized MCMC Bayesian Inference Agent A Agent B Agent A Agent B (2) Sample from a proposal distribution (3) Judge by an acceptance rate , where (1) Sample from the posterior distribution (4) Update parameters ) Decomposition Composition If the acceptance decision is made with the probability the naming game is equivalent to Metropolis-Hastings algorithm, mathematically. It is guaranteed that signs are shared, and categories are formed among agents to approximate posterior distribution 𝑃𝑃(𝑤𝑤, 𝑧𝑧, 𝜃𝜃, 𝜙𝜙 |𝑜𝑜) (given the two agents' observations).
15. Collective Predictive Coding Hypothesis Sign Agent 1 Agent 2 Agent K Communication (Sampling and acceptance) Inference Representation learning Collective representation learning Internal representation Observation Decomposition Inference Y z Inference (Inference model) Generation (Generative/predictive model) Observation Latent variable Y z Inference (Inference model) Generation (Generative/predictive model) Observation Latent variable Y z Inference (Inference model) Generation (Generative/predictive model) Observation Latent variable Taniguchi, T. (2023, August 15). Collective Predictive Coding Hypothesis: Symbol Emergence as Decentralized Bayesian Inference. PsyArXiv Preprints, https://doi.org/10.31234/osf.io/d2ty6 谷口 忠大, 自由エネルギー原理と記号・言語創発—集合的予測符号化から大規模言語モデルまで—, 人工知能, 2023, 38 巻, 6 号, p. 810-817, https://doi.org/10.11517/jjsai.38.6_810 • Could symbol/language emergence in human societies be viewed as collective predictive coding? • Language games that promote symbol emergence can be viewed as decentralized Bayesian inference. • The social version of free energy principle. 15
16. Action Perception Internal representations (World model) Environment (World) From Predictive Coding (World Model Learning) to Collective Predictive Coding (Symbol Emergence) 𝑃𝑃(𝑍𝑍𝑖𝑖 |𝑋𝑋𝑖𝑖 ) Internal representations
17. Action Perception Internal representations (World model) Environment (World) From Predictive Coding (World Model Learning) to Collective Predictive Coding (Symbol Emergence) 𝑃𝑃 𝑍𝑍𝑖𝑖 𝑋𝑋𝑖𝑖 𝑖𝑖 Internal representations
18. Action Perception Internal representations (World model) Language (Emergent symbol system) Environment (World) Utterance Interpretation Constraint Organization From Predictive Coding (World Model Learning) to Collective Predictive Coding (Symbol Emergence) 𝑃𝑃 𝑊𝑊, {𝑍𝑍 } {𝑋𝑋 } Internal representations External representations
19. Scientific activity is analogous to the CPC Action Perception Internal representations (World model) Language (Emergent symbol system) Environment (World) Utterance Interpretation Constraint Organization  Descriptive scientific knowledge is also a symbol system.  Scientific activity is the collective formation of explicit knowledge (i.e., external representations).  The typical understanding of the goal of science is to model the world in order to improve our predictive abilities. Let's look at science activities through the lens of the CPC.
20. Collective Predictive Coding as Model of Science: Formalizing Scientific Activities Towards Generative Science [Taniguchi+ 2024]  Collective predictive coding, which encodes a model of the world as explicit knowledge through distributed interactions, seems to be an appropriate model for scientific activity.  CPC-MS = Collective Predictive Coding as Model of Science 20 Taniguchi, T., Takagi, S., Otsuka, J., Hayashi, Y., & Hamada, H. T. (2024). Collective Predictive Coding as Model of Science: Formalizing Scientific Activities Towards Generative Science. arXiv preprint arXiv:2409.00102. Scientific knowledge
21. MH naming game and scientific discussion  Scientific discussion, especially the peer review process, is structurally similar to the MH (Mutual Hypothesis) naming game.  The overall process of scientific activities can possibly be viewed as decentralized Bayesian inference, in the same way as the MH naming game.  This leads us to the idea of CPC as a model of science (CPC-MS). Observation o u Semiotic Communication Representation learning Object Agent A Agent B Internal representations Speaker utters a sign as sampling Listener judges if it accepts the sign Observation Sign Taniguchi, T., Takagi, S., Otsuka, J., Hayashi, Y., & Hamada, H. T. (2024). Collective Predictive Coding as Model of Science: Formalizing Scientific Activities Towards Generative Science. arXiv preprint arXiv:2409.00102.
22. Conclusion  The Collective Predictive Coding (CPC) hypothesis provides a computational framework for understanding symbol emergence in society, viewing it as a form of decentralized Bayesian inference through language games.  Scientific activities share striking structural similarities with symbol emergence systems, particularly in how knowledge is proposed, evaluated, and collectively refined through processes like peer review.  Collective Predictive Coding as Model of Science (CPC-MS) extends the CPC framework to formalize scientific activities, offering new perspectives on how scientific knowledge emerges and evolves through collective effort.
23. Information Email： taniguchi@i.kyoto-u.ac.jp X (personal): @tanichu 谷口忠大（編）『記号創発システム論ー 来るべきAI共生社会の「意味」理解にむけ て（ワードマップ） 』新曜社（2024）