From Whole-Brain Architecture to General-Purpose Intelligent Robots

The Third International Whole Brain Architecture Workshop in ICONIP2025
2025/11/24

Main Topic: Whole Brain Probabilistic Generative Model: WB-PGM
– An Integrated Cognitive Model Learned from the Brain and Implementable in Robots

Contents
– Recent Trends in Robotics
– How to construct WB-PGM: PGM, SERKET
– Individual Research Examples Toward WB-PGM Implementation
– Construction method of Brain Reference Architecture (BRA)
– Examples of BRA-based PGMs and robot implementation
– Current Challenges and Pathways to Implementation

Transcript

1. Akira Taniguchi Ritsumeikan University The Third International Whole Brain Architecture Workshop in ICONIP2025 2025/11/24 From Whole-Brain Architecture to General-Purpose Intelligent Robots
2. Self-introduction Name: Akira Taniguchi Position: Lecturer Affiliation: College of Information Science and Engineering, Ritsumeikan University Career History ◆ 2018.3: Graduate School of Information Science and Engineering, Ritsumeikan University; PhD (Doctor of Engineering) ◆ 2017.4–2019.3: Japan Society for the Promotion of Science (JSPS) Research Fellow (DC2, PD) ◆ 2019.4–2022.3: Specially Appointed Assistant Professor, Ritsumeikan University ◆ 2022.4–Present: Lecturer, Ritsumeikan University 2 Cognitive Developmental Systems Emergent Communication Symbol Emergence in Robotics Intelligent Service Robots Brain-Inspired AI Brain Reference Architecture Driven Development
3. An Integrated Cognitive Model Bridging Neuroscience and Robotics • Main Topic: Whole Brain Probabilistic Generative Model：WB-PGM – An Integrated Cognitive Model Learned from the Brain and Implementable in Robots • Contents – Recent Trends in Robotics – How to construct WB-PGM • PGM, SERKET – Individual Research Examples Toward WB-PGM Implementation – Construction method of Brain Reference Architecture (BRA) – Examples of BRA-based PGMs and robot implementation – Current Challenges and Pathways to Implementation 3
4. Cognitive Models in the Real World • Human cognitive functions are realized through the whole-brain architecture, physical constraints by body, and environmental interaction. • To realize cognitive functions, the following are crucial: – (1) Building cognitive models referencing the brain's overall function and structure – (2) Implementing these cognitive models onto robots to drive them – (3) Validating them in the real world • A body capable of actively and autonomously exploring physical and social environments is necessary, and using robots capable of acting in the real world is effective. 4 Brain Environ ment Body
5. Recent Trends in Robotics • Robot foundation models / Vision, Language, and Action Models for Robots – Large amounts of data, large models, high computational costs – Still limitations to doing everything with a single model – Very few models that reference the whole brain 5 1. A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” arXiv [cs.RO], Jul. 28, 2023. [Online]. Available: http://arxiv.org/abs/2307.15818 2. https://robotics-transformer-x.github.io/
6. Whole brain Probabilistic Generative Model (WB-PGM) • Purpose – Building a humanlike integrative artificial cognitive system, that is, an artificial general intelligence (AGI). – Developing a cognitive architecture using probabilistic generative models (PGMs) to fully mirror the human cognitive system. • Question – What set of cognitive modules should be implemented? – How can they be integrated to enable them to work together? • Two approaches: – (1) brain-inspired AI: learning human brain architecture to build human-level intelligence and – (2) PGM-based cognitive system: developing an integrative cognitive system for developmental robots by integrating PGMs, a promising approach. 6 Taniguchi, T., Yamakawa, H., Nagai, T., Doya, K., Sakagami, M., Suzuki, M., Nakamura, T., & Taniguchi, A. (2022). Whole brain Probabilistic Generative Model toward Realizing Cognitive Architecture for Developmental Robots. In Neural Networks. http://arxiv.org/abs/2103.08183
7. Probabilistic generative model (PGM) • Models the process by which data is generated • Captures statistical properties of data by assuming probability distributions • Graphical models allow visualization of model structure • Graphical Model – Dependency relationships among variables are represented by a graph structure – Nodes: random variables – Edge: Dependency relationship 7 Generative model 𝑥𝑖 ~ 𝑝 𝑥𝑖 𝜃 = 𝑝𝜃 𝑥𝑖 𝑦𝑖 ~ 𝑝 𝑦𝑖 𝑥𝑖 , 𝜃) = 𝑝𝜃 𝑦𝑖 𝑥𝑖 ) Inference model 𝑞 𝑥𝑖 𝑦𝑖 , 𝜙 = 𝑞𝜙 𝑥𝑖 |𝑦𝑖 Generative Inference Both（VAE）
8. Why a probabilistic generative model? • Graphical model allows easy visualization of structures • Easy integration of PGMs （by Neuro-SERKET framework [Taniguchi 2020] ） ➢ To Whole-Brain PGM • Supervised learning, unsupervised learning, and reinforcement learning can all be interpreted as inferences within PGM. • Discussed and theorized in neuroscience and other fields – Bayesian brain hypothesis [Doya 2007] – Free Energy Principle/Predictive Coding Hypothesis [Friston 2012] – World models 8 [Taniguchi 2020] Tadahiro Taniguchi, Tomoaki Nakamura, Masahiro Suzuki, Ryo Kuniyasu, Kaede Hayashi, Akira Taniguchi, Takato Horii, and Takayuki Nagai, “Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models”, New Generation Computing, Jan. 2020. [Doya 2007] Doya, K., Ishii, S., Pouget, A., Rao, R.P.N., Bayesian Brain: Probabilistic Approaches to Neural Coding. MIT Press. 2007. [Friston 2012] Friston, Karl, and Ping Ao. "Free energy, value, and attractors.“ Computational and mathematical methods in medicine 2012. ↑By Prof. Tadahiro Taniguchi
9. Neuro-SERKET: Development of Integrative Cognitive System Through the Composition of Deep Probabilistic Generative Models [Taniguchi+ 20] 9 • Nakamura T, Nagai T and Taniguchi T, SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model. Front. Neurorobot. 12:25. (2018) doi: 10.3389/fnbot.2018.00025 • Taniguchi, T., Nakamura, T., Suzuki, M. et al. Neuro-SERKET: Development of Integrative Cognitive System Through the Composition of Deep Probabilistic Generative Models. New Gener. Comput. 38, 23–48 (2020). https://doi.org/10.1007/s00354-019-00084-w 1. Belief propagation 2. SIR 3. MH  Connecting cognitive modules developed as probabilistic generative models and letting them work together as a single unsupervised learning system.  This enables the transfer of probabilities between modules, ensuring theoretical consistency.  Neuro-SERKET supports deep generative models, i.e., VAE, as well. Enables efficient development of complex cognitive systems!
10. • There are three types of integration. 1. In module 1, 𝑃(𝑧|𝑥) is computed. 2. 𝑃(𝑧|𝑥) is sent to module 2. 3. In module 2, the probability distribution 𝑃(𝑧|𝑦), which represents the relationships between 𝑥 and 𝑦, is estimated using 𝑃(𝑧|𝑥). 4. 𝑃(𝑧|𝑦) is sent to module 1. 5. In module 1, the latent variable 𝑧 is estimated, and the parameters of 𝑃(𝑥|𝑧) are updated. Mathematical Foundations of Graphical Model Integration 10 Head-to-Tail Tail-to-Tail Head-to-Head Head-to-Tail Decomposition
11. SERKET-SDE 11 El Hafi, Lotfi & Zheng, Youwei & Shirouzu, Hiroshi & Nakamura, Tomoaki & Taniguchi, Tadahiro. (2023). Serket-SDE: A Containerized Software Development Environment for the Symbol Emergence in Robotics Toolkit. 1-6. 10.1109/SII55687.2023.10039424. • Provides a bridge connecting SERKET and ROS • Enables independent development of individual modules • Facilitates module reconfiguration within the model
12. Taniguchi, T., Nakamura, T., Suzuki, M. et al. Neuro-SERKET: Development of Integrative Cognitive System Through the Composition of Deep Probabilistic Generative Models. New Gener. Comput. 38, 23–48 (2020). https://doi.org/10.1007/s00354-019-00084-w Example: Unsupervised categorization of image and speech [Taniguchi+ 2020]
13. SpCoSLAM [Taniguchi 17] This model integrates SLAM, GMM, multimodal place categorization, lexical acquisition and as one PGM. 13 [Taniguchi 17] Taniguchi, A., et al. : Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping, IEEE/RSJ IROS, pp. 811-818 (2017) Position distribution (Gaussian distribution) 1 − t x  x t  x t+1 t z t u t C t i  l    0 0 , m 0 0 , V 1 − t z 1 − t u 1 + t z 1 + t u  LM t y AM t S  l  t f  ∞ ∞ m l W k  k  Simultaneous localization and mapping (SLAM) Place Image Words Lexical acquisition Multimodal place categorization “Third table” “Meeting space”
14. Prototype of WB-PGM [Miyazawa+ 2019] 14 Miyazawa, K., Aoki, T., Horii, T., & Nagai, T.: Integration of Multiple Generative Modules for Robot Learning. Workshop on Deep Probabilistic Generative Models for Cognitive Architecture in Robotics (in IROS2019). (2019) Miyazawa K, Horii T, Aoki T and Nagai T (2019) Integrated Cognitive Architecture for Robot Learning of Action and Language. Front. Robot. AI 6:131. • Integration of representation learning, language learning, and reinforcement learning modules using Neuro-SERKET • Designed based on rough brain functions and connections at rough brain region levels • Next step, it is important to connect this to BRA- driven development
15. Construction method of Brain Reference Architecture (BRA) • Investigate neuroscience findings and organize them by function of the region of interest. • Define the top-level functions and assign variables corresponding to each function to construct a probabilistic graphical model. 15 Yamakawa, H. (2021). The whole brain architecture approach: Accelerating the development of artificial general intelligence by referring to the brain. In Neural Networks. http://arxiv.org/abs/2103.06123
16. GIPA: Generation-inference process allocation • Transform anatomical structure of brain regions into dependencies of variables in a generative model • Limitations of PGM – Need to be Directed acyclic graphs (cannot create loops) • Basic GIPA procedure 1. Separate edges into generative and inferential models (using Table 1 as the main reference) 2. Introduce a representation of next-time generation (𝚫𝒕) 16 WB-PGM論文より t←t+1
17. Egocentric visual information (Subjective viewpoint) Allocentric visual information (Objective viewpoint) Self-localization (Path integration) Motion information (rotation speed) Predictive information Refinement of past estimates Abstracted visual information of place (representation learning) Integrate two pieces of information from LEC and MEC 17 HF-PGM： Hippocampal formation-inspired probabilistic generative model Akira Taniguchi, Ayako Fukawa, Hiroshi Yamakawa, "Hippocampal formation- inspired probabilistic generative model", Neural Networks, Vol.151, pp.317- 335, 2022. Top-level function: (i) Self-localization (ii) Place categorization
18. Neuroscientific Findings from the HF 18 Navigation • Animals select an appropriate path to reach its destination while obtaining environmental information. Cognitive map • Animals form an environmental through exploration and use the map to guide their behavior. Self-localization (including Path integration) • As a prerequisite for navigation and cognitive map, the animals are always aware of their current location. Place cells • Fires when in certain parts of the environment • Found in the hippocampus of rodents Grid cells • Fires when in certain parts of the environment • Found in the medial entorhinal cortex: MEC of rodents • Believed to be involved in the execution of PI In mammals, the HF plays an important role in navigation and the formation of cognitive maps. It is believed that place and grid cells involved in the execution of spatial cognitive processing. Grieves, Roddy M., and Kate J. Jeffery. "The representation of space in the brain." Behavioural processes 135 (2017): 113-131.
19. Implementation of a HF-inspired PGM Self-localization task ✔ Improved Self-localization performance compared to the conventional models ✔ Resolving the Robot Kidnapping Problem Takeshi Nakashima, Shunsuke Otake, Akira Taniguchi, Katsuyoshi Maeyama, Lotfi El Hafi, Tadahiro Taniguchi, Hiroshi Yamakawa, "Hippocampal formation-inspired global self-localization: quick recovery from the kidnapped robot problem from an egocentric perspective", Frontiers in Computational Neuroscience, Vol. 18, May 2024. 19 World model (RSSM) ＋ Metric map (SLAM) ⇒RSSM+SLAM models
20. AF24Hippocampus-Amygdala’s BIF 20 We have developed a BRA data by integrating “TM24Amygdala ver4” and “TN24HippocampalFormation”. Amygdala’s BRA Akira Taniguchi*, Atsushi Fujii, Takeshi Nakashima, Tatsuya Miyamoto, Yoshimasa Tawatsuji, Hiroshi Yamakawa, "Data for Brain Reference Architecture of AF24Hippocampus-Amygdala: Integrating BRA for Spatial Cognition and Fear Conditioning", The 2nd International Whole Brain Architecture Workshop, 7 pages, Feb 2025. In Tokyo, Japan. DOI: https://doi.org/10.51094/jxiv.1058 Hippocampus’s BRA
21. AF24Hippocampus-Amygdala’s BIF 21 By constructing a computational model from this BIF, we will be able to realize emotion-adaptive navigation to implement to the mobile robot. Akira Taniguchi*, Atsushi Fujii, Takeshi Nakashima, Tatsuya Miyamoto, Yoshimasa Tawatsuji, Hiroshi Yamakawa, "Data for Brain Reference Architecture of AF24Hippocampus-Amygdala: Integrating BRA for Spatial Cognition and Fear Conditioning", The 2nd International Whole Brain Architecture Workshop, 7 pages, Feb 2025. In Tokyo, Japan. DOI: https://doi.org/10.51094/jxiv.1058
22. Double articulation analysis (DAA) is a function that analyzes the double articulation structure (phonemes and words) in spoken language. (i) Investigation of the anatomical structure and function involved in the DAA. (ii) Construction of a novel probabilistic generative model (PGM) consistent with the anatomical structure and function based on existing DAA models. Akira Taniguchi, Maoko Muro, Hiroshi Yamakawa, Tadahiro Taniguchi, "Brain-inspired probabilistic generative model for double articulation analysis of spoken language", IEEE International Conference on Development and Learning (ICDL), Sep. 2022. 22 Speech Syllabification 音素 保存 Wordization Tactile Vision Prosody Voice discrimination Non- voice Categori zation Integration Acoustic features Double articulation analysis of spoken language
23. 画像 単語分 割 触覚 音 かえる の ぬいぐるみ ふわふ わ 画像 触覚 音 かえる の ぬいぐるみ ふわふ わ Vision Co-occurrence information Tactile Auditory かえるの ぬいぐるみ ちいさい ぼーるだよ Speech regarding each object これはぬい ぐるみだよ これはまる いぼーる Phoneme sequence Word sequence A A A B D A C B C Inference CB AB BD AC Phoneme/word discovery “AB” “ACDF” “BE” Category 1 “XYZ” “ST” “PQ” Category 2 Multimodal object categorization Mutual utilization Lexical Acquisition Using Multimodal Observations from Robots 23 Ex.2: “おいしい|ね” Ex.1: “おかし|です” Akira Taniguchi, Hiroaki Murakami, Ryo Ozaki, Tadahiro Taniguchi, "Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cues", IEEE Transactions on Cognitive and Developmental Systems, Volume: 15, Issue: 4, pp.1825 - 1840, Aug. 2023.
24. Computational Models of Brain Region Modules Feedback on Verification Results Refer to connections between brain regions Module reconfiguration and replacement Brain-Reference Architecture Driven Development Model Integration by SERKET + Multimodal large language models, Foundation models Whole-Brain Robotics Evaluation through real-world tasks
25. Current Challenges and Pathways to Implementation • Refresh of individual modules – Adopt deep generative models and Transformer-based models • Extension of SERKET-SDE – Synchronous and asynchronous computation between sensor data and modules – Enable easy reconfiguration of modules to compare and evaluate more suitable models – Enable real-time communication between models and ROS for sequential learning and inference • Linking with the WBAI roadmap – Implementing and validating on real-world robots, referencing the constructed Whole Brain Reference Architecture – Considering Scaling problems in development • Autonomy, Brain Dynamics and Development 25
26. Concept Formation using Multimodal Transformer [Miyazawa 23] • Representation learning through masked prediction learning – Categories were formed as internal representations, suggesting that object concepts were formed by the Transformer. 26 [Miyazawa 23] K. Miyazawa and T. Nagai, “Concept formation through multimodal integration using multimodal BERT and VQ-VAE,” Adv. Robot., vol. 37, no. 4, pp. 281–296, Feb. 2023. [Nakamura 18] T. Nakamura and T. Nagai, “Ensemble-of-Concept Models for Unsupervised Formation of Multiple Categories,” IEEE TCDS, vol. 10, no. 4, pp. 1043–1057, Dec. 2018. Transformer Transformer this is a stuffed animal. its color is pink. … Frozen VQ-VAE Encoder Tokenize Multimodal Representation Multimodal Object Dataset 165 [Nakamura 2018] Vision Audio Haptics Text
27. Expansion of SERKET: Integration beyond probabilistic models • Neural Networks – Neural Conversion Adapter [Nakamura 24] • Variational Inference [Submitted] • Contrastive Learning (SimSiam) [Hoang 24] • Vision-Language models (ProbVLM, CLIP) [Matsui 24] • Latent Diffusion Model [Submitted] • RSSM+SLAM (Particle Filter) [Nakashima 24] • Gaussian Process [Saito 24] • POMDP, Reinforcement Learning [Nakamura 23] • World models [Under preparation, Nomura 25] 27 • Nakamura, T., Suzuki, M., Taniguchi, A., & Taniguchi, T. (2024). Symbol emergence and representation learning by integrating GPLVM and neural network. Journal of the Robotics Society of Japan (Letters). (in Japanese) • Hoang, N. L., Taniguchi, T., Tianwei, F., & Taniguchi, A. (2024). Simsiam naming game: A unified approach for representation learning and emergent communication. arXiv preprint arXiv:2410.21803. • Matsui, Y., Yamaki, R., Ueda, R., Shinagawa, S., & Taniguchi, T. (2025). Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inference. arXiv preprint arXiv:2504.09620. • Nakashima, T., Otake, S., Taniguchi, A., Maeyama, K., El Hafi, L., Taniguchi, T., & Yamakawa, H. (2024). Hippocampal formation-inspired global self-localization: quick recovery from the kidnapped robot problem from an egocentric perspective. Frontiers in Computational Neuroscience, 18. • Saito, I., Nakamura, T., Taniguchi, A., Taniguchi, T., Hayamizu, Y., & Zhang, S. (2024). Emergence of continuous signals as shared symbols through emergent communication. IEEE International Conference on Development and Learning. • Nakamura, T., Taniguchi, A., & Taniguchi, T. (2023). Control as probabilistic inference as an emergent communication mechanism in multi-agent reinforcement learning. • Nomura, K., Aoki, T., Taniguchi, T., & Horii, T. (2025). Decentralized collective world model for emergent communication and coordination. arXiv preprint arXiv:2504.03353.
28. Active SpCoSLAM （Right videos）  Active semantic mapping enhancing environmental understanding through mapping It provides autonomy by representing neuroscientific findings as inference algorithms. Autonomy: Robotic Applications of Active Inference 28 実世界環境 Learning through active exploration Camera ×16 Learning result Taniguchi, A., Tabuchi, Y., Ishikawa, T., Hafi, L. E., Hagiwara, Y., & Taniguchi, T. (2023). Active Exploration based on Information Gain by Particle Filter for Efficient Spatial Concept Formation. Advanced robotics. Advanced Robotics Best Paper Award Tomochika Ishikawa, Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro Taniguchi, "Active Semantic Mapping for Household Robots: Rapid Indoor Adaptation and Reduced User Burden", IEEE International Conference on Systems, Man, and Cybernetics (SMC), Oct. 2023. SpCoAE  Robots acquire spatial concepts by moving to “semantically unknown” locations and actively asking users questions. [Taniguchi 2022]  Theoretically, it connects to active inference in minimizing expected free energy.
29. Shoichi Hasegawa, Kento Murata, Tomochika Ishikawa, Yoshinobu Hagiwara, Akira Taniguchi, Lotfi El Hafi, Gustavo Garcia, and Tadahiro Taniguchi, "Reducing Cost of On-site Learning by Multi-robot Knowledge Integration and Task Decomposition via Large Language Models," Journal of the Robotics Society of Japan, Volume 43, Issue 7, Pages 703-706, Sep 2025. (Letter paper in Japanese)． Spatial Concept-based Prompts and Sequential Feedback for LLM-based Robot Action Planning (SpCoRAP) 29 avigation ect etection Pick Place ontrol ommand Success or Failed Fle BE Bring a cup to t e kitc en. ect Image ord Position ect name Place name ect placement PT Prompt
30. Correspondence of LLM and Brain • Does LLM only correspond to language areas? • Is there structural similarity? • Is there data efficiency and learning efficiency? 30 https://news.mit.edu/2025/large-language-models-reason-about-diverse-data- general-way-0219 Taniguchi, T., Ueda, R., Nakamura, T., Suzuki, M., & Taniguchi, A. (2024). Generative Emergent Communication: Large Language Model is a Collective World Model. arXiv preprint arXiv:2501.00226. Li, Y., Michaud, E. J., Baek, D. D., Engels, J., Sun, X., & Tegmark, M. (2025). The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4), 344. Doerig, A., Kietzmann, T.C., Allen, E. et al. High-level visual representations in the human brain are aligned with large language models. Nat Mach Intell 7, 1220– 1234 (2025). https://doi.org/10.1038/s42256-025-01072-0 mae, S ogo, and Keiko mae. “T e rain versus AI: orld-model-based versatile circuit computation underlying diverse functions in the neocortex and cere ellum.” arXiv preprint arXiv:2411.16075 (2024). https://arxiv.org/abs/2411.16075?s=09 https://ledge.ai/articles/llm_conceptual_structure_sae How can we map large language models to the brain / BRA?
31. How to correspond LLMs and WBRA? 31 LLMs (Collective world model) ⇒Collective Human Brain model? HB-LLM? Typical Human Brain (from Neuroscience findings) Taniguchi, T., Ueda, R., Nakamura, T., Suzuki, M., & Taniguchi, A. (2024). Generative emergent communication: Large language model is a collective world model. arXiv preprint arXiv:2501.00226. Reference WBRA WB-PGM How to match them? Do LLMs similar to the human brain? Can we make LLM closer to the human brain? Implementation alignment Feed Back
32. Towards further more human-like intelligence • Is the WBRA of the human brain alone sufficient to achieve human-like intelligence? • Isn't it necessary to imitate the human body as well? – Humanoid robot! 32 Brain Environ ment Body https://nvlabs.github.io/SONIC/ Unitree G1 robot iCub robot
33. THANK YOU FOR YOUR KIND ATTENTION. 33 Symbol Emergence in Robotics for Future Human-Machine Collaboration WAKATE, etc. [Ongoing and Ended] [Ended] [Ended] [Ongoing] [Ongoing]