[SII23] Inferring Place-Object Relationships by Integrating Probabilistic Logic and Multimodal Spatial Concepts

We propose a novel method that integrates probabilistic logic and multimodal spatial concepts to enable a robot to acquire the relationships between places and objects in a new environment with a few learning times. Using predicate logic with probability values (i.e., probabilistic logic) to represent commonsense knowledge of place-object relationships, we combine logical inference using probabilistic logic with the cross modal inference that can calculate the conditional probabilities of other modalities given one modality. This allows the robot to infer the place of the object to find even when it does not know the likely place of the object in the home environment. We conducted experiments in which a robot searched for daily objects, including objects with undefined places, in a simulated home environment using four approaches:
1) multimodal spatial concepts only,
2) commonsense knowledge only,
3) commonsense knowledge and multimodal spatial concepts,
and 4) probabilistic logic and multimodal spatial concepts (proposed).
We confirmed the effectiveness of the proposed method by comparing the number of place visits it took for the robot to find all the objects. We also observed that our proposed approach reduces the on-site learning cost by a factor of 1.6 over the three baseline methods when the robot performs the task of finding objects with undefined places in a new home environment.

Transcript

1. Inferring Place-Object Relationships by Integrating Probabilistic Logic and Multimodal Spatial Concepts Ritsumeikan University 〇Shoichi Hasegawa, Akira Taniguchi, Yoshinobu Hagiwara, Lotfi El Hafi, Tadahiro Taniguchi 2023 IEEE/SICE International Symposium on System Integration (SII2023) Paper Number：73 FrA1M1 Hybrid Special Session：10:30-12:00, Paper FrA1M1.1
2. Introduction 2
3. Research Background [1] H. Okada, et al. “What competitions were conducted in the service categories of the World Robot Summit?” Advanced Robotics, Vol. 33, No. 17, pp. 900–910, 2019. [2] C. Luis, et al. "Towards general purpose service robots: World Robot Summit–Partner Robot Challenge" Advanced Robotics, Vol.36, No. 17-18, pp.812-824, 2022. Services desired to be performed by robots [1, 2] • Bring me task：Deliver a specified object to a user • Tidy up task：Clean up scattered objects Cleanup Tasks [1] It’s important to understand which object is likely to be in each place, or in other words, evaluate the object existence probability in each place. Because the object existence probability in each place vary depending on environments, the robot need to learn it efficiently from data obtained in each environment. Object existence probability Object existence probability Object existence probability 3
4. Our Approach Infer where the object is living kitchen bedroom bathroom [3] R. Luc, et al. “ProbLog: A Probabilistic Prolog and its Application in Link Discovery”, International Joint Conference on Artificial Intelligence, pp. 2468–2473, 2007. [4] A. Taniguchi, et al. “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020. Contribution 1. Reduction of the cost of on-site learning in a new environment by commonsense knowledge. 2. Prediction of the place where an undefined object is located by probabilistic logical inference. Definition：Undefined objects Objects for which the probability of existence for each place has not been calculated yet. On-site learning [4] 4 Probabilistic planning Logical inference based on commonsense knowledge [3] sheep doll is in the doll category, ... doll category Look for a sheep doll Reduction of the cost of on-site learning Target Prediction of places where an undefined object exists
5. 5 Verify whether the combination of on-site learning and probabilistic logical inference can reduce the on-site learning cost for finding specific objects in a new environment. Research Question How much can the robot reduce the on-site learning cost by incorporating logical inference based on commonsense knowledge into on-site learning? Research Purpose ・・・ Search for the sheep doll！ I visited 0 rooms to find target object. I visited N rooms to find target object. Reduce the number of learnings Onsite learning times is 0. Onsite learning times is N.
6. Previous Research 6
7. On-site Learning of an Environment SpCoSLAM [4] • Simultaneous acquisition of spatial concepts, vocabularies, and a map • Sequential Bayesian estimation with particle filter [10] • Cross-modal inference is possible [4] A. Taniguchi, et.al, “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020. [9] A. Taniguchi, et al. "Autonomous planning based on spatial concepts to tidy up home environments with service robots“, Advanced Robotics, Vol.35, No.8, pp.471-489, 2021. [10] K. R. Canini, et al. “Online Inference of Topics with Latent Dirichlet Allocation”, International Conference on Artificial Intelligence and Statistics (AISTATS), Vol.9, No.1999, pp.65–72, 2009. The robot need to obtain a certain amount of observations in the environment. 7 Spatial concept formation method based on probabilistic generative model [4, 9] Definition：Spatial concepts A set of parameters on probability distributions formed by unsupervised learning based on multimodal information as places. The robot can construct statical k w “which object is likely to h p ” through observation. Gaussian dist. ⇒ Reduce on-site learning cost by using commonsense knowledge
8. Leveraging Commonsense Knowledge [5] Y. Zhang, et.al, “Efficient Dynamic Object Search in Home Environment by Mobile Robot: A Priori Knowledge-Based Approach”, IEEE Transactions on Vehicular Technology, Vol. 68, No. 10, pp. 9466–9477, 2019. [6] A. C. Hernandez, et al. "Efficient Object Search Through Probability-Based Viewpoint Selection“, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.6172-6179, 2020. Large scale image data and ontology [5] • Use the frequency of labeled objects in the image • The positional relationships between objects and furniture by symbol (isOn(Cup, Cabinet)) I ’ ff h j f h j existence probability is not given in advance. Commonsense knowledge on dataset [5, 6] 8 ⇒ Introduce probabilistic logical inference doll cup apple
9. Proposed Method 9
10. f p ma es ords e ts ser ations , I f p p , , , ・・・ ommonsense no ed e a out t e o e t e isten e pro a i it , , ositions p p 10 Overview of Our Method Receive an object word sheep doll
11. 11 ProbLog (Probabilistic Prolog) [3]：Probabilistic logic programming language for probabilistic logic [3] R. Lu , . . “ProbLog: A Probabilistic Prolog and its Application in Link Discovery”, International Joint Conference on Artificial Intelligence(IJCAI), pp. 2468–2473, 2007. How to describe knowledge in ProbLog Object category doll (pig_doll). doll(sheep_doll). doll(penguin_doll). … Explanation as logic symbol Logic symbol Expression in ProbLog Implication ← :- Disjunction ⋁ ; Not ¥+ Give a query query(exist(sheep_doll, Y)). Object existence probability 𝑃1 :: exist(pig_doll, living). 𝑃2 :: exist(pig_doll, bedroom). … The relationship between object category and place word 𝑃1 ′ :: exist(X, living); 𝑃2 ′ :: exist(X, bedroom); 𝑃3 ′ :: exist(X, kitchen); 𝑃4 ′ :: exist(X, bathroom) :- doll(X). … Probabilistic Logical Inference (Probabilistic Logic) ・・・ 0.28 0.5 0.1 0.13 By repeated resolution and unification， unification Empty clause
12. 12 Objects Bag-of-Objects Image Place word Bag-of-Words [4] A. Taniguchi, et.al, “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020. [11] J. Redmon, et al. "You Only Look Once: Unified, Real-Time Object Detection“, IEEE conference on computer vision and pattern recognition (CVPR), pp. 779-788, 2016. [12] K. He, et al. “Deep Residual Learning for Image Recognition”, IEEE conference on computer vision and pattern recognition (CVPR), pp. 770–778, 2016. Spatial Concept model based on SpCoSLAM (Online Spatial Concept Acquisition and SLAM) [4] On-site Learning (Multimodal Spatial Concepts) ResNet [12] [11] What is possible： Cross-modal inference from an object label to place regions. duck sponge The parameters of the probability distribution are learned from sequential Bayesian estimation with particle filter. Main change
13. 13 ・・・ 0.2 0.7 0.05 0.05 𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔 𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔 𝑃 𝑄 ෍ 𝐹 ڂ 𝑅 |=𝑄 ෑ 𝑓 ∈𝐹 𝑃 𝑓 ෑ 𝑓 ∉𝐹 − 𝑃(𝑓) ・・・ 0.28 0.5 0.1 0.13 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑤𝑡 𝑜𝑡 ) 𝜂𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔 − 𝜂 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 ・・・ 0.35 0.3 0.2 0.15 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 𝑃 𝑤𝑡 𝑜𝑡 Θ) ∝ ෍ 𝐶𝑡 𝑃 𝑤𝑡 𝑊𝐶𝑡 ) 𝑃 𝑜𝑡 𝜉𝐶𝑡 ) 𝑃(𝐶𝑡 |𝜋) Integration of Probability Distributions Probabilistic logical inference (probabilistic logic) On-site learning (multimodal spatial concepts) D v f f f B ’ h by using parameters (𝜉 𝑊 𝜋) of probability distribution obtained by on-site learning. Q Query F Facts R Rules 𝑤𝑡 Place word 𝑜𝑡 Object label Θ A set of parameters (𝜉 𝑊 𝜋) Constructed by summing possible solutions to a query based on distribution semantics [13] [13] T. S , “A Statistical Learning Method for Logic Programs with Distribution Semantics”, International Conference on Logic Programming (ICLP), 1995. Weighted averaging
14. 14 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑖𝑡 𝑜𝑡 ) ෍ 𝑤 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 𝑖𝑡 𝑤𝑡 ) 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑤𝑡 𝑜𝑡 ) 1. Given the object name, calculate the probability of the region where the object exists. 2. Planning in order of probability 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑖𝑡 𝑜𝑡 ) 𝑖𝑡 Index of position dist. 𝐶𝑡 Index of spatial concepts 𝑤𝑡 Word information 𝑜𝑡 Object information ෍ 𝑤 ෍ 𝐶 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 𝑖𝑡 𝐶𝑡 ) 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚 𝐶𝑡 𝑤𝑡 ) 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑 𝑤𝑡 𝑜𝑡 ) Probabilistic Planning
15. Experiment 15
16. Experiment Overview 16 Purpose Verify how much the on-site learning cost can be reduced by leveraging logical inference based on commonsense knowledge when searching for objects containing undefined objects. 1. Construct commonsense knowledge 2. Conduct on-site learning 3. Conduct object search task 4. Evaluate with comparison methods Definition：On-site learning cost The number of room visits in learning Procedure On-site learning Commonsense knowledge Probabilistic logical inference SpCoSLAM ✓ Prior* ✓ SpCoSLAM + Prior ✓ ✓ SpCoSLAM + ProbLog (Proposed) ✓ ✓ ✓ Comparison Methods *Prior A method that uses the prior knowledge of smoothed questionnaire results as commonsense knowledge
17. 17 Questionnaire as the arrangement of the objects (63 men and women (20~60’s)) Ex. Objects in the toiletries category Ex. towel Construction of Commonsense Knowledge Place names Place names The object existence probability The object existence probability
18. 18 (Under condition learned what name each area on the occupancy grid map is called) The objects existence probability at each place learned by SpCoSLAM Observation data • Position (XY) • Image • The number of detected objects Learn by 1 observation data at 1 place. (2 visits for each place, total is 12 visits.) 1. living 3. kitchen 2. bedroom 4. bathroom Procedures for On-site Learning Undefined objects Simulation Environment ×5 Place categorization by SpCoSLAM Object detection by YOLO Capture the images
19. 19 Explanation of the task Bad (Room visits are 3.) Good (Room visits are 1.) ×16 ×16 Target Object Target Object Object Search Task Give an object name to a robot Conducted as 24 objects Onsite learning times is 0. ・・・ Look for the sheep doll！ I visited 3 rooms to find target object. I visited 1 room to find target object. Onsite learning times is 12.
20. 20 Our：0.88 (21 objects / 24 objects) (Rooms visited during learning is 5 times) SpCoSLAM + Prior：0.92 (22 objects / 24 objects) (Room visited during learning is 8 times) 𝑁 ෍ 𝑖 𝑁 𝑛𝑖 𝑁 (𝑁 4)：T e num er of o e ts p a ed in t e en ironment 𝑛𝑖 ： The number of objects found by the robot Reduced the on-site learning cost by a factor of 1.6 over more than other approaches. Experimental Result Proportions of objects found by the robot in one search (Satisfaction threshold was set at 80%.)
21. 21 Prediction of Places Where an Undefined Object Exists by Our Model Clean up task Leveraging knowledge of objects in the same category as target by probabilistic logical inference. Target object Duck shaped sponge
22. Conclusion 22
23. 23 Summary • Proposed a model that integrates probabilistic logic and multimodal spatial concept model • Experiment：Object search task with undefined objects Our approach showed that reducing the on-site learning cost by a factor of 1.6 over more than other approaches. Future Works Conclusion Leveraging probabilistic distribution from Large Language Model (e.g., PaLM [14], GPT-3 [15]). Example of Tasks For an object, answer where it is placed in the house. Object: plate Location: kitchen Query Object: apple Location: https://gpt3demo.com/apps/openai-gpt-3-playground [14] A. Chowdhery, et al. "PaLM: Scaling Language Modeling with Pathways" arXiv preprint arXiv:2204.02311, 2022. [15] B. Tom, et al. "Language Models are Few-Shot Learners“, Advances in neural information processing systems, Vol.33, pp1877-1901, 2020. Large Language Model