[SII23] Inferring Place-Object Relationships by Integrating Probabilistic Logic and Multimodal Spatial Concepts

Transcript

1. Inferring Place-Object Relationships by Integrating
   Probabilistic Logic and Multimodal Spatial Concepts
   Ritsumeikan University
   〇Shoichi Hasegawa, Akira Taniguchi, Yoshinobu Hagiwara,
   Lotfi El Hafi, Tadahiro Taniguchi
   2023 IEEE/SICE International Symposium on System Integration (SII2023)
   Paper Number：73
   FrA1M1 Hybrid Special Session：10:30-12:00, Paper FrA1M1.1
   

   View Slide

2. Introduction
   2
   

   View Slide

3. Research Background
   [1] H. Okada, et al. “What competitions were conducted in the service categories of the World Robot Summit?” Advanced Robotics, Vol. 33, No. 17, pp. 900–910, 2019.
   [2] C. Luis, et al. "Towards general purpose service robots: World Robot Summit–Partner Robot Challenge" Advanced Robotics, Vol.36, No. 17-18, pp.812-824, 2022.
   Services desired to be performed by robots [1, 2]
   • Bring me task：Deliver a specified object to a user
   • Tidy up task：Clean up scattered objects
   Cleanup Tasks [1]
   It’s important to understand which object is likely to be in each place,
   or in other words, evaluate the object existence probability in each place.
   Because the object existence probability in each place vary depending on environments,
   the robot need to learn it efficiently from data obtained in each environment.
   Object existence
   probability
   Object existence
   probability
   Object existence
   probability
   3
   

   View Slide

4. Our Approach
   Infer where the object is
   living
   kitchen
   bedroom
   bathroom
   [3] R. Luc, et al. “ProbLog: A Probabilistic Prolog and its Application in Link Discovery”, International Joint Conference on Artificial Intelligence, pp. 2468–2473, 2007.
   [4] A. Taniguchi, et al. “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020.
   Contribution
   1. Reduction of the cost of on-site learning in a new environment by commonsense knowledge.
   2. Prediction of the place where an undefined object is located by probabilistic logical inference.
   Definition：Undefined objects
   Objects for which the probability of existence
   for each place has not been calculated yet.
   On-site learning [4]
   4
   Probabilistic planning
   Logical inference based on
   commonsense knowledge [3]
   sheep doll is in the doll category, ...
   doll category
   Look for
   a sheep doll
   Reduction of the cost of
   on-site learning
   Target
   Prediction of places where
   an undefined object exists
   

   View Slide

5. 5
   Verify whether the combination of on-site learning and probabilistic logical inference
   can reduce the on-site learning cost for finding specific objects in a new environment.
   Research Question
   How much can the robot reduce the on-site learning cost by incorporating logical inference based
   on commonsense knowledge into on-site learning?
   Research Purpose
   ・・・
   Search for the sheep doll！
   I visited 0 rooms to
   find target object.
   I visited N rooms to
   find target object.
   Reduce the number of learnings
   Onsite learning times is 0. Onsite learning times is N.
   

   View Slide

6. Previous Research
   6
   

   View Slide

7. On-site Learning of an Environment
   SpCoSLAM [4]
   • Simultaneous acquisition of spatial concepts, vocabularies,
   and a map
   • Sequential Bayesian estimation with particle filter [10]
   • Cross-modal inference is possible
   [4] A. Taniguchi, et.al, “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020.
   [9] A. Taniguchi, et al. "Autonomous planning based on spatial concepts to tidy up home environments with service robots“, Advanced Robotics, Vol.35, No.8, pp.471-489, 2021.
   [10] K. R. Canini, et al. “Online Inference of Topics with Latent Dirichlet Allocation”, International Conference on Artificial Intelligence and Statistics (AISTATS), Vol.9, No.1999, pp.65–72, 2009.
   The robot need to obtain a certain amount of observations in the environment.
   7
   Spatial concept formation method based on
   probabilistic generative model [4, 9]
   Definition：Spatial concepts
   A set of parameters on probability
   distributions formed by unsupervised learning
   based on multimodal information as places.
   The robot can construct statical
   k w “which object is likely to
   h p ” through observation.
   Gaussian
   dist.
   ⇒ Reduce on-site learning cost by using commonsense knowledge
   

   View Slide

8. Leveraging Commonsense Knowledge
   [5] Y. Zhang, et.al, “Efficient Dynamic Object Search in Home Environment by Mobile Robot: A Priori Knowledge-Based Approach”, IEEE Transactions on Vehicular Technology, Vol. 68, No. 10, pp. 9466–9477, 2019.
   [6] A. C. Hernandez, et al. "Efficient Object Search Through Probability-Based Viewpoint Selection“, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.6172-6179, 2020.
   Large scale image data and ontology [5]
   • Use the frequency of labeled objects in the image
   • The positional relationships between objects and furniture by symbol (isOn(Cup, Cabinet))
   I ’ ff h j f h j
   existence probability is not given in advance.
   Commonsense knowledge on dataset [5, 6]
   8
   ⇒ Introduce probabilistic logical inference
   doll
   cup
   apple
   

   View Slide

9. Proposed Method
   9
   

   View Slide

10. f
    p
    ma es
    ords
    e ts
    ser ations
    ,
    I f p
    p
    , ,
    ,
    ・・・
    ommonsense no ed e a out
    t e o e t e isten e pro a i it
    , ,
    ositions
    p p
    10
    Overview of Our Method
    Receive an
    object word
    sheep doll
    

    View Slide

11. 11
    ProbLog (Probabilistic Prolog) [3]：Probabilistic logic programming language for probabilistic logic
    [3] R. Lu , . . “ProbLog: A Probabilistic Prolog and its Application in Link Discovery”, International Joint Conference on Artificial Intelligence(IJCAI), pp. 2468–2473, 2007.
    How to describe knowledge in ProbLog
    Object category
    doll (pig_doll). doll(sheep_doll). doll(penguin_doll). …
    Explanation as
    logic symbol
    Logic
    symbol
    Expression in
    ProbLog
    Implication ← :-
    Disjunction ⋁ ;
    Not ¥+
    Give a query
    query(exist(sheep_doll, Y)).
    Object existence probability
    𝑃1
    :: exist(pig_doll, living). 𝑃2
    :: exist(pig_doll, bedroom). …
    The relationship between object category and place word
    𝑃1
    ′ :: exist(X, living); 𝑃2
    ′ :: exist(X, bedroom); 𝑃3
    ′ :: exist(X, kitchen); 𝑃4
    ′ :: exist(X, bathroom) :- doll(X). …
    Probabilistic Logical Inference (Probabilistic Logic)
    ・・・
    0.28
    0.5
    0.1
    0.13
    By repeated resolution and unification，
    unification
    Empty clause
    

    View Slide

12. 12
    Objects
    Bag-of-Objects
    Image
    Place word
    Bag-of-Words
    [4] A. Taniguchi, et.al, “Improved and scalable online learning of spatial concepts and language models with mapping”, Autonomous Robots, Vol.44, pp927-946, 2020.
    [11] J. Redmon, et al. "You Only Look Once: Unified, Real-Time Object Detection“, IEEE conference on computer vision and pattern recognition (CVPR), pp. 779-788, 2016.
    [12] K. He, et al. “Deep Residual Learning for Image Recognition”, IEEE conference on computer vision and pattern recognition (CVPR), pp. 770–778, 2016.
    Spatial Concept model based on SpCoSLAM (Online Spatial Concept Acquisition and SLAM) [4]
    On-site Learning (Multimodal Spatial Concepts)
    ResNet [12]
    [11]
    What is possible：
    Cross-modal inference from an object label
    to place regions.
    duck sponge
    The parameters of the probability distribution are learned
    from sequential Bayesian estimation with particle filter.
    Main change
    

    View Slide

13. 13
    ・・・
    0.2
    0.7
    0.05
    0.05
    𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔
    𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔
    𝑃 𝑄
    ෍
    𝐹 ڂ 𝑅 |=𝑄
    ෑ
    𝑓 ∈𝐹
    𝑃 𝑓 ෑ
    𝑓 ∉𝐹
    − 𝑃(𝑓)
    ・・・
    0.28
    0.5
    0.1
    0.13
    𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑤𝑡
    𝑜𝑡
    )
    𝜂𝑃𝑝𝑟𝑜𝑏𝑙𝑜𝑔
    − 𝜂 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    ・・・
    0.35 0.3
    0.2 0.15
    𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    𝑃 𝑤𝑡
    𝑜𝑡
    Θ)
    ∝ ෍
    𝐶𝑡
    𝑃 𝑤𝑡
    𝑊𝐶𝑡
    ) 𝑃 𝑜𝑡
    𝜉𝐶𝑡
    ) 𝑃(𝐶𝑡
    |𝜋)
    Integration of Probability Distributions
    Probabilistic logical inference
    (probabilistic logic)
    On-site learning
    (multimodal spatial concepts)
    D v f f f B ’ h by using
    parameters (𝜉 𝑊 𝜋) of probability distribution obtained by on-site learning.
    Q Query
    F Facts
    R Rules
    𝑤𝑡
    Place word
    𝑜𝑡
    Object label
    Θ A set of parameters
    (𝜉 𝑊 𝜋)
    Constructed by summing possible solutions to a query
    based on distribution semantics [13]
    [13] T. S , “A Statistical Learning Method for Logic Programs with Distribution Semantics”, International Conference on Logic Programming (ICLP), 1995.
    Weighted averaging
    

    View Slide

14. 14
    𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑖𝑡
    𝑜𝑡
    ) ෍
    𝑤
    𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    𝑖𝑡
    𝑤𝑡
    ) 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑤𝑡
    𝑜𝑡
    )
    1. Given the object name, calculate the probability of the region where the object exists.
    2. Planning in order of probability 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑖𝑡
    𝑜𝑡
    )
    𝑖𝑡
    Index of position dist.
    𝐶𝑡
    Index of spatial concepts
    𝑤𝑡
    Word information
    𝑜𝑡
    Object information
    ෍
    𝑤
    ෍
    𝐶
    𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    𝑖𝑡
    𝐶𝑡
    ) 𝑃𝑠𝑝𝑐𝑜𝑠𝑙𝑎𝑚
    𝐶𝑡
    𝑤𝑡
    ) 𝑃𝑖𝑛𝑡𝑒𝑔𝑟𝑎𝑡𝑒𝑑
    𝑤𝑡
    𝑜𝑡
    )
    Probabilistic Planning
    

    View Slide

15. Experiment
    15
    

    View Slide

16. Experiment Overview
    16
    Purpose
    Verify how much the on-site learning cost can be reduced by leveraging logical inference based
    on commonsense knowledge when searching for objects containing undefined objects.
    1. Construct commonsense knowledge
    2. Conduct on-site learning
    3. Conduct object search task
    4. Evaluate with comparison methods
    Definition：On-site learning cost
    The number of room visits in learning
    Procedure
    On-site learning Commonsense
    knowledge
    Probabilistic logical
    inference
    SpCoSLAM ✓
    Prior* ✓
    SpCoSLAM + Prior ✓ ✓
    SpCoSLAM + ProbLog (Proposed) ✓ ✓ ✓
    Comparison Methods
    *Prior
    A method that uses the prior knowledge of smoothed
    questionnaire results as commonsense knowledge
    

    View Slide

17. 17
    Questionnaire as the arrangement of the objects (63 men and women (20~60’s))
    Ex. Objects in the toiletries category
    Ex. towel
    Construction of Commonsense Knowledge
    Place names
    Place names
    The object existence probability
    The object existence probability
    

    View Slide

18. 18
    (Under condition learned what name each area on the occupancy grid map is called)
    The objects existence probability at each place learned by SpCoSLAM
    Observation data
    • Position (XY)
    • Image
    • The number of detected objects
    Learn by 1 observation data at 1 place.
    (2 visits for each place, total is 12 visits.)
    1. living
    3. kitchen
    2. bedroom 4. bathroom
    Procedures for On-site Learning
    Undefined objects
    Simulation Environment
    ×5 Place categorization
    by SpCoSLAM
    Object detection by YOLO
    Capture the images
    

    View Slide

19. 19
    Explanation of the task
    Bad (Room visits are 3.)
    Good (Room visits are 1.)
    ×16 ×16 Target Object
    Target Object
    Object Search Task
    Give an object name to a robot
    Conducted as 24
    objects
    Onsite learning times is 0.
    ・・・
    Look for the sheep doll！
    I visited 3 rooms to
    find target object.
    I visited 1 room to
    find target object.
    Onsite learning times is 12.
    

    View Slide

20. 20
    Our：0.88 (21 objects / 24 objects)
    (Rooms visited during learning is 5 times)
    SpCoSLAM + Prior：0.92 (22 objects / 24 objects)
    (Room visited during learning is 8 times)
    𝑁
    ෍
    𝑖
    𝑁
    𝑛𝑖
    𝑁 (𝑁 4)：T e num er of o e ts p a ed in t e en ironment
    𝑛𝑖
    ： The number of objects found by the robot
    Reduced the on-site learning cost by a
    factor of 1.6 over more than other approaches.
    Experimental Result
    Proportions of objects found by the robot in one search (Satisfaction threshold was set at 80%.)
    

    View Slide

21. 21
    Prediction of Places Where an Undefined Object Exists by Our Model
    Clean up task
    Leveraging knowledge of objects in the same category as target by probabilistic logical inference.
    Target object
    Duck shaped sponge
    

    View Slide

22. Conclusion
    22
    

    View Slide

23. 23
    Summary
    • Proposed a model that integrates probabilistic logic and multimodal spatial concept model
    • Experiment：Object search task with undefined objects
    Our approach showed that reducing the on-site learning cost by a factor of 1.6 over more than
    other approaches.
    Future Works
    Conclusion
    Leveraging probabilistic distribution from Large Language Model (e.g., PaLM [14], GPT-3 [15]).
    Example of Tasks
    For an object, answer
    where it is placed in the house.
    Object: plate
    Location: kitchen
    Query
    Object: apple
    Location:
    https://gpt3demo.com/apps/openai-gpt-3-playground
    [14] A. Chowdhery, et al. "PaLM: Scaling Language Modeling with Pathways" arXiv preprint arXiv:2204.02311, 2022.
    [15] B. Tom, et al. "Language Models are Few-Shot Learners“, Advances in neural information processing systems, Vol.33, pp1877-1901, 2020.
    Large Language Model
    

    View Slide