See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

Transcript

1. Set the Glass Half Full: Reasoning about Liquid Containers, their

   Volume and Content Mottaghi et al. ICCV 2017 @mocobt

2. • CGͰ༡ΜͰ͓ۚΛ໯͍ͬͯΔ • ڧԽֶशͷྠಡձ։࠵தʂ - https://yurufuwa-reading.connpass.com/event/154933/ • ྲྀମΛ৮Γ͍ͨ೥ࠒ - ΄΅ແؔ܎Ͱ͕͢ɼϜϦϠϦབྷΊ͖ͯ·ͨ͠ʂ

   - CG࿦จಡΈձ΍Γ͍ͨ…ͳ͓धཁ͕ߦํෆ໌ About me @mocobt

3. ಥવͰ͕͢ɼ ࠨͷάϥεͷਫɼԿml?

4. ಥવͰ͕͢ɼ ࠨͷάϥεͷਫɼԿml? ࠓ೔ͷςʔϚ Single RGB Image͔Β൑அ͍ͨ͠ʂ

5. Set the Glass Half Full [Mottaghi++, ICCV 2017] എܠ: ͋Δ༰ثͷӷମྔΛը૾ͷΈ͔Βਪఆ͢Δख๏͸ͳ͔ͬͨ

   ৽نੑ: single RGB imageதͷ༰ثʹର͠ɼӷମྔͷਪఆ౳Λߦ͏ख๏ͷఏҊ ՝୊: ख๏͕͍͍Ճݮ (ޙड़) ॴײ: • ໰୊ઃఆͷΦϦδφϦςΟ͕ڧ͍ • ͱ͍͏͔ɼ͜ΕҎલ΋Ҏ߱΋୭΋΍ͬͯͳ͍… චऀ΋๞͖͍ͯΔ • Open Question͕ଟ͍ͷͰωλʹࠔͬͨΒੋඇ

6. Funky Introduction ΧοϓΛ܏͚ͨ৔߹ɼ 5ϲ݄ࣇ͸ӷମ͕ᷓΕΔ͔൱͔൑அՄೳ [Hespos++. Psychol Sci 2009] Φϥϯ΢ʔλϯ͸ɼ ΧοϓதͷӷମͷମੵΛਪఆՄೳɽ

   ͞Βʹଞͷ༰ثʹҠͤΔ͔΋൑அՄೳ [Call and Rochat. J. Comp Psychol 1997] Ͳ͔͜Βݟ͚ͭͨΜͩΑ..͜ͷݚڀ….

7. Related Work Ψϥε੡඼ͷ࢟੎ਪఆ&Reconstruction [Phillips++. RSS 2016] ը૾தͷ෺ମΛԡͨ͠ͱ͖ͷي੻ਪఆ [Mottaghi++. ECCV 2016]

   Differential DP for pouring tasks (࣮ݧࣨ؀ڥ) [Yamaguchi and Atkeson. Humanoids 2015] Teaching pouring to robots (Additional sensors) [Rozo++. RoMoCo 2013]

8. Proposed Tasks Container volume estimation ༰ੵਪఆ 1,000 ml 200 ml

   Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml

9. COQE Dataset (Containers Of liQuid contEnt) • 5,000ຕҎ্ͷը૾σʔληοτ • ֤ը૾தʹ͸2ͭҎ্ͷ༰ث͕ଘࡏ͠ɼͦΕͧΕΧςΰϦ(e.g.

   bottle, glass)͕ҟͳΔ • ֤༰ثʹରͯ͠͸CAD modelׂ͕ΓৼΒΕ͍ͯΔ • ͨͩ͠model͕34छྨ͔͠ͳ͔ͬͨͨΊɼ࠷΋ࣅ͍ͯΔ΋ͷΛ࢖༻

10. Approach & Evaluation Container volume estimation ༰ੵਪఆ 1,000 ml 200

    ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml

11. Volume and Content Estimation Container volume estimation ༰ੵਪఆ 1,000 ml

    200 ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml

12. Approach: Volume and Content Estimation (A) Input: 4ch (RGB +

    Bounding Box smoothed by a Gaussian kernel) (B) Instance segmentation with Multipath network (COCO 80 Class + Background) (C) Contextual ResNet for Containers (CRC): ResNet-18ͷதؒ૚ʹmask݁߹ (D) ClassificationͰ༰ੵ൑ఆ + ӷମͷׂ߹ਪఆ (A) (B) (C) (D)

13. Evaluation: Volume Estimation • 10 Class෼ྨ (50, 100, 200, 300,

    500, 750, 1000, 2000, 3000, ∞ : ୯Ґ͸mL) • CRC͸AlexNetΑΓͪΐͬͱϚγͱݴ͏ఔ౓….ඍົ͗͢Δ… ← maskͳ͠ ← mask͋Γ

14. Evaluation: Content Estimation • ΍͸Γඍົ… • 6 Class෼ྨ (0%, 33%,

    50%, 66%, 100%, opaque)

15. Comparative Volume Estimation Container volume estimation ༰ੵਪఆ 1,000 ml 200

    ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml

16. Approach: Comparative Volume Estimation ༰ثB RGB + Mask ༰ثA RGB

    + Mask FC “yes”, “no”, “can’t tell” (Classification) “yes”: ʮ༰ثAͷӷମΛ༰ثBʹҠ͢͜ͱ͕Ͱ͖Δʯͷҙ weight sharing

17. How to Evaluate: Comparative Volume Estimation 2ͭͷ༰ثͷ༰ੵΛ , ݱࡏͷӷମͷׂ߹Λ ͱ͢Δͱɼ

    Λຬͨ͢ͱ͖ɼ༰ث1͔Β༰ث2ʹӷମΛ஫͙͜ͱ͕Ͱ͖Δ (͜ͷͱ͖ਖ਼ղϥϕϧ͸”yes”) 200 0.5 100 1000 1000 0.00 OK!

18. Evaluation: Comparative Volume Estimation • 3 Class෼ྨ (“yes”, “no”, “can’t

    tell”) • ͜ͷλεΫʹؔͯ͠͸ɼAlexNetΑΓ͔ͳΓྑ͍

19. Pouring Prediction Container volume estimation ༰ੵਪఆ 1,000 ml 200 ml

    Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml

20. Problems: Pouring Prediction ΍Γ͍ͨ͜ͱ: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔΛ஌Γ͍ͨ Rotate ͭΒΈ1 গ͚ͩ͠ճసͯ͠΋ ӷମྔʹมԽ͕ͳ͍ ͭΒΈ2

    ͭΒΈ3 ༰ثͷܗঢ়ʹΑͬͯ ྲྀΕग़Δྔ͕ҟͳΔ ༰ثͷେ͖͞ʹΑͬͯ ྲྀΕग़Δ͕࣌ؒҟͳΔ ଟ গ ௕ ୹

21. Approach: Pouring Prediction • ࣌ܥྻ༧ଌ໰୊ͱΈͳͯ͠ɼRNNΛ૊ΈࠐΜͩCRCΛ࢖༻ • Input: RGB + ༰ثͷBounding

    Box + ܏͚Δ֯౓ • Output: ࣌ؒຖͷ༰ث಺ͷӷମྔ͔ΒͳΔSequence • CAD modelΛ༻͍ͯγϛϡϨʔγϣϯͨ݁͠ՌΛϥϕϧͱͯ͠ɼTime stepຖʹ෼ྨ • Time stepͷఆٛ: (࠷ऴతʹ܏͚Δ֯౓) / ࠷େstep਺

22. Solved Problems: Pouring Prediction ΍Γ͍ͨ͜ͱ: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔΛ஌Γ͍ͨ Rotate ͭΒΈ1 গ͚ͩ͠ճసͯ͠΋ ӷମྔʹมԽ͕ͳ͍

    ͭΒΈ2 ͭΒΈ3 ༰ثͷܗঢ়ʹΑͬͯ ྲྀΕग़Δྔ͕ҟͳΔ ༰ثͷେ͖͞ʹΑͬͯ ྲྀΕग़Δ͕࣌ؒҟͳΔ ଟ গ ௕ ୹ CAD model + Simulation Time stepΛ ༗ݶݸʹ෼ׂ No description Solved Solved Solved?

23. Evaluation: Pouring Prediction • ࢒ଘྔͷׂ߹Ͱ11 Class෼ྨ (0.0, 0.1, …, 0.9,

    1.0, opaque : opaque͸෼ྨෆՄͷҙ) • GTͷsequenceͱ֤εςοϓͰ׬શʹҰகͨ͠ͱ͖ͷΈΛcorrectͱ͢Δ • Ͳͷฤूڑ཭΋AlexNetΑΓ͸ߴ͍…͜Ε͍͍ͬͯΜ͚ͩͬ…? 1ߦ໨͸׬શʹҰகͨ͠ཁૉ਺ɼҎ߱͸ฤूڑ཭

24. Set the Glass Half Full [Mottaghi++, ICCV 2017] എܠ: ͋Δ༰ثͷӷମྔΛը૾ͷΈ͔Βਪఆ͢Δख๏͸ͳ͔ͬͨ

    ৽نੑ: single RGB imageதͷ༰ثʹରͯ͠ɼҎԼΛߦ͏NNఏҊ + Dataset࡞੒ A. Container volume estimation: ༰ثͷ༰ੵਪఆ B. Content estimation: ༰ثதͷӷମྔ(ׂ߹)ਪఆ C. Comparative volume estimation: ͋Δ༰ثதͷӷମΛɼผͷ༰ثʹҠ͢͜ͱ͕Ͱ͖Δ͔Λ൑ఆ D. Pouring prediction: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹɼ༰ث಺ʹ࢒͍ͬͯΔӷମྔͷਪఆ ՝୊: ख๏͕શମతʹ͍͍Ճݮ ॴײ: • ໰୊ઃఆ͸໘ന͍ͷͰɼKaggleԽͯ͘͠Ε • Open Question͕ଟ͍ͷͰωλʹࠔͬͨΒੋඇ