Upgrade to Pro — share decks privately, control downloads, hide ads and more …

See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

E2dd989b2ba0f83d8a981b9cb3197bf1?s=47 mocobt
November 19, 2019

See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content

This is an unofficial explanation slide about [Mottaghi++. ICCV 2017] in Japanese.
It will be presented at https://lpixel.connpass.com/event/154432/

E2dd989b2ba0f83d8a981b9cb3197bf1?s=128

mocobt

November 19, 2019
Tweet

Transcript

  1. Set the Glass Half Full: Reasoning about Liquid Containers, their

    Volume and Content Mottaghi et al. ICCV 2017 @mocobt
  2. • CGͰ༡ΜͰ͓ۚΛ໯͍ͬͯΔ • ڧԽֶशͷྠಡձ։࠵தʂ - https://yurufuwa-reading.connpass.com/event/154933/ • ྲྀମΛ৮Γ͍ͨ೥ࠒ - ΄΅ແؔ܎Ͱ͕͢ɼϜϦϠϦབྷΊ͖ͯ·ͨ͠ʂ

    - CG࿦จಡΈձ΍Γ͍ͨ…ͳ͓धཁ͕ߦํෆ໌ About me @mocobt
  3. ಥવͰ͕͢ɼ ࠨͷάϥεͷਫɼԿml?

  4. ಥવͰ͕͢ɼ ࠨͷάϥεͷਫɼԿml? ࠓ೔ͷςʔϚ Single RGB Image͔Β൑அ͍ͨ͠ʂ

  5. Set the Glass Half Full [Mottaghi++, ICCV 2017] എܠ: ͋Δ༰ثͷӷମྔΛը૾ͷΈ͔Βਪఆ͢Δख๏͸ͳ͔ͬͨ

    ৽نੑ: single RGB imageதͷ༰ثʹର͠ɼӷମྔͷਪఆ౳Λߦ͏ख๏ͷఏҊ ՝୊: ख๏͕͍͍Ճݮ (ޙड़) ॴײ: • ໰୊ઃఆͷΦϦδφϦςΟ͕ڧ͍ • ͱ͍͏͔ɼ͜ΕҎલ΋Ҏ߱΋୭΋΍ͬͯͳ͍… චऀ΋๞͖͍ͯΔ • Open Question͕ଟ͍ͷͰωλʹࠔͬͨΒੋඇ
  6. Funky Introduction ΧοϓΛ܏͚ͨ৔߹ɼ 5ϲ݄ࣇ͸ӷମ͕ᷓΕΔ͔൱͔൑அՄೳ [Hespos++. Psychol Sci 2009] Φϥϯ΢ʔλϯ͸ɼ ΧοϓதͷӷମͷମੵΛਪఆՄೳɽ

    ͞Βʹଞͷ༰ثʹҠͤΔ͔΋൑அՄೳ [Call and Rochat. J. Comp Psychol 1997] Ͳ͔͜Βݟ͚ͭͨΜͩΑ..͜ͷݚڀ….
  7. Related Work Ψϥε੡඼ͷ࢟੎ਪఆ&Reconstruction [Phillips++. RSS 2016] ը૾தͷ෺ମΛԡͨ͠ͱ͖ͷي੻ਪఆ [Mottaghi++. ECCV 2016]

    Differential DP for pouring tasks (࣮ݧࣨ؀ڥ) [Yamaguchi and Atkeson. Humanoids 2015] Teaching pouring to robots (Additional sensors) [Rozo++. RoMoCo 2013]
  8. Proposed Tasks Container volume estimation ༰ੵਪఆ 1,000 ml 200 ml

    Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml
  9. COQE Dataset (Containers Of liQuid contEnt) • 5,000ຕҎ্ͷը૾σʔληοτ • ֤ը૾தʹ͸2ͭҎ্ͷ༰ث͕ଘࡏ͠ɼͦΕͧΕΧςΰϦ(e.g.

    bottle, glass)͕ҟͳΔ • ֤༰ثʹରͯ͠͸CAD modelׂ͕ΓৼΒΕ͍ͯΔ • ͨͩ͠model͕34छྨ͔͠ͳ͔ͬͨͨΊɼ࠷΋ࣅ͍ͯΔ΋ͷΛ࢖༻
  10. Approach & Evaluation Container volume estimation ༰ੵਪఆ 1,000 ml 200

    ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml
  11. Volume and Content Estimation Container volume estimation ༰ੵਪఆ 1,000 ml

    200 ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml
  12. Approach: Volume and Content Estimation (A) Input: 4ch (RGB +

    Bounding Box smoothed by a Gaussian kernel) (B) Instance segmentation with Multipath network (COCO 80 Class + Background) (C) Contextual ResNet for Containers (CRC): ResNet-18ͷதؒ૚ʹmask݁߹ (D) ClassificationͰ༰ੵ൑ఆ + ӷମͷׂ߹ਪఆ (A) (B) (C) (D)
  13. Evaluation: Volume Estimation • 10 Class෼ྨ (50, 100, 200, 300,

    500, 750, 1000, 2000, 3000, ∞ : ୯Ґ͸mL) • CRC͸AlexNetΑΓͪΐͬͱϚγͱݴ͏ఔ౓….ඍົ͗͢Δ… ← maskͳ͠ ← mask͋Γ
  14. Evaluation: Content Estimation • ΍͸Γඍົ… • 6 Class෼ྨ (0%, 33%,

    50%, 66%, 100%, opaque)
  15. Comparative Volume Estimation Container volume estimation ༰ੵਪఆ 1,000 ml 200

    ml Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml
  16. Approach: Comparative Volume Estimation ༰ثB RGB + Mask ༰ثA RGB

    + Mask FC “yes”, “no”, “can’t tell” (Classification) “yes”: ʮ༰ثAͷӷମΛ༰ثBʹҠ͢͜ͱ͕Ͱ͖Δʯͷҙ weight sharing
  17. How to Evaluate: Comparative Volume Estimation 2ͭͷ༰ثͷ༰ੵΛ , ݱࡏͷӷମͷׂ߹Λ ͱ͢Δͱɼ

    Λຬͨ͢ͱ͖ɼ༰ث1͔Β༰ث2ʹӷମΛ஫͙͜ͱ͕Ͱ͖Δ (͜ͷͱ͖ਖ਼ղϥϕϧ͸”yes”) 200 0.5 100 1000 1000 0.00 OK!
  18. Evaluation: Comparative Volume Estimation • 3 Class෼ྨ (“yes”, “no”, “can’t

    tell”) • ͜ͷλεΫʹؔͯ͠͸ɼAlexNetΑΓ͔ͳΓྑ͍
  19. Pouring Prediction Container volume estimation ༰ੵਪఆ 1,000 ml 200 ml

    Content estimation ӷମͷׂ߹ਪఆ 90% 70% Comparative volume estimation ӷମΛҠͤΔ͔൑ఆ (bool) True False Pouring prediction ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔਪఆ Rotate 100 ml
  20. Problems: Pouring Prediction ΍Γ͍ͨ͜ͱ: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔΛ஌Γ͍ͨ Rotate ͭΒΈ1 গ͚ͩ͠ճసͯ͠΋ ӷମྔʹมԽ͕ͳ͍ ͭΒΈ2

    ͭΒΈ3 ༰ثͷܗঢ়ʹΑͬͯ ྲྀΕग़Δྔ͕ҟͳΔ ༰ثͷେ͖͞ʹΑͬͯ ྲྀΕग़Δ͕࣌ؒҟͳΔ ଟ গ ௕ ୹
  21. Approach: Pouring Prediction • ࣌ܥྻ༧ଌ໰୊ͱΈͳͯ͠ɼRNNΛ૊ΈࠐΜͩCRCΛ࢖༻ • Input: RGB + ༰ثͷBounding

    Box + ܏͚Δ֯౓ • Output: ࣌ؒຖͷ༰ث಺ͷӷମྔ͔ΒͳΔSequence • CAD modelΛ༻͍ͯγϛϡϨʔγϣϯͨ݁͠ՌΛϥϕϧͱͯ͠ɼTime stepຖʹ෼ྨ • Time stepͷఆٛ: (࠷ऴతʹ܏͚Δ֯౓) / ࠷େstep਺
  22. Solved Problems: Pouring Prediction ΍Γ͍ͨ͜ͱ: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹ࢒͍ͬͯΔӷମྔΛ஌Γ͍ͨ Rotate ͭΒΈ1 গ͚ͩ͠ճసͯ͠΋ ӷମྔʹมԽ͕ͳ͍

    ͭΒΈ2 ͭΒΈ3 ༰ثͷܗঢ়ʹΑͬͯ ྲྀΕग़Δྔ͕ҟͳΔ ༰ثͷେ͖͞ʹΑͬͯ ྲྀΕग़Δ͕࣌ؒҟͳΔ ଟ গ ௕ ୹ CAD model + Simulation Time stepΛ ༗ݶݸʹ෼ׂ No description Solved Solved Solved?
  23. Evaluation: Pouring Prediction • ࢒ଘྔͷׂ߹Ͱ11 Class෼ྨ (0.0, 0.1, …, 0.9,

    1.0, opaque : opaque͸෼ྨෆՄͷҙ) • GTͷsequenceͱ֤εςοϓͰ׬શʹҰகͨ͠ͱ͖ͷΈΛcorrectͱ͢Δ • Ͳͷฤूڑ཭΋AlexNetΑΓ͸ߴ͍…͜Ε͍͍ͬͯΜ͚ͩͬ…? 1ߦ໨͸׬શʹҰகͨ͠ཁૉ਺ɼҎ߱͸ฤूڑ཭
  24. Set the Glass Half Full [Mottaghi++, ICCV 2017] എܠ: ͋Δ༰ثͷӷମྔΛը૾ͷΈ͔Βਪఆ͢Δख๏͸ͳ͔ͬͨ

    ৽نੑ: single RGB imageதͷ༰ثʹରͯ͠ɼҎԼΛߦ͏NNఏҊ + Dataset࡞੒ A. Container volume estimation: ༰ثͷ༰ੵਪఆ B. Content estimation: ༰ثதͷӷମྔ(ׂ߹)ਪఆ C. Comparative volume estimation: ͋Δ༰ثதͷӷମΛɼผͷ༰ثʹҠ͢͜ͱ͕Ͱ͖Δ͔Λ൑ఆ D. Pouring prediction: ͋Δ֯౓Ͱ༰ثΛ܏͚ͨޙʹɼ༰ث಺ʹ࢒͍ͬͯΔӷମྔͷਪఆ ՝୊: ख๏͕શମతʹ͍͍Ճݮ ॴײ: • ໰୊ઃఆ͸໘ന͍ͷͰɼKaggleԽͯ͘͠Ε • Open Question͕ଟ͍ͷͰωλʹࠔͬͨΒੋඇ