Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces

2022/10/22に開催されたECCV2022読み会＠PRMU研究会において発表した資料です。
本研究のProject pageはこちら(https://leonidk.github.io/fuzzy-metaballs/)

Transcript

1. Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces 2022/10/22 ECCV2022ಡΈձˏPRMUݚڀձ @tattaka_sun
2. Profile HN: tattaka (Twitter:@tattaka_sun) ॴଐɿө૾੍࡞ձࣾͷR&D৽ଔ2೥໨ झຯɿػցֶशίϯϖɾ೤ଳڕࣂҭ ڵຯɿComputer VisionɾComputer GraphicsɾDeep Learning ͦͷଞɿֶੜ࣌୅͸ඍ෼ՄೳϨϯμϥʔΛ࢖ͬͨݚڀΛ͍ͯͨ͠ / DL4USੜ·ΕKaggleҭͪͷDLωΠςΟϒੈ୅
3. ࠓճ঺հ͢Δݚڀ “Fuzzy Metaballs” • metaballͱ͍͏ղऍ͕͠΍͍͢3DදݱΛ 
 ඍ෼ՄೳͳܗͰϨϯμϦϯά͢Δख๏ͷఏҊ • ଞͷඍ෼ՄೳϨϯμϥΛ༻͍ͨख๏ΑΓϩόετ͔ͭ 
 ߴ଎ͳ͜ͱΛෳ਺ͷλεΫͰ֬ೝ • ݸਓతͳਪ͠ϙΠϯτɿݹయతͳख๏Λݱ୅෩ʹ࠶ղऍɾ 
 CPUͰ΋ߴ଎ͳ఺΍ϩόετੑ͕ߴ͍ͳͲଞख๏ͱ͸ҧͬͨ࢖͍ํ͕Ͱ͖ΔͷͰ͸ • Jax࣮૷΋͋Γ·͢(https://github.com/leonidk/fuzzy-metaballs)
4. Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces
5. Metaballͱ͸ᶃ • ಉ৺ٿঢ়ʹ֎ଆʹີ౓͕ബ͘ͳ͍ͬͯ͘ີ౓෼෍Λߟ͑ͨ࣌ʹɺ 
 ີ౓͕ҰఆʹͳΔ౳஋໘ΛऔΓग़͢ͱಘΒΕΔٿ • Blinn’s algebraic surfacesͱ΋ 
 ݺ͹ΕΔ • ϝογϡʹൺ΂ύϥϝʔλ͕ 
 গͳ͘༗ػతͳܗঢ়ΛදͤΔ https://en.wikipedia.org/wiki/Metaballs
6. Metaballͱ͸ᶄ ౳஋໘্ͷ3࣍ݩͷ఺ ͸ҎԼΛຬͨ͢ ͨͩ͠ɹ͸ᮢ஋ɺ ͸ີ౓෼෍ʢଟ࣍ݩΨ΢ε෼෍Λ࢖༻ʣ ͭ·ΓɺύϥϝτϦοΫʹද໘ΛఆٛͰ͖ͳ͍ 
 ӄؔ਺දݱͷҰछͰ͋Δͱݴ͑Δ
7. Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces
8. ϨϯμϦϯάϓϩηεͷޯ഑Λऔಘ͢Δ͜ͱʹΑͬͯɺ 2Dը૾͕ग़ͯ͘ΔΑ͏ͳ3DදݱΛޯ഑๏ʹΑͬͯٻΊΒΕΔ ඍ෼ՄೳϨϯμϦϯάʹ͍ͭͯ 3Dදݱ 2Dը૾ 📸 ϨϯμϦϯά δ2Dը૾ / δ3Dදݱ
9. ৭ʑͳඍ෼ՄೳϨϯμϦϯά • MeshදݱΛϨϯμϦϯά͢Δ ◦ ϨΠτϨʔγϯά(ྫ: Mitsuba3) ◦ ϥελϥΠζ(ྫ: Neural Mesh Renderer, Soft Rasterizer) • Neural Rendering ◦ ྫ: Neural Voxel Renderer, NeRF, GPNR • ӄؔ਺දݱΛϨϯμϦϯά͢Δ ◦ ྫ: SDFDiff, 
 Differentiable Signed Distance Function Rendering
10. ৭ʑͳඍ෼ՄೳϨϯμϦϯά • MeshදݱΛϨϯμϦϯά͢Δ ◦ ϨΠτϨʔγϯά(ྫ: Mitsuba3) ◦ ϥελϥΠζ(ྫ: Neural Mesh Renderer, Soft Rasterizer) • Neural Rendering ◦ ྫ: Neural Voxel Renderer, NeRF, GPNR • ӄؔ਺දݱΛϨϯμϦϯά͢Δ ← ࠓճ͸ίϨʂ ◦ ྫ: SDFDiff, 
 Differentiable Signed Distance Function Rendering
11. Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces
12. ͜ͷݚڀͰఏҊ͞Ε͍ͯΔ͜ͱ • Fuzzy MetaballsΛϨϯμϦϯά͢ΔͨΊͷ2ͭͷखॱ ◦ ೚ҙͷ࠲ඪ͔Βඈ͹ͨ͠ϨΠͱ֤Ψ΢ε෼෍ͱͷަࠩ൑ఆ ◦ ϐΫηϧ͝ͱͰͷަࠩ൑ఆͷॏͶ͋Θͤ • Fuzzy MetaballsΛ3Dදݱͱͯ͠Ͳ͏࣮૷͢Δ͔ 
 ɾଞͷ3Dදݱͱͷม׵ ▪ ྫ͑͹NeRF͸ϝογϡͱ૬ޓม׵͕೉͍͠
13. Ψ΢ε෼෍ͱͷަࠩ൑ఆ Ұ൪୯७ͳํ๏͸ɺϨΠͱަΘΔΨ΢ε෼෍ͷ࠷େ஋͕ަࠩ఺ͱ͢Δ ଟ࣍ݩΨ΢ε෼෍ͷࣜ 
 Λ༻͍ͯɺ ΛϨΠ Ͱ୅ೖ͢Δ͜ͱΛߟ͑ ࠷େʹͳΔ఺Λ୳͢ͱࢹ఺ͱަ఺ͷؒͷڑ཭ ͸ ͱٻ·Δ 2D্ۭؒͰͷγϛϡϨʔγϣϯ
14. ަࠩ൑ఆͷผͷߟ͑ํ ผͷߟ͑ํͱͯ͠ɺΨ΢ε෼෍ͷޯ഑͕࠷େʹͳΔ఺ΛͱΔ • 3͕࣍ࣜग़ͯ͘ΔͷͰͦΕΛղ͘ • 3࣍ࣜ͸ෆ҆ఆͳͷͰ2࣍ࣜʹۙࣅͯ͠ղ͘ Β͍͕͠଎౓ɾ҆ఆੑͷ໘Ͱ1ͭ໨ͷߟ͑ํͰྑ͍Β͍͠ͷͰུ
15. ަࠩ൑ఆͷॏͶ͋Θͤ ͦΕͧΕͷΨ΢ε෼෍ͱͷަ఺΁ͷڑ཭ΛҎԼͷࣜͰॏΈ෇͚͢Δ ͨͩ͠ ͸ަ఺ͷ֬཰ͱΧϝϥ͔Βͷڑ཭ͷόϥϯεΛͱΔ ɾ෺ମͷεέʔϧʹґଘ͢Δఆ਺ Λ࢖ͬͯද͢ ͳ͓ ͸Sigmoidؔ਺
16. ֆ࡞Γ ࢹ໺಺ͷશͯͷΨ΢ε෼෍Λߟྀͨ͠ ͕ܭࢉͰ͖ͨͷͰ͜ΕΛ࢖ͬͯ 
 alpha maskΛ࡞੒͢Δ ͸ͦΕͧΕεέʔϧΛิਖ਼͢ΔͨΊͷϋΠύʔύϥϝʔλ
17. ϋΠύʔύϥϝʔλʹ͍ͭͯ ఆࣜԽ͢Δͱ߹ܭ5ͭͷϋΠύϥ͕ଘࡏ͢Δ( ) ͦΕΒΛֶशର৅ͷύϥϝʔλͱͯ͠࠷దԽ͢Δ͜ͱ΋ՄೳͰ͸͋Δ͕ɺ ࿦จதͷ࣮ݧͰ͸ελϯϑΥʔυόχʔΛ༻͍ͨখ͞ͳσʔληοτͰ 
 ͋Β͔͡ΊϒϥοΫϘοΫε࠷దԽख๏(pycma)Λ༻͍ͯ࠷దԽ͍ͯ͠Δ
18. Ϩϯμϥͷ࣮૷ • 40ߦ͘Β͍Ͱ࣮૷Մೳ • ೖྗͱͯ͠GMMͷύϥϝʔλɺ 
 ΧϝϥύϥϝʔλɺϋΠύϥ 
 ΛೖΕΔͱը૾͕ग़ྗ͞ΕΔ
19. Fuzzy Metaballsͷ࣮૷ɾଞͷ3Dදݱͱͷม׵ • ཁ͢Δʹࠞ߹Ψ΢ε෼෍ͳͷͰ 
 sklearn.mixture.GaussianMixtureΛ࢖͏ • ϝογϡ΍఺܈→Fuzzy Metaballs: 
 ද໘΋͘͠͸಺෦͔Β఺ΛαϯϓϦϯά͠ɺGMMͰFit͢Δ • Fuzzy Metaballs→ ◦ ఺܈: GMM͔ΒαϯϓϦϯάΛߦ͏ ◦ ϝογϡ: Λ࠷దԽ্ͨ͠ͰMarching Cube๏Ͱද໘ΛுΔ
20. ࣮ݧ • Pose Estimation • Shape From Silhouette
21. ࣮ݧ: Pose Estimation ༩͑ΒΕͨtargetը૾ʹରͯ͠ϨϯμϦϯάը૾ͱ߹க͢ΔΑ͏ʹ 
 poseΛ࠷దԽ͢ΔλεΫ target initialization optimized result https://people.csail.mit.edu/tzumao/diffrt/supplementary_webpage/
22. ࣮ݧ: Shape From Silhouette ෳ਺ࢹ఺ͷγϧΤοτը૾͔Βܗঢ়Λ࠶ߏங͢ΔλεΫ https://www.ri.cmu.edu/pub_files/pub4/cheung_kong_man_2005_1/cheung_kong_man_2005_1.pdf
23. ࣮ݧͷલʹ...... ҟͳΔ3DදݱΛൺֱ͢Δͱ͖ʹදݱྗΛἧ͑ͳ͍ͱ 
 ެฏͳධՁͰ͸ͳ͍ ࣮ݧͷൺֱର৅Ͱ͋Δ఺܈ͱϝογϡʹରͯ͠ಉ౳ͷධՁͰ͖ΔΑ͏ʹ ͠ͳ͚Ε͹ͳΒͳ͍ දݱྗΛද͢Կ͔͠Βͷࢦඪ͕ඞཁ......ʁ
24. Perturbation sensitivity ΦϒδΣΫτͷ࢟੎Λਖ਼͍͠ํ޲ʹ޲͚ͨޙɺσϓεϚοϓʹ߹க͢Δ 
 Α͏ʹ࢟੎Λ࠷దԽͨ࣌͠ͷฒਐޡࠩͱճసޡࠩͷزԿฏۉΛܭࢉ͢Δ ఺܈ͱϝογϡ͸ICP(Iterative Closest Point)Λ༻͍Δ ݁Ռɺ400paramͷFuzzy Metaballsͱ 
 810paramͷϝογϡɺ1290paramͷ఺܈ 
 ͕ಉ౳ͱݟͳ͢͜ͱ͕Ͱ͖Δ
25. ༻͍Δσʔλ • Stanford Model Repository͔Β 
 arma, buddha, dragon, lucy, bunny • Thingi10K͔Β gear, eiffel, rebel • ShapeNet͔Β Plane • Sketchfab dataset͔Β Yoga (ࠨͷը૾͸Ұ෦) http://graphics.stanford.edu/data/3Dscanrep/ https://ten-thousand-models.appspot.com/ https://arxiv.org/abs/1901.05567
26. ൺֱର৅ͷඍ෼ՄೳϨϯμϥ • ϝογϡදݱ ◦ Soft Rasterizer • ఺܈ ◦ Direct Surface Splatting ◦ Pulsar
27. ͬ͘͟Γ༻͍Δඍ෼ՄೳϨϯμϥʹ͍ͭͯ • Soft Rasterizer ◦ ϨϯμϦϯά݁ՌΛ΅΍͔͢͜ͱͰޯ഑Λ࡞Δ • Direct Surface Splatting ◦ ఺܈Λପԁମͷू߹ͱଊ͑ͯϨϯμϦϯά͢Δ • Pulser ◦ ্ͱಉ͡Α͏ʹ఺Λପԁମͱͯ͠ଊ͑Δ͕ϨϯμϦϯάର৅ͷ ٿͷબ୒Λ޻෉͢Δ͜ͱͰߴ଎Խ
28. ࣮ݧ: Pose Estimation ༩͑ΒΕͨtargetը૾ʹରͯ͠ϨϯμϦϯάը૾ͱ߹க͢ΔΑ͏ʹ 
 poseΛ࠷దԽ͢ΔλεΫ target initialization optimized result https://people.csail.mit.edu/tzumao/diffrt/supplementary_webpage/
29. ࣮ݧ: Pose Estimation • +-45°ͷճసͱ3DϞσϧͷ50%·Ͱͷ 
 ϥϯμϜͳฏߦҠಈͨ͠΋ͷͷσϓεϚοϓΛ༩͑Δ • લड़ʹՃ͑ͯICP๏΋࣮ݧ • ճసޡࠩͱฒਐޡࠩͷزԿฏۉ • ༩͑ΔσϓεϚοϓʹ 
 ϊΠζΛՃ͑ͨ΋ͷͱ 
 ͦ͏Ͱͳ͍΋ͷΛ༻͍Δ(ϊΠζͷ 
 ৄࡉ͸ৄ͘͠͸ॻ͍͍ͯͳ͍)
30. ࣮ݧ: Shape From Silhouette ෳ਺ࢹ఺ͷγϧΤοτը૾͔Βܗঢ়Λ࠶ߏங͢ΔλεΫ https://www.ri.cmu.edu/pub_files/pub4/cheung_kong_man_2005_1/cheung_kong_man_2005_1.pdf
31. ࣮ݧ: Shape From Silhouette • 64x64pixͷγϧΤοτը૾Λ32ࢹ఺༩͑ͯܗঢ়Λ࠶ߏ੒͢Δ • ϊΠζ͋ΓσʔλͰ͸ɺ16ࢹ఺ʹରͯ͠ϊΠζΛՃ͑Δ • લड़ͷඍ෼ՄೳϨϯμϦϯάख๏ͷଞʹҎԼͷೋͭͷख๏Λ༻͍Δ ◦ NeRF ◦ Voxel Carving • γϧΤοτͱͷ 
 CEଛࣦΛܭࢉ͢Δ ϊΠζ͋Γը૾ͷྫ
32. ࣮ݧ: Shape From Silhouetteᶄ ϊΠζͳ݁͠Ռ
33. ࣮ݧ: Shape From Silhouetteᶅ ϊΠζ͋Γ݁Ռ
34. ࣮ݧ: Shape From Silhouetteᶅ Fuzzy MetaballsͷΈ
35. ٞ࿦ɾLimitations • ຊख๏Ͱ͸ܗঢ়ΛϨϯμϦϯά͢Δ͜ͱʹয఺Λ౰ͯͨ ◦ ࣮ݧ͸͍ͯ͠ͳ͍͕Fuzzy Metaballs͝ͱʹ৭ΛׂΓ౰ͯΔ͜ͱ͸Մೳ • GPUʹಛԽͨ͠࠷దԽ͸͍ͯ͠ͳ͍ͷͰߴ଎Խͷ༨஍͋Γ • Ϟσϧͷࣗ༝౓ͷ௿͔͞Β҉໧తͳਖ਼ଇԽ͕ಇ͖ϊΠζʹڧ͍ͱߟ͑ΒΕΔ ◦ ηάϝϯςʔγϣϯϞσϧग़ྗͷΤϥʔमਖ਼ͳͲʹར༻Մೳ • ࠷దԽ͞ΕͨFuzzy Metaballsͷύϥϝʔλ͸زԿతʹղऍ͕Ͱ͖Δ 
 (GMMͳͷͰ) • ࣮ࡍͷϢʔεέʔεͱͯ͠ɺߴ඼࣭ͳը૾औಘ͕೉͍͠ͱ͖ʹ࢖͑ͦ͏
36. ٞ࿦ɾLimitations
37. ·ͱΊɾײ૝ • Fuzzy Metaballsͱ͍͏ࠞ߹Ψ΢ε෼෍Λݩʹͨ͠ 
 3DදݱΛఏҊ͠ɺۙࣅඍ෼ՄೳϨϯμϦϯάख๏ΛఏҊ • Mesh΍఺܈ͱͷެฏੑΛߟ͑ͨධՁํ๏Λ༻͍ɺطଘͷํ๏ʹඖఢ͢Δਫ਼౓ 
 ͱͱ΋ʹߴ଎͔ͭϊΠζʹରͯ͠ϩόετͳ݁ՌʹͳΔ͜ͱΛࣔͨ͠ • খܕҠಈϩϘοτͳͲGPUΛੵΈͨͯ͘΋ੵΊͳ͍Α͏ͳ༻్Ͱ 
 ΦϯϥΠϯͳܗঢ়ೝࣝͳͲͰ͖Ε͹໘നͦ͏
38. ࢀߟจݙ • هࡌͷͳ͍ݶΓ͸Fuzzy Metaballsͷ࿦จ͔ΒҾ༻ • ͦͷଞ ◦ ϨΠτϨʔγϯάʹΑΔίϯϐϡʔλάϥϑΟΫεೖ໳ - ϝλ Ϙʔϧ - ◦ ඍ෼ՄೳϨϯμϦϯά (CVIMݚڀձ νϡʔτϦΞϧ) ◦ Differentiable Rendering: A Survey