Tweet
Tweet

Slide 1

Slide 1 text

Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces 2022/10/22 ECCV2022ಡΈձˏPRMUݚڀձ @tattaka_sun

Slide 2

Slide 2 text

Profile HN: tattaka (Twitter:@tattaka_sun) ॴଐɿө૾੍࡞ձࣾͷR&D৽ଔ2೥໨ झຯɿػցֶशίϯϖɾ೤ଳڕࣂҭ ڵຯɿComputer VisionɾComputer GraphicsɾDeep Learning ͦͷଞɿֶੜ࣌୅͸ඍ෼ՄೳϨϯμϥʔΛ࢖ͬͨݚڀΛ͍ͯͨ͠ / DL4USੜ·ΕKaggleҭͪͷDLωΠςΟϒੈ୅

Slide 3

Slide 3 text

ࠓճ঺հ͢Δݚڀ “Fuzzy Metaballs” ● metaballͱ͍͏ղऍ͕͠΍͍͢3DදݱΛ 
 ඍ෼ՄೳͳܗͰϨϯμϦϯά͢Δख๏ͷఏҊ ● ଞͷඍ෼ՄೳϨϯμϥΛ༻͍ͨख๏ΑΓϩόετ͔ͭ 
 ߴ଎ͳ͜ͱΛෳ਺ͷλεΫͰ֬ೝ ● ݸਓతͳਪ͠ϙΠϯτɿݹయతͳख๏Λݱ୅෩ʹ࠶ղऍɾ 
 CPUͰ΋ߴ଎ͳ఺΍ϩόετੑ͕ߴ͍ͳͲଞख๏ͱ͸ҧͬͨ࢖͍ํ͕Ͱ͖ΔͷͰ͸ ● Jax࣮૷΋͋Γ·͢(https://github.com/leonidk/fuzzy-metaballs)

Slide 4

Slide 4 text

Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces

Slide 5

Slide 5 text

Metaballͱ͸ᶃ ● ಉ৺ٿঢ়ʹ֎ଆʹີ౓͕ബ͘ͳ͍ͬͯ͘ີ౓෼෍Λߟ͑ͨ࣌ʹɺ 
 ີ౓͕ҰఆʹͳΔ౳஋໘ΛऔΓग़͢ͱಘΒΕΔٿ ● Blinn’s algebraic surfacesͱ΋ 
 ݺ͹ΕΔ ● ϝογϡʹൺ΂ύϥϝʔλ͕ 
 গͳ͘༗ػతͳܗঢ়ΛදͤΔ https://en.wikipedia.org/wiki/Metaballs

Slide 6

Slide 6 text

Metaballͱ͸ᶄ ౳஋໘্ͷ3࣍ݩͷ఺ ͸ҎԼΛຬͨ͢ ͨͩ͠ɹ͸ᮢ஋ɺ ͸ີ౓෼෍ʢଟ࣍ݩΨ΢ε෼෍Λ࢖༻ʣ ͭ·ΓɺύϥϝτϦοΫʹද໘ΛఆٛͰ͖ͳ͍ 
 ӄؔ਺දݱͷҰछͰ͋Δͱݴ͑Δ

Slide 7

Slide 7 text

Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces

Slide 8

Slide 8 text

ϨϯμϦϯάϓϩηεͷޯ഑Λऔಘ͢Δ͜ͱʹΑͬͯɺ 2Dը૾͕ग़ͯ͘ΔΑ͏ͳ3DදݱΛޯ഑๏ʹΑͬͯٻΊΒΕΔ ඍ෼ՄೳϨϯμϦϯάʹ͍ͭͯ 3Dදݱ 2Dը૾ 📸 ϨϯμϦϯά δ2Dը૾ / δ3Dදݱ

Slide 9

Slide 9 text

৭ʑͳඍ෼ՄೳϨϯμϦϯά ● MeshදݱΛϨϯμϦϯά͢Δ ○ ϨΠτϨʔγϯά(ྫ: Mitsuba3) ○ ϥελϥΠζ(ྫ: Neural Mesh Renderer, Soft Rasterizer) ● Neural Rendering ○ ྫ: Neural Voxel Renderer, NeRF, GPNR ● ӄؔ਺දݱΛϨϯμϦϯά͢Δ ○ ྫ: SDFDiff, 
 Differentiable Signed Distance Function Rendering

Slide 10

Slide 10 text

৭ʑͳඍ෼ՄೳϨϯμϦϯά ● MeshදݱΛϨϯμϦϯά͢Δ ○ ϨΠτϨʔγϯά(ྫ: Mitsuba3) ○ ϥελϥΠζ(ྫ: Neural Mesh Renderer, Soft Rasterizer) ● Neural Rendering ○ ྫ: Neural Voxel Renderer, NeRF, GPNR ● ӄؔ਺දݱΛϨϯμϦϯά͢Δ ← ࠓճ͸ίϨʂ ○ ྫ: SDFDiff, 
 Differentiable Signed Distance Function Rendering

Slide 11

Slide 11 text

Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces

Slide 12

Slide 12 text

͜ͷݚڀͰఏҊ͞Ε͍ͯΔ͜ͱ ● Fuzzy MetaballsΛϨϯμϦϯά͢ΔͨΊͷ2ͭͷखॱ ○ ೚ҙͷ࠲ඪ͔Βඈ͹ͨ͠ϨΠͱ֤Ψ΢ε෼෍ͱͷަࠩ൑ఆ ○ ϐΫηϧ͝ͱͰͷަࠩ൑ఆͷॏͶ͋Θͤ ● Fuzzy MetaballsΛ3Dදݱͱͯ͠Ͳ͏࣮૷͢Δ͔ 
 ɾଞͷ3Dදݱͱͷม׵ ■ ྫ͑͹NeRF͸ϝογϡͱ૬ޓม׵͕೉͍͠

Slide 13

Slide 13 text

Ψ΢ε෼෍ͱͷަࠩ൑ఆ Ұ൪୯७ͳํ๏͸ɺϨΠͱަΘΔΨ΢ε෼෍ͷ࠷େ஋͕ަࠩ఺ͱ͢Δ ଟ࣍ݩΨ΢ε෼෍ͷࣜ 
 Λ༻͍ͯɺ ΛϨΠ Ͱ୅ೖ͢Δ͜ͱΛߟ͑ ࠷େʹͳΔ఺Λ୳͢ͱࢹ఺ͱަ఺ͷؒͷڑ཭ ͸ ͱٻ·Δ 2D্ۭؒͰͷγϛϡϨʔγϣϯ

Slide 14

Slide 14 text

ަࠩ൑ఆͷผͷߟ͑ํ ผͷߟ͑ํͱͯ͠ɺΨ΢ε෼෍ͷޯ഑͕࠷େʹͳΔ఺ΛͱΔ ● 3͕࣍ࣜग़ͯ͘ΔͷͰͦΕΛղ͘ ● 3࣍ࣜ͸ෆ҆ఆͳͷͰ2࣍ࣜʹۙࣅͯ͠ղ͘ Β͍͕͠଎౓ɾ҆ఆੑͷ໘Ͱ1ͭ໨ͷߟ͑ํͰྑ͍Β͍͠ͷͰུ

Slide 15

Slide 15 text

ަࠩ൑ఆͷॏͶ͋Θͤ ͦΕͧΕͷΨ΢ε෼෍ͱͷަ఺΁ͷڑ཭ΛҎԼͷࣜͰॏΈ෇͚͢Δ ͨͩ͠ ͸ަ఺ͷ֬཰ͱΧϝϥ͔Βͷڑ཭ͷόϥϯεΛͱΔ ɾ෺ମͷεέʔϧʹґଘ͢Δఆ਺ Λ࢖ͬͯද͢ ͳ͓ ͸Sigmoidؔ਺

Slide 16

Slide 16 text

ֆ࡞Γ ࢹ໺಺ͷશͯͷΨ΢ε෼෍Λߟྀͨ͠ ͕ܭࢉͰ͖ͨͷͰ͜ΕΛ࢖ͬͯ 
 alpha maskΛ࡞੒͢Δ ͸ͦΕͧΕεέʔϧΛิਖ਼͢ΔͨΊͷϋΠύʔύϥϝʔλ

Slide 17

Slide 17 text

ϋΠύʔύϥϝʔλʹ͍ͭͯ ఆࣜԽ͢Δͱ߹ܭ5ͭͷϋΠύϥ͕ଘࡏ͢Δ( ) ͦΕΒΛֶशର৅ͷύϥϝʔλͱͯ͠࠷దԽ͢Δ͜ͱ΋ՄೳͰ͸͋Δ͕ɺ ࿦จதͷ࣮ݧͰ͸ελϯϑΥʔυόχʔΛ༻͍ͨখ͞ͳσʔληοτͰ 
 ͋Β͔͡ΊϒϥοΫϘοΫε࠷దԽख๏(pycma)Λ༻͍ͯ࠷దԽ͍ͯ͠Δ

Slide 18

Slide 18 text

Ϩϯμϥͷ࣮૷ ● 40ߦ͘Β͍Ͱ࣮૷Մೳ ● ೖྗͱͯ͠GMMͷύϥϝʔλɺ 
 ΧϝϥύϥϝʔλɺϋΠύϥ 
 ΛೖΕΔͱը૾͕ग़ྗ͞ΕΔ

Slide 19

Slide 19 text

Fuzzy Metaballsͷ࣮૷ɾଞͷ3Dදݱͱͷม׵ ● ཁ͢Δʹࠞ߹Ψ΢ε෼෍ͳͷͰ 
 sklearn.mixture.GaussianMixtureΛ࢖͏ ● ϝογϡ΍఺܈→Fuzzy Metaballs: 
 ද໘΋͘͠͸಺෦͔Β఺ΛαϯϓϦϯά͠ɺGMMͰFit͢Δ ● Fuzzy Metaballs→ ○ ఺܈: GMM͔ΒαϯϓϦϯάΛߦ͏ ○ ϝογϡ: Λ࠷దԽ্ͨ͠ͰMarching Cube๏Ͱද໘ΛுΔ

Slide 20

Slide 20 text

࣮ݧ ● Pose Estimation ● Shape From Silhouette

Slide 21

Slide 21 text

࣮ݧ: Pose Estimation ༩͑ΒΕͨtargetը૾ʹରͯ͠ϨϯμϦϯάը૾ͱ߹க͢ΔΑ͏ʹ 
 poseΛ࠷దԽ͢ΔλεΫ target initialization optimized result https://people.csail.mit.edu/tzumao/diffrt/supplementary_webpage/

Slide 22

Slide 22 text

࣮ݧ: Shape From Silhouette ෳ਺ࢹ఺ͷγϧΤοτը૾͔Βܗঢ়Λ࠶ߏங͢ΔλεΫ https://www.ri.cmu.edu/pub_files/pub4/cheung_kong_man_2005_1/cheung_kong_man_2005_1.pdf

Slide 23

Slide 23 text

࣮ݧͷલʹ...... ҟͳΔ3DදݱΛൺֱ͢Δͱ͖ʹදݱྗΛἧ͑ͳ͍ͱ 
 ެฏͳධՁͰ͸ͳ͍ ࣮ݧͷൺֱର৅Ͱ͋Δ఺܈ͱϝογϡʹରͯ͠ಉ౳ͷධՁͰ͖ΔΑ͏ʹ ͠ͳ͚Ε͹ͳΒͳ͍ දݱྗΛද͢Կ͔͠Βͷࢦඪ͕ඞཁ......ʁ

Slide 24

Slide 24 text

Perturbation sensitivity ΦϒδΣΫτͷ࢟੎Λਖ਼͍͠ํ޲ʹ޲͚ͨޙɺσϓεϚοϓʹ߹க͢Δ 
 Α͏ʹ࢟੎Λ࠷దԽͨ࣌͠ͷฒਐޡࠩͱճసޡࠩͷزԿฏۉΛܭࢉ͢Δ ఺܈ͱϝογϡ͸ICP(Iterative Closest Point)Λ༻͍Δ ݁Ռɺ400paramͷFuzzy Metaballsͱ 
 810paramͷϝογϡɺ1290paramͷ఺܈ 
 ͕ಉ౳ͱݟͳ͢͜ͱ͕Ͱ͖Δ

Slide 25

Slide 25 text

༻͍Δσʔλ ● Stanford Model Repository͔Β 
 arma, buddha, dragon, lucy, bunny ● Thingi10K͔Β gear, eiffel, rebel ● ShapeNet͔Β Plane ● Sketchfab dataset͔Β Yoga (ࠨͷը૾͸Ұ෦) http://graphics.stanford.edu/data/3Dscanrep/ https://ten-thousand-models.appspot.com/ https://arxiv.org/abs/1901.05567

Slide 26

Slide 26 text

ൺֱର৅ͷඍ෼ՄೳϨϯμϥ ● ϝογϡදݱ ○ Soft Rasterizer ● ఺܈ ○ Direct Surface Splatting ○ Pulsar

Slide 27

Slide 27 text

ͬ͘͟Γ༻͍Δඍ෼ՄೳϨϯμϥʹ͍ͭͯ ● Soft Rasterizer ○ ϨϯμϦϯά݁ՌΛ΅΍͔͢͜ͱͰޯ഑Λ࡞Δ ● Direct Surface Splatting ○ ఺܈Λପԁମͷू߹ͱଊ͑ͯϨϯμϦϯά͢Δ ● Pulser ○ ্ͱಉ͡Α͏ʹ఺Λପԁମͱͯ͠ଊ͑Δ͕ϨϯμϦϯάର৅ͷ ٿͷબ୒Λ޻෉͢Δ͜ͱͰߴ଎Խ

Slide 28

Slide 28 text

࣮ݧ: Pose Estimation ༩͑ΒΕͨtargetը૾ʹରͯ͠ϨϯμϦϯάը૾ͱ߹க͢ΔΑ͏ʹ 
 poseΛ࠷దԽ͢ΔλεΫ target initialization optimized result https://people.csail.mit.edu/tzumao/diffrt/supplementary_webpage/

Slide 29

Slide 29 text

࣮ݧ: Pose Estimation ● +-45°ͷճసͱ3DϞσϧͷ50%·Ͱͷ 
 ϥϯμϜͳฏߦҠಈͨ͠΋ͷͷσϓεϚοϓΛ༩͑Δ ● લड़ʹՃ͑ͯICP๏΋࣮ݧ ● ճసޡࠩͱฒਐޡࠩͷزԿฏۉ ● ༩͑ΔσϓεϚοϓʹ 
 ϊΠζΛՃ͑ͨ΋ͷͱ 
 ͦ͏Ͱͳ͍΋ͷΛ༻͍Δ(ϊΠζͷ 
 ৄࡉ͸ৄ͘͠͸ॻ͍͍ͯͳ͍)

Slide 30

Slide 30 text

࣮ݧ: Shape From Silhouette ෳ਺ࢹ఺ͷγϧΤοτը૾͔Βܗঢ়Λ࠶ߏங͢ΔλεΫ https://www.ri.cmu.edu/pub_files/pub4/cheung_kong_man_2005_1/cheung_kong_man_2005_1.pdf

Slide 31

Slide 31 text

࣮ݧ: Shape From Silhouette ● 64x64pixͷγϧΤοτը૾Λ32ࢹ఺༩͑ͯܗঢ়Λ࠶ߏ੒͢Δ ● ϊΠζ͋ΓσʔλͰ͸ɺ16ࢹ఺ʹରͯ͠ϊΠζΛՃ͑Δ ● લड़ͷඍ෼ՄೳϨϯμϦϯάख๏ͷଞʹҎԼͷೋͭͷख๏Λ༻͍Δ ○ NeRF ○ Voxel Carving ● γϧΤοτͱͷ 
 CEଛࣦΛܭࢉ͢Δ ϊΠζ͋Γը૾ͷྫ

Slide 32

Slide 32 text

࣮ݧ: Shape From Silhouetteᶄ ϊΠζͳ݁͠Ռ

Slide 33

Slide 33 text

࣮ݧ: Shape From Silhouetteᶅ ϊΠζ͋Γ݁Ռ

Slide 34

Slide 34 text

࣮ݧ: Shape From Silhouetteᶅ Fuzzy MetaballsͷΈ

Slide 35

Slide 35 text

ٞ࿦ɾLimitations ● ຊख๏Ͱ͸ܗঢ়ΛϨϯμϦϯά͢Δ͜ͱʹয఺Λ౰ͯͨ ○ ࣮ݧ͸͍ͯ͠ͳ͍͕Fuzzy Metaballs͝ͱʹ৭ΛׂΓ౰ͯΔ͜ͱ͸Մೳ ● GPUʹಛԽͨ͠࠷దԽ͸͍ͯ͠ͳ͍ͷͰߴ଎Խͷ༨஍͋Γ ● Ϟσϧͷࣗ༝౓ͷ௿͔͞Β҉໧తͳਖ਼ଇԽ͕ಇ͖ϊΠζʹڧ͍ͱߟ͑ΒΕΔ ○ ηάϝϯςʔγϣϯϞσϧग़ྗͷΤϥʔमਖ਼ͳͲʹར༻Մೳ ● ࠷దԽ͞ΕͨFuzzy Metaballsͷύϥϝʔλ͸زԿతʹղऍ͕Ͱ͖Δ 
 (GMMͳͷͰ) ● ࣮ࡍͷϢʔεέʔεͱͯ͠ɺߴ඼࣭ͳը૾औಘ͕೉͍͠ͱ͖ʹ࢖͑ͦ͏

Slide 36

Slide 36 text

ٞ࿦ɾLimitations

Slide 37

Slide 37 text

·ͱΊɾײ૝ ● Fuzzy Metaballsͱ͍͏ࠞ߹Ψ΢ε෼෍Λݩʹͨ͠ 
 3DදݱΛఏҊ͠ɺۙࣅඍ෼ՄೳϨϯμϦϯάख๏ΛఏҊ ● Mesh΍఺܈ͱͷެฏੑΛߟ͑ͨධՁํ๏Λ༻͍ɺطଘͷํ๏ʹඖఢ͢Δਫ਼౓ 
 ͱͱ΋ʹߴ଎͔ͭϊΠζʹରͯ͠ϩόετͳ݁ՌʹͳΔ͜ͱΛࣔͨ͠ ● খܕҠಈϩϘοτͳͲGPUΛੵΈͨͯ͘΋ੵΊͳ͍Α͏ͳ༻్Ͱ 
 ΦϯϥΠϯͳܗঢ়ೝࣝͳͲͰ͖Ε͹໘നͦ͏

Slide 38

Slide 38 text

ࢀߟจݙ ● هࡌͷͳ͍ݶΓ͸Fuzzy Metaballsͷ࿦จ͔ΒҾ༻ ● ͦͷଞ ○ ϨΠτϨʔγϯάʹΑΔίϯϐϡʔλάϥϑΟΫεೖ໳ - ϝλ Ϙʔϧ - ○ ඍ෼ՄೳϨϯμϦϯά (CVIMݚڀձ νϡʔτϦΞϧ) ○ Differentiable Rendering: A Survey