VC'23 チュートリアル講演『三次元メッシュと深層学習』

Slide 1

Slide 1 text

Visual Computing 2023

Slide 2

Slide 2 text

CG Polygon Mesh (c) Autodesk, Inc. Simulation Rendering Modeling (c) Autodesk, Inc. 5

Slide 3

Slide 3 text

... SIGGRAPH ( ) Photoshop : People are looking at a presentation about very difficult math, particularly about geometry. There are about ten people, but only some of them can understand the presentation. So, there are three people in front of the white board discussing eagerly about the presentation, while the other people get bored. ( ) 10 3 6

Slide 4

Slide 4 text

Gartner , Hype Cycle for Emerging Technologies 2023 , ( ) AI 2-5 . ( ) → . ( ) l l l l , etc. 7

Slide 5

Slide 5 text

, . 8 Polygon Mesh NeRF Deep Implicits Point Sets

Slide 6

Slide 6 text

, . , CG ... 9

Slide 7

Slide 7 text

... → Poisson Surface Reconstruction ... , alpha shape(*1) ball pivoting(*2) OK! *1) Delauney , *2) 3 10

Slide 8

Slide 8 text

(Polygonal Mesh Processing ) l Remeshing ( ) l Simplification ( ) l Parameterization ( ) l Smoothing ( ) l Model Repair ( ) 12 , , ... , .

Slide 9

Slide 9 text

l Differentiable Surface Triangulation [Rakotosaona et al., TOG 2021] l Neural Mesh Simplification [Potamias et al., CVPR 2022] l Learning Direction Fields for Quadrangulation [Dielen et al., SGP 2021] 13 Remeshing & Simplification [Rakotosaona et al., TOG 2021] [Dielen et al., SGP 2021]

Slide 10

Slide 10 text

l Neural Subdivision [Liu et al., TOG 2020] l Neural Progressive Meshes [Chen et al., TOG 2023] 14 Remeshing & Simplification [Chen et al., TOG 2023]

Slide 11

Slide 11 text

: 1: 2: ([Ohtake et al., VMV 2022] ) , 線 15 Smoothing & Denoising l NN , ○ DNF-Net [Li et al., TVCG 2020] ○ GCN-Denoiser [Shen et al., TOG 2022]) l ○ GeoBi-GNN [Zhang et al., CAD 2022] l Deep Image Prior [Ulyanov et al., CVPR 2018] , . End-to-End . ○ Dual-DMP [Hattori et al., ECCV 2022]) [Hattori et al., ECCV 2022]

Slide 12

Slide 12 text

l Functional Map , ○ Deep Functional Maps [Litany et al., ICCV 2017] ○ Weakly Supervised DFM [Sharma and Ovsjanikov 2020], etc. Parameterization 16 [Litany et al., ICCV 2017] l Poincare disk 0, 1 ! ○ AtlasNet [Groueix et al., CVPR 2018] ○ Neural Shape Maps [Morreale et al., CVPR 2021] ○ DA Wand [Liu et al., CVPR 2023], etc. [Morreale et al., CVPR 2021]

Slide 13

Slide 13 text

l , . ( ) l , DMP-Inpaint [Hattori et al., arXiv 2023] Model Repair 17 Hattori et al., “Learning Self-Prior for Mesh Inpainting Using Self-Supervised GCNs,” arXiv, 2023.

Slide 14

Slide 14 text

l l l 3 . 18 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]

Slide 15

Slide 15 text

l CG CAD l , 19

Slide 16

Slide 16 text

l l , 20

Slide 17

Slide 17 text

Progressive Meshes . [Hoppe et al., SIGGRAPH 1996] ( ) l → l → , 21

Slide 18

Slide 18 text

Pooling , . , ( , ). 22 : Graph Pooling l Gated Global Pooling [Li et al., ICLR 2015] softmax global pooling l gPool Layer [Gao and Ji, ICML 2019] top-k , , l diffpool Layer , softmax l Eigen Pooling [Ma et al., SIGKDD 2019] , [Ma et al., SIGKDD 2019]

Slide 19

Slide 19 text

l l , l → l → l CG → 23

Slide 20

Slide 20 text

l l l 3 . 24 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]

Slide 21

Slide 21 text

l l l 3 . 25 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]

Slide 22

Slide 22 text

Quiz: Hint: The Graph Neural Network Model 1. 26 Answer: 2008 SIGGRAPH, CVPR Neural Network 算 (CVPR Network Flow, Bayesian Network 1 )

Slide 23

Slide 23 text

“The Graph Neural Network Model” [Scarselli et al., IEEE Trans. NN, 2008] Feed-Forward Network (FFN) Almeida-Pinda Generalization of Back-Propagation to Recurrent Neural Networks (1987 ). Hinton Backpropagation . 1. 27 g: f: ...Almeida-Pinda 𝜃 𝑓(𝑥; 𝜃) 𝑥 = 𝑓(𝑥; 𝜃) 𝜃 = 𝑓(𝜃; 𝑥) 𝜃

Slide 24

Slide 24 text

, : 𝐡" # 𝐡$ # 𝐡$%& # 𝐡$%! # 𝐡" #%& = 1 𝑁 ' $∈𝒩(") 𝜎 𝐖𝐡$ # + 𝐛 𝐡" # 𝜎 𝐖, 𝐛 , ( ) 1. 28

Slide 25

Slide 25 text

l : 𝑀# l : 𝑈# Message Passing Neural Network [Gilmer et al., ICML 2017] 𝐡" # 𝐡$ # 𝐡$%& # 𝐡$%! # 𝐦! "#$ = # %∈𝒩(!) 𝑀" (𝐡! ", 𝐡% ", 𝑒!% ) 𝐡! "#$ = 𝑈" (𝐡! ", 𝐦! ") 1. 29

Slide 26

Slide 26 text

GraphSAGE [Hamilton et al., NeurIPS 2017] l l Mean, LSTM, Max-pooling aggregators : 1. 30 Graph Attention Network [Veličković et al., ICLR 2017] l Attention . Heterogeneous Graph Transformer [Hu et al., WWW 2020] l , , Transformer GAT [Hu et al., WWW 2020] [Hamilton et al., NIPS 2017]

Slide 27

Slide 27 text

l RNN l → Walk Attention AttWalk [Izhak et al., WACV 2022] 1. 31 MeshWalker [Lahav and Tal, TOG 2020] ( , )

Slide 28

Slide 28 text

: MPNN , . → 1. 32 l ACNN [Boscaini et al., NeurIPS 2016], GCNN [Masci et al., 3DRR 2015], MoNet [Monti et al., CVPR 2017], FeaStNet [Verma et al., CVPR 2018], etc. l ZerNet [Sun et al., CGF 2020] l Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] l Gauge Equivalent Transformer [He et al., NeurIPS 2021], etc. [Monti et al., CVPR 2017]

Slide 29

Slide 29 text

l , Exponential Map , ( ) l Zernike ( ) 1. 33 ZerNet [Sun et al., CGF 2020] Zernike 𝑍! (𝑟, 𝜃) 𝑓 𝑟, 𝜃 = # !*$ + 𝛼! 𝑍! (𝑟, 𝜃) Zernike . . 𝑓 ∗ 𝑔 𝑝 = # !*$ + 𝛼! ,𝛼! -

Slide 30

Slide 30 text

Group Equivalent CNN (GE-CNN) [Cohen & Welling, NeurIPS 2016] 1. 34 Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] [Linmans et al., arXiv 2018] GE-CNN: ( ) CNN . Group (cyclic group) . 2 (= ) .

Slide 31

Slide 31 text

GEM-CNN : 1. 35 Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] Gauge equivalent (gauge) . , (Lagrangian) . , (≒ ) . , 𝐾 𝑝 , 𝑔.→0 , 𝑞 𝑝 . → ,

Slide 32

Slide 32 text

l Zernike , ( ) l , l 1. 36 : [Glimer et al., ICML 2017] MPNN [Pfaff et al., ICLR 2021]: MPNN

Slide 33

Slide 33 text

l l l 3 . 37 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]

Slide 34

Slide 34 text

l , cf.) = l ( ) : 2. 38

Slide 35

Slide 35 text

= 𝐾 ∗ 𝑓 = ℱ!"(ℱ 𝐾 ⋅ ℱ 𝑓 ) 𝑓# = ℱ!"(Θ ∘ ℱ 𝑓 ) , [Hammond et al., 2011] Hammond et al., “Wavelets on graphs via spectral graph theory,” Applied ComputationalHarmonic Analysis, 2011. 2. 39

Slide 36

Slide 36 text

0 1 2 3 𝐖 = 0 1 1 0 1 0 1 0 1 1 0 1 0 0 1 0 : 𝐋 = 𝐈 − 𝐃! " $𝐖𝐃! " $ 𝐷%% = 3 & 𝑊%& (𝐃 ) 𝐋: 2. 40

Slide 37

Slide 37 text

( ) l 𝐋 = 𝐔𝚲𝐔5 (𝐔 , 𝚲 ) , 2. 41

Slide 38

Slide 38 text

= ( ) ℱ 𝜔 = 2 ,- - 𝑓 𝑡 𝑒,!."/#𝑑𝑡 ( 𝜔 , cos 2𝜋𝜔𝑡 + 𝑖 sin 2𝜋𝜔𝑡 ) 𝐟 = (𝑓$ , 𝑓1 , … , 𝑓2 ) , . ℱ 𝐟 = 𝐔(𝐟 2. 42

Slide 39

Slide 39 text

l (or ) ... 𝐖 = 0 1 1 0 1 0 1 0 1 1 0 1 0 0 1 0 𝐋 = 𝐈 − 𝐃! " $𝐖𝐃! " $ l (= ) l , (+ ) ℱ 𝐟 = 𝐔(𝐟 ℱ!" 7 𝐟 = 𝐔 7 𝐟 : : : : ( , 𝐔 ) 2. 43

Slide 40

Slide 40 text

( ) ( ) 1 → − 0 − 0 2. 44

Slide 41

Slide 41 text

Spectral CNN [Bruna et al., ICLR 2013] l l f 𝑔 ( ) ... 7 𝐟 = ℱ 𝐟 = 𝐔(𝐟, 𝐟 ∗ 𝐠 = ℱ!" 7 𝐟 ⊙ ; 𝐠 = 𝐔< 𝐆𝐔(𝐟 ; 𝐠 = ℱ 𝐠 = 𝐔(𝐠 , E 𝐆 G 𝐠 8 𝐠 2. 45

Slide 42

Slide 42 text

1. l 𝑁 , 𝑁×𝑁 𝐔 ( : 𝒪 𝑁! ) Spectral CNN 𝐟 ∗ 𝐠 = ℱ!" 7 𝐟 ⊙ ; 𝐠 = 𝐔< 𝐆𝐔(𝐟 2. l l , , 2. 46

Slide 43

Slide 43 text

l 1 , ChebNet [Defferrard et al., NeurIPS 2016] l → l 𝐋 𝐔 𝐟0 = ' 123 4,& 𝜃1 𝐋1𝐟 = 𝐔 ' 123 4,& 𝜃1 𝚲1 𝐔5𝐟 (𝜃3 ) 2. 47

Slide 44

Slide 44 text

Chebyshev l , 𝐋 → Chebyshev ChebNet [Defferrard et al., NeurIPS 2016] Chebyshev 𝐟0 = ' 123 4,& 𝜃1𝑇1 𝐋 𝐟 = 𝐔 ' 123 4,& 𝜃1𝑇1(𝚲) 𝐔5𝐟 𝑇) 𝜆 = 2𝜆𝑇)!" 𝜆 − 𝑇)!$ 𝜆 𝑇) 𝜆 = cos(𝑛𝜆) 𝐋 : 2. 48

Slide 45

Slide 45 text

ChebNet l , → [Kipf & Welling 2017] = ChebNet , Graph Convolutional Network [Kipf & Welling, ICLR 2017] GCN , (PyTorch Geometric ) 2. 49

Slide 46

Slide 46 text

Graph Convolutional Network [Kipf & Welling, ICLR 2017] GCN l Chebyshev 𝐾 = 2 , 𝐋 𝜆678 = 2 𝐟0 = 2𝜃&𝑇& F 𝐋 𝐟 − 𝜃3𝑇3 F 𝐋 𝐟 = 2𝜃& F 𝐋𝐟 − 𝜃3𝐟 = 2𝜃& 2 ⁄ 𝐋 2 − 𝐈 𝐟 − 𝜃3𝐟 = −𝜃(𝐋 − 𝐈)𝐟 + 𝜃𝐟 = 𝜃𝐃, & !𝐖𝐃, & !𝐟 + 𝜃𝐟 = 𝜃 𝐈 + 𝐃, & !𝐖𝐃, & ! 𝐟 → = 𝜃& 0 𝐋 − 𝐈 𝐟 + 𝜃3 0𝐟 l 𝜃 = 𝜃3 0 = −𝜃& 0 2. 50

Slide 47

Slide 47 text

Graph Convolutional Network [Kipf & Welling, ICLR 2017] Rescaling Trick 𝐟0 = 𝜃 𝐈 + 𝐃,! "𝐖𝐃,! " 𝐟 , 𝐈 + 𝐃,! "𝐖𝐃,! " [0, 2] → , K 𝐖 = 𝐈 + 𝐖 K 𝐷"" = ' $ K 𝑊"$ (* 𝐃 ) , . 𝐟0 = 𝜃K 𝐃, & ! K 𝐖K 𝐃, & !𝐟 * 𝐃!" # * 𝐖* 𝐃!" # [-1, 1] → GCN 1 2. 51

Slide 48

Slide 48 text

→ : Kipf-Welling’s GCN 2. 52 2. Chebyshev Spectral CNN (ChebNet) - [Defferrad et al., NeurIPS 2016] 2 Chebyshev . . 3. Graph Convolutional Network - [Kipf and Welling, ICLR 2017] ChebNet . , . 1. Spectral CNN - [Bruna et al., ICLR 2013] . 2 . ( ) ,

Slide 49

Slide 49 text

Laplacian l CayleyNets [Levie et al., IEEE Trans SP 2018] l ChebNet II [He et al., NeurIPS 2022] l HodgeNet [Smirnov & Solomon, TOG 2021], etc. : 2. 53 l DiffusionNet [Sharp et al., TOG 2022] l Laplacian Mesh Transformer [Li et al., ECCV 2022] l Laplacian Pooling Network [Qiao et al., TVCG 2022] l Laplacian2Mesh [Dong et al., TVCG 2023]

Slide 50

Slide 50 text

Hodge 2. 54 HodgeNet [Smirnov & Solomon, TOG 2021] Hodge , , , , , ( ⋆! , ⋆" , ⋆# ). 𝐋 =⋆3 ,& 𝑑5 ⋆& 𝑑 : Hodge Laplacian (𝑑 ＝) → Laplacian

Slide 51

Slide 51 text

2. 55 HodgeNet [Smirnov & Solomon, TOG 2021] Hodge : Hodge ⋆4 (𝐹): 𝐹 𝑓5 : ℝ67 → ℝ NN . , , . ⋆$ (𝐹): , 4 𝑔5 : ℝ87 → ℝ NN . [Meyer et al., 2003] Meyer et al., “Discrete Differential Geometry Operators for Triangulated 2-Manifolds,” Visualization and Mathematics III, 2003. ※ Laplacian , 𝑓$ , 𝑔$ → Laplacian ,

Slide 52

Slide 52 text

Laplacian Spectral Clustering , . 2. 56 Laplacian Pooling Network [Qiao et al., TVCG 2022] super-patch . Pooling/Unpooling . [Qiao et al., TVCG 2023] Spectral Clustering , ( ) [Qiao et al., TVCG 2023]

Slide 53

Slide 53 text

Laplacian , (= Pooling / Unpooling) . , . 2. 57 Laplacian2Mesh [Dong et al., TVCG 2023] : Spectral Compression of Mesh Geometry [Karni & Gotsman, SIGGRAPH 2000] [Zhang et al., CGF 2010]

Slide 54

Slide 54 text

NN , (= ) SE-ResNet Block [Hu et al., PAMI 2020] Pooling / Unpooling Laplacian 2. 58 Laplacian2Mesh [Dong et al., TVCG 2023]

Slide 55

Slide 55 text

, 2. 59 DiffusionNet [Sharp et al., TOG 2022] ℎ# 𝑢 = 𝑀 + 𝛿𝑡 ⋅ 𝐿 ,&𝑀𝑢 (∆= 𝑀9$𝐿 ∆= ΦΛΦ: ) ℎ# 𝑢 = Φ diag[𝑒,<##, 𝑒,

Slide 56

Slide 56 text

: l 𝑢 𝑡 : ℎ#(𝑢) l 1-ring : 𝑧= = 𝐺𝑢 l 𝑤> , 𝐴 . , 𝑤> , : Re b 𝑤> ⊙ 𝐴𝑤> 2. 60 DiffusionNet [Sharp et al., TOG 2022]

Slide 57

Slide 57 text

: , , DiffusionNet , DiffusionNet , , . 2. 61 DiffusionNet [Sharp et al., TOG 2022] : DiffusionNet , , , , .

Slide 58

Slide 58 text

l , . l , , ... ( ...) 2. 62 :

Slide 59

Slide 59 text

Heat Kernel Signature [Sun et al., SGP 2009] , . 2. 63 Hand-Crafted Feature ( ) [Sun et al., SGP 2009] Heat Kernel Signature (HKS) . , ( ) . l HKS-based Structural Encoding (HKSSE) [Wong, CVPR 2023] → , HKS MLP . l Mesh-MLP [Dong, arXiv Jun. 8th, 2023] (CVPR ) → HKS , MLP , . ( Concurrent ) [Wong, CVPR 2023] HKS Laplacian , .

Slide 60

Slide 60 text

l l l 3 . 64 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]

Slide 61

Slide 61 text

MeshNet [Feng et al., AAAI 2018] l , 3 : MeshCNN [Hanocka et al., TOG 2019] l Progressive Meshes 3. 65

Slide 62

Slide 62 text

MeshCNN [Hanocka et al., TOG 2019] & l l 4 , l (= ) [Hanocka et al., TOG 2019] 3. 66

Slide 63

Slide 63 text

(edge collapse) Progressive Meshes [Hoppe, SIGGRAPH 1996] l l 3. 67

Slide 64

Slide 64 text

MeshCNN MeshCNN l → l → → , [Hanocka et al. 2019] 3. 68

Slide 65

Slide 65 text

MeshCNN l Primal-Dual Mesh CNN [Milano et al., NeurIPS 2020] l HalfedgeCNN [Ludwig et al., CGF 2023] l PicassoNet++ [Lei et al., IEEE Trans. NN, 2023] l Vertex/Face-based MeshCNN [Perez et al., TVCG 2023] : 3. 69 [Milano et al., NeurIPS 2020] Subdivision l SubdivNet [Hu et al., TOG 2022] l SPMM-Net [Shi et al., CAD 2023]

Slide 66

Slide 66 text

Picasso . 2 l : l : 3. 70 PicassoNet++ [Lei et al., IEEE Trans. NN 2023] QEM , , QEM disjoint . QEM . → [Lei et al., 2023]

Slide 67

Slide 67 text

Loop Subdivision → 4 1 3. 71 SubdivNet [Hu et al., TOG 2022] : Loop Subdivision : Loop Subdivision [Hu et al., 2022] [Hu et al., 2022] ※ MAPS [Lee et al., SIGGRAPH 1998] Neural Subdivision [Liu et al., TOG 2020]

Slide 68

Slide 68 text

SubdivNet → Pooling 3. 72 SPMM-Net [Shi et al., CAD 2023] SubdivNet [Hu et al., TOG 2022] QEM , Pooling . , Hourglass HRNet [Wang et al., PAMI 2019] . , , .

Slide 69

Slide 69 text

l MeshCNN , , , . l , , . SubdivNet CAE . l Task-specific Mesh Decimation (PicassoNet++ QEM ) 3. 73 : [Ludwig et al., CGF 2023] Convolution/Pooling

Slide 70

Slide 70 text

l (MPNN , GCN ) → GNN CG 74 C G DS l Remeshing Smoothing , Model Repair Parameterization l ( ) , → ,

Slide 71

Slide 71 text

1: , . 頂 . 75 Take Home Message: ( , ) . , 1 . NeRF AI , [Sahillioğlu & Horsman, TOG 2022] 2: , .