$30 off During Our Annual Pro Sale. View Details »

VC'23 チュートリアル講演 『三次元メッシュと深層学習』

VC'23 チュートリアル講演 『三次元メッシュと深層学習』

VC'23 (https://visualcomputing.jp/vc2023/) で行ったチュートリアル講演のスライドです.

Tatsuya Yatagawa

September 17, 2023
Tweet

More Decks by Tatsuya Yatagawa

Other Decks in Technology

Transcript

  1. Visual Computing 2023

    View Slide

  2. CG
    Polygon Mesh
    (c) Autodesk, Inc.
    Simulation
    Rendering
    Modeling
    (c) Autodesk, Inc.
    5

    View Slide

  3. ...
    SIGGRAPH ( )
    Photoshop :
    People are looking at a presentation about very
    difficult math, particularly about geometry. There are
    about ten people, but only some of them can
    understand the presentation. So, there are three
    people in front of the white board discussing eagerly
    about the presentation, while the other people get
    bored.
    ( )
    10
    3
    6

    View Slide

  4. Gartner , Hype Cycle for Emerging Technologies 2023
    , ( )
    AI 2-5 . ( )
    → . ( )
    l
    l
    l
    l , etc.
    7

    View Slide

  5. , .
    8
    Polygon Mesh
    NeRF Deep Implicits Point Sets

    View Slide

  6. , .
    , CG ...
    9

    View Slide

  7. ...
    → Poisson Surface Reconstruction
    ...
    , alpha
    shape(*1) ball pivoting(*2) OK!
    *1) Delauney ,
    *2)
    3
    10

    View Slide

  8. (Polygonal Mesh Processing )
    l Remeshing ( )
    l Simplification ( )
    l Parameterization ( )
    l Smoothing ( )
    l Model Repair ( )
    12
    , ,
    ...
    , .

    View Slide

  9. l Differentiable Surface Triangulation [Rakotosaona et al., TOG 2021]
    l Neural Mesh Simplification [Potamias et al., CVPR 2022]
    l Learning Direction Fields for Quadrangulation [Dielen et al., SGP 2021]
    13
    Remeshing & Simplification
    [Rakotosaona et al., TOG 2021] [Dielen et al., SGP 2021]

    View Slide

  10. l Neural Subdivision [Liu et al., TOG 2020]
    l Neural Progressive Meshes [Chen et al., TOG 2023]
    14
    Remeshing & Simplification
    [Chen et al., TOG 2023]

    View Slide

  11. :
    1:
    2: ([Ohtake et al., VMV 2022]
    ) , 線
    15
    Smoothing & Denoising
    l NN ,
    ○ DNF-Net [Li et al., TVCG 2020]
    ○ GCN-Denoiser [Shen et al., TOG 2022])
    l
    ○ GeoBi-GNN [Zhang et al., CAD 2022]
    l Deep Image Prior [Ulyanov et al., CVPR 2018] ,
    . End-to-End .
    ○ Dual-DMP [Hattori et al., ECCV 2022])
    [Hattori et al., ECCV 2022]

    View Slide

  12. l Functional Map ,
    ○ Deep Functional Maps [Litany et al., ICCV 2017]
    ○ Weakly Supervised DFM [Sharma and Ovsjanikov 2020], etc.
    Parameterization 16
    [Litany et al., ICCV 2017]
    l Poincare disk 0, 1 !
    ○ AtlasNet [Groueix et al., CVPR 2018]
    ○ Neural Shape Maps [Morreale et al., CVPR 2021]
    ○ DA Wand [Liu et al., CVPR 2023], etc.
    [Morreale et al., CVPR 2021]

    View Slide

  13. l , .
    ( )
    l , DMP-Inpaint [Hattori et al., arXiv 2023]
    Model Repair 17
    Hattori et al., “Learning Self-Prior for Mesh Inpainting Using Self-Supervised GCNs,” arXiv, 2023.

    View Slide

  14. l
    l
    l
    3 .
    18
    Agenda
    [Hanocka et al., TOG 2019]
    [Sharp et al., TOG 2022]
    [Sun et al., CGF 2022]

    View Slide

  15. l CG CAD
    l ,
    19

    View Slide

  16. l
    l ,
    20

    View Slide

  17. Progressive Meshes .
    [Hoppe et al., SIGGRAPH 1996]
    ( )
    l →
    l → ,
    21

    View Slide

  18. Pooling , .
    , ( , ).
    22
    : Graph Pooling
    l Gated Global Pooling [Li et al., ICLR 2015]
    softmax
    global pooling
    l gPool Layer [Gao and Ji, ICML 2019]
    top-k , ,
    l diffpool Layer
    , softmax
    l Eigen Pooling [Ma et al., SIGKDD 2019]
    ,
    [Ma et al., SIGKDD 2019]

    View Slide

  19. l
    l ,
    l

    l

    l CG

    23

    View Slide

  20. l
    l
    l
    3 .
    24
    Agenda
    [Hanocka et al., TOG 2019]
    [Sharp et al., TOG 2022]
    [Sun et al., CGF 2022]

    View Slide

  21. l
    l
    l
    3 .
    25
    Agenda
    [Hanocka et al., TOG 2019]
    [Sharp et al., TOG 2022]
    [Sun et al., CGF 2022]

    View Slide

  22. Quiz:
    Hint: The Graph Neural Network Model
    1.
    26
    Answer: 2008
    SIGGRAPH, CVPR Neural Network 算
    (CVPR Network Flow, Bayesian Network 1 )

    View Slide

  23. “The Graph Neural Network Model” [Scarselli et al., IEEE Trans. NN, 2008]
    Feed-Forward
    Network (FFN)
    Almeida-Pinda
    Generalization of Back-Propagation to Recurrent
    Neural Networks (1987 ). Hinton
    Backpropagation .
    1.
    27
    g:
    f:
    ...Almeida-Pinda 𝜃
    𝑓(𝑥; 𝜃) 𝑥 = 𝑓(𝑥; 𝜃) 𝜃 =
    𝑓(𝜃; 𝑥) 𝜃

    View Slide

  24. ,
    :
    𝐡"
    #
    𝐡$
    #
    𝐡$%&
    #
    𝐡$%!
    # 𝐡"
    #%& =
    1
    𝑁
    '
    $∈𝒩(")
    𝜎 𝐖𝐡$
    # + 𝐛
    𝐡"
    #
    𝜎
    𝐖, 𝐛 , ( )
    1.
    28

    View Slide

  25. l : 𝑀#
    l : 𝑈#
    Message Passing Neural Network [Gilmer et al., ICML 2017]
    𝐡"
    #
    𝐡$
    #
    𝐡$%&
    #
    𝐡$%!
    #
    𝐦!
    "#$ = #
    %∈𝒩(!)
    𝑀"
    (𝐡!
    ", 𝐡%
    ", 𝑒!%
    )
    𝐡!
    "#$ = 𝑈"
    (𝐡!
    ", 𝐦!
    ")
    1.
    29

    View Slide

  26. GraphSAGE [Hamilton et al., NeurIPS 2017]
    l
    l Mean, LSTM, Max-pooling aggregators
    :
    1.
    30
    Graph Attention Network [Veličković et al., ICLR 2017]
    l Attention
    .
    Heterogeneous Graph Transformer [Hu et al., WWW 2020]
    l ,
    , Transformer GAT
    [Hu et al., WWW 2020]
    [Hamilton et al., NIPS 2017]

    View Slide

  27. l RNN
    l
    → Walk Attention AttWalk [Izhak et al., WACV 2022]
    1.
    31
    MeshWalker [Lahav and Tal, TOG 2020]
    ( , )

    View Slide

  28. : MPNN , .

    1.
    32
    l ACNN [Boscaini et al., NeurIPS 2016], GCNN [Masci et al., 3DRR 2015],
    MoNet [Monti et al., CVPR 2017], FeaStNet [Verma et al., CVPR 2018], etc.
    l ZerNet [Sun et al., CGF 2020]
    l Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021]
    l Gauge Equivalent Transformer [He et al., NeurIPS 2021], etc.
    [Monti et al., CVPR 2017]

    View Slide

  29. l , Exponential Map
    , ( )
    l Zernike
    ( )
    1.
    33
    ZerNet [Sun et al., CGF 2020]
    Zernike 𝑍!
    (𝑟, 𝜃)
    𝑓 𝑟, 𝜃 = #
    !*$
    +
    𝛼!
    𝑍!
    (𝑟, 𝜃)
    Zernike .
    .
    𝑓 ∗ 𝑔 𝑝 = #
    !*$
    +
    𝛼!
    ,𝛼!
    -

    View Slide

  30. Group Equivalent CNN (GE-CNN) [Cohen & Welling, NeurIPS 2016]
    1.
    34
    Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021]
    [Linmans et al., arXiv 2018]
    GE-CNN:
    ( ) CNN
    . Group (cyclic group) .
    2 (=
    )
    .

    View Slide

  31. GEM-CNN :
    1.
    35
    Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021]
    Gauge equivalent
    (gauge)
    . ,
    (Lagrangian) . ,
    (≒ ) .
    , 𝐾 𝑝
    , 𝑔.→0
    , 𝑞
    𝑝 .
    → ,

    View Slide

  32. l Zernike , ( )
    l ,
    l
    1.
    36
    :
    [Glimer et al., ICML 2017] MPNN
    [Pfaff et al., ICLR 2021]: MPNN

    View Slide

  33. l
    l
    l
    3 .
    37
    Agenda
    [Hanocka et al., TOG 2019]
    [Sharp et al., TOG 2022]
    [Sun et al., CGF 2022]

    View Slide

  34. l ,
    cf.) =
    l ( )
    :
    2.
    38

    View Slide

  35. =
    𝐾 ∗ 𝑓 = ℱ!"(ℱ 𝐾 ⋅ ℱ 𝑓 )
    𝑓# = ℱ!"(Θ ∘ ℱ 𝑓 )
    ,
    [Hammond et al., 2011]
    Hammond et al., “Wavelets on graphs via spectral graph theory,” Applied
    ComputationalHarmonic Analysis, 2011.
    2.
    39

    View Slide

  36. 0
    1
    2
    3
    𝐖 =
    0 1 1 0
    1 0 1 0
    1 1 0 1
    0 0 1 0
    :
    𝐋 = 𝐈 − 𝐃!
    "
    $𝐖𝐃!
    "
    $
    𝐷%%
    = 3
    &
    𝑊%&
    (𝐃 )
    𝐋:
    2.
    40

    View Slide

  37. ( )
    l
    𝐋 = 𝐔𝚲𝐔5 (𝐔 , 𝚲 )
    ,
    2.
    41

    View Slide

  38. = ( )
    ℱ 𝜔 = 2
    ,-
    -
    𝑓 𝑡 𝑒,!."/#𝑑𝑡
    ( 𝜔 , cos 2𝜋𝜔𝑡 + 𝑖 sin 2𝜋𝜔𝑡 )
    𝐟 = (𝑓$
    , 𝑓1
    , … , 𝑓2
    ) ,
    .
    ℱ 𝐟 = 𝐔(𝐟
    2.
    42

    View Slide

  39. l (or )
    ...
    𝐖 =
    0 1 1 0
    1 0 1 0
    1 1 0 1
    0 0 1 0
    𝐋 = 𝐈 − 𝐃!
    "
    $𝐖𝐃!
    "
    $
    l (= )
    l , (+ )
    ℱ 𝐟 = 𝐔(𝐟 ℱ!" 7
    𝐟 = 𝐔 7
    𝐟
    : :
    : :
    ( , 𝐔 )
    2.
    43

    View Slide

  40. ( )
    ( )
    1 →

    0

    0
    2.
    44

    View Slide

  41. Spectral CNN [Bruna et al., ICLR 2013]
    l
    l f 𝑔 ( ) ...
    7
    𝐟 = ℱ 𝐟 = 𝐔(𝐟,
    𝐟 ∗ 𝐠 = ℱ!" 7
    𝐟 ⊙ ;
    𝐠 = 𝐔<
    𝐆𝐔(𝐟
    ;
    𝐠 = ℱ 𝐠 = 𝐔(𝐠
    , E
    𝐆 G
    𝐠
    8
    𝐠
    2.
    45

    View Slide

  42. 1.
    l 𝑁 , 𝑁×𝑁 𝐔 ( : 𝒪 𝑁! )
    Spectral CNN
    𝐟 ∗ 𝐠 = ℱ!" 7
    𝐟 ⊙ ;
    𝐠 = 𝐔<
    𝐆𝐔(𝐟
    2.
    l
    l , ,
    2.
    46

    View Slide

  43. l 1 ,
    ChebNet [Defferrard et al., NeurIPS 2016]
    l →
    l 𝐋 𝐔
    𝐟0 = '
    123
    4,&
    𝜃1
    𝐋1𝐟 = 𝐔 '
    123
    4,&
    𝜃1
    𝚲1 𝐔5𝐟 (𝜃3
    )
    2.
    47

    View Slide

  44. Chebyshev
    l , 𝐋 → Chebyshev
    ChebNet [Defferrard et al., NeurIPS 2016]
    Chebyshev
    𝐟0 = '
    123
    4,&
    𝜃1𝑇1 𝐋 𝐟 = 𝐔 '
    123
    4,&
    𝜃1𝑇1(𝚲) 𝐔5𝐟
    𝑇)
    𝜆 = 2𝜆𝑇)!"
    𝜆 − 𝑇)!$
    𝜆
    𝑇)
    𝜆 = cos(𝑛𝜆)
    𝐋
    :
    2.
    48

    View Slide

  45. ChebNet
    l ,
    → [Kipf & Welling 2017] = ChebNet ,
    Graph Convolutional Network [Kipf & Welling, ICLR 2017]
    GCN ,
    (PyTorch Geometric )
    2.
    49

    View Slide

  46. Graph Convolutional Network [Kipf & Welling, ICLR 2017]
    GCN
    l Chebyshev 𝐾 = 2 , 𝐋 𝜆678 = 2
    𝐟0 = 2𝜃&𝑇&
    F
    𝐋 𝐟 − 𝜃3𝑇3
    F
    𝐋 𝐟 = 2𝜃&
    F
    𝐋𝐟 − 𝜃3𝐟
    = 2𝜃& 2 ⁄
    𝐋 2 − 𝐈 𝐟 − 𝜃3𝐟
    = −𝜃(𝐋 − 𝐈)𝐟 + 𝜃𝐟
    = 𝜃𝐃,
    &
    !𝐖𝐃,
    &
    !𝐟 + 𝜃𝐟
    = 𝜃 𝐈 + 𝐃,
    &
    !𝐖𝐃,
    &
    ! 𝐟 →
    = 𝜃&
    0 𝐋 − 𝐈 𝐟 + 𝜃3
    0𝐟
    l 𝜃 = 𝜃3
    0 = −𝜃&
    0
    2.
    50

    View Slide

  47. Graph Convolutional Network [Kipf & Welling, ICLR 2017]
    Rescaling Trick
    𝐟0 = 𝜃 𝐈 + 𝐃,!
    "𝐖𝐃,!
    " 𝐟 , 𝐈 + 𝐃,!
    "𝐖𝐃,!
    " [0, 2]
    → ,
    K
    𝐖 = 𝐈 + 𝐖 K
    𝐷"" = '
    $
    K
    𝑊"$ (*
    𝐃 )
    , .
    𝐟0 = 𝜃K
    𝐃,
    &
    ! K
    𝐖K
    𝐃,
    &
    !𝐟
    *
    𝐃!"
    #
    *
    𝐖*
    𝐃!"
    # [-1, 1] →
    GCN 1
    2.
    51

    View Slide


  48. : Kipf-Welling’s GCN
    2.
    52
    2. Chebyshev Spectral CNN (ChebNet) - [Defferrad et al., NeurIPS 2016]
    2 Chebyshev .
    .
    3. Graph Convolutional Network - [Kipf and Welling, ICLR 2017]
    ChebNet .
    , .
    1. Spectral CNN - [Bruna et al., ICLR 2013]
    . 2 .
    ( )
    ,

    View Slide

  49. Laplacian
    l CayleyNets [Levie et al., IEEE Trans SP 2018]
    l ChebNet II [He et al., NeurIPS 2022]
    l HodgeNet [Smirnov & Solomon, TOG 2021], etc.
    :
    2.
    53
    l DiffusionNet [Sharp et al., TOG 2022]
    l Laplacian Mesh Transformer [Li et al., ECCV 2022]
    l Laplacian Pooling Network [Qiao et al., TVCG 2022]
    l Laplacian2Mesh [Dong et al., TVCG 2023]

    View Slide

  50. Hodge
    2.
    54
    HodgeNet [Smirnov & Solomon, TOG 2021]
    Hodge
    , , , ,
    , ( ⋆!
    , ⋆"
    , ⋆#
    ).
    𝐋 =⋆3
    ,& 𝑑5 ⋆& 𝑑
    : Hodge Laplacian
    (𝑑 =)
    → Laplacian

    View Slide

  51. 2.
    55
    HodgeNet [Smirnov & Solomon, TOG 2021]
    Hodge
    : Hodge
    ⋆4
    (𝐹): 𝐹
    𝑓5
    : ℝ67 → ℝ NN . ,
    , .
    ⋆$
    (𝐹): , 4
    𝑔5
    : ℝ87 → ℝ NN
    .
    [Meyer et al., 2003]
    Meyer et al., “Discrete Differential Geometry Operators for Triangulated
    2-Manifolds,” Visualization and Mathematics III, 2003.
    ※ Laplacian , 𝑓$
    , 𝑔$
    → Laplacian ,

    View Slide

  52. Laplacian Spectral Clustering , .
    2.
    56
    Laplacian Pooling Network [Qiao et al., TVCG 2022]
    super-patch .
    Pooling/Unpooling .
    [Qiao et al., TVCG 2023]
    Spectral Clustering ,
    ( )
    [Qiao et al., TVCG 2023]

    View Slide

  53. Laplacian , (= Pooling /
    Unpooling) . , .
    2.
    57
    Laplacian2Mesh [Dong et al., TVCG 2023]
    : Spectral Compression of Mesh Geometry [Karni & Gotsman, SIGGRAPH 2000]
    [Zhang et al., CGF 2010]

    View Slide

  54. NN , (= )
    SE-ResNet Block [Hu et al., PAMI 2020]
    Pooling / Unpooling Laplacian
    2.
    58
    Laplacian2Mesh [Dong et al., TVCG 2023]

    View Slide

  55. ,
    2.
    59
    DiffusionNet [Sharp et al., TOG 2022]
    ℎ# 𝑢 = 𝑀 + 𝛿𝑡 ⋅ 𝐿 ,&𝑀𝑢
    (∆= 𝑀9$𝐿 ∆= ΦΛΦ: )
    ℎ# 𝑢 = Φ diag[𝑒,<##, 𝑒,l
    l
    (𝑀 , )

    View Slide

  56. :
    l 𝑢 𝑡 : ℎ#(𝑢)
    l 1-ring : 𝑧= = 𝐺𝑢
    l 𝑤>
    , 𝐴 .
    , 𝑤>
    , : Re b
    𝑤>
    ⊙ 𝐴𝑤>
    2.
    60
    DiffusionNet [Sharp et al., TOG 2022]

    View Slide

  57. : , ,
    DiffusionNet , DiffusionNet
    ,
    , .
    2.
    61
    DiffusionNet [Sharp et al., TOG 2022]
    : DiffusionNet , ,
    ,
    ,
    .

    View Slide

  58. l ,
    .
    l , , ...
    ( ...)
    2.
    62
    :

    View Slide

  59. Heat Kernel Signature [Sun et al., SGP 2009]
    , .
    2.
    63
    Hand-Crafted Feature ( )
    [Sun et al., SGP 2009]
    Heat Kernel Signature (HKS)
    .
    ,
    ( ) .
    l HKS-based Structural Encoding (HKSSE) [Wong, CVPR 2023]
    → , HKS MLP .
    l Mesh-MLP [Dong, arXiv Jun. 8th, 2023] (CVPR )
    → HKS , MLP ,
    . ( Concurrent )
    [Wong, CVPR 2023]
    HKS Laplacian ,
    .

    View Slide

  60. l
    l
    l
    3 .
    64
    Agenda
    [Hanocka et al., TOG 2019]
    [Sharp et al., TOG 2022]
    [Sun et al., CGF 2022]

    View Slide

  61. MeshNet [Feng et al., AAAI 2018]
    l ,
    3
    :
    MeshCNN [Hanocka et al., TOG 2019]
    l Progressive Meshes
    3.
    65

    View Slide

  62. MeshCNN [Hanocka et al., TOG 2019]
    &
    l
    l 4
    ,
    l (= )
    [Hanocka et al., TOG 2019]
    3.
    66

    View Slide

  63. (edge collapse)
    Progressive Meshes [Hoppe, SIGGRAPH 1996]
    l
    l
    3.
    67

    View Slide

  64. MeshCNN
    MeshCNN
    l →
    l →

    ,
    [Hanocka et al. 2019]
    3.
    68

    View Slide

  65. MeshCNN
    l Primal-Dual Mesh CNN [Milano et al., NeurIPS 2020]
    l HalfedgeCNN [Ludwig et al., CGF 2023]
    l PicassoNet++ [Lei et al., IEEE Trans. NN, 2023]
    l Vertex/Face-based MeshCNN [Perez et al., TVCG 2023]
    :
    3.
    69
    [Milano et al., NeurIPS 2020]
    Subdivision
    l SubdivNet [Hu et al., TOG 2022]
    l SPMM-Net [Shi et al., CAD 2023]

    View Slide

  66. Picasso . 2
    l :
    l :
    3.
    70
    PicassoNet++ [Lei et al., IEEE Trans. NN 2023]
    QEM , , QEM disjoint .
    QEM .

    [Lei et al., 2023]

    View Slide

  67. Loop Subdivision
    → 4 1
    3.
    71
    SubdivNet [Hu et al., TOG 2022]
    : Loop Subdivision
    :
    Loop Subdivision [Hu et al., 2022]
    [Hu et al., 2022]
    ※ MAPS [Lee et al., SIGGRAPH 1998] Neural Subdivision
    [Liu et al., TOG 2020]

    View Slide

  68. SubdivNet
    → Pooling
    3.
    72
    SPMM-Net [Shi et al., CAD 2023]
    SubdivNet [Hu et al., TOG 2022]
    QEM
    , Pooling .
    , Hourglass
    HRNet [Wang et al., PAMI 2019] .
    , , .

    View Slide

  69. l MeshCNN , ,
    , .
    l , ,
    . SubdivNet CAE .
    l Task-specific Mesh Decimation
    (PicassoNet++ QEM )
    3.
    73
    :
    [Ludwig et al., CGF 2023]
    Convolution/Pooling

    View Slide

  70. l (MPNN , GCN )
    → GNN CG
    74
    C G
    DS
    l Remeshing Smoothing
    , Model Repair Parameterization
    l (
    ) ,
    → ,

    View Slide

  71. 1:
    ,
    . 頂 .
    75
    Take Home Message: ( , )
    . , 1 .
    NeRF AI ,
    [Sahillioğlu & Horsman, TOG 2022]
    2:
    ,
    .

    View Slide