VCIP 2025

Transcript

1. SALIENCY PREDICTION ON 3D MESHES USING RESIDUAL FEASTCONV-BASED GRAPH NEURAL NETWORKS Olivier L´ EZORAY1, Zaineb IBORK1,2, Anass NOURI1,2, Christophe CHARRIER1 1Universit´ e Caen Normandie, ENSICAEN, Normandie Univ, GREYC UMR 6072, Caen, FRANCE 2Laboratoire des Syst` emes ´ Electroniques, Traitement de l’Information, M´ ecanique et ´ Energ´ etique, Ibn Tofail University, Kenitra, MOROCCO olivier.lezoray@unicaen.fr https://lezoray.users.greyc.fr
2. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 2 / 27
3. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 3 / 27
4. Introduction ▶ Recent technological advances have led to the generation of huge amounts of 3D data ▶ Even with cheap hardware and software, one can easily generate 3D data ▶ 3D data acquisition with common smartphone (e.g., Photogrammetry) O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 4 / 27
5. New applications Fields ▶ With this proliferation of such 3D Data, new application fields have appeared ▶ Digital Forensics, Cultural Heritage, Body Scanning ▶ Whatever the field, it is often mandatory to identify the most important areas of the 3D content → Saliency O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 5 / 27
6. Mesh Saliency Saliency for images ? ▶ Salient regions of an image are visually more noticeable by their contrast with respect to surrounding regions Saliency for 3D meshes ? ▶ If a point from the 3D data stands out strongly from its surrounding, then, it could be considered as a salient 3D point. Mesh saliency application: adaptive mesh compression O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 6 / 27
7. Mesh Saliency – State of the Art Existing approaches ▶ Handcrafted features (Lee et al., Mesh Saliency, ACM TOG, 2005) O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 7 / 27
8. Mesh Saliency – State of the Art Existing approaches ▶ Handcrafted features (Lee et al., Mesh Saliency, ACM TOG, 2005) ▶ View-based CNNs (Song et al., 3D Visual Saliency: An Independent Perceptual Measure or A Derivative of 2D Image Saliency?, IEEE TPAMI, 2023) O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 7 / 27
9. Mesh Saliency – State of the Art Existing approaches ▶ Handcrafted features (Lee et al., Mesh Saliency, ACM TOG, 2005) ▶ View-based CNNs (Song et al., 3D Visual Saliency: An Independent Perceptual Measure or A Derivative of 2D Image Saliency?, IEEE TPAMI, 2023) ▶ PointNet attention-based methods (Liu et al., Attention-embedding mesh saliency, The Visual Computer, 2023) O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 7 / 27
10. Mesh Saliency – State of the Art Existing approaches ▶ Handcrafted features (Lee et al., Mesh Saliency, ACM TOG, 2005) ▶ View-based CNNs (Song et al., 3D Visual Saliency: An Independent Perceptual Measure or A Derivative of 2D Image Saliency?, IEEE TPAMI, 2023) ▶ PointNet attention-based methods (Liu et al., Attention-embedding mesh saliency, The Visual Computer, 2023) Observation Surprinsingly, no approaches based on Graph Neural Networks (GNNs) ⇒ Our proposal O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 7 / 27
11. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 8 / 27
12. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 9 / 27
13. 3D Mesh Saliency Estimation ▶ Goal: predict a saliency value per vertex on a 3D mesh ▶ Saliency represents the perceptual importance of each region ▶ We propose a tailored Graph Neural Network (GNN) that operates directly on the geometric and topological structure of triangular meshes ▶ Model name: SARMA – Saliency Analysis with a Residual Mesh-based Architecture O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 10 / 27
14. Mesh as a Graph ▶ Mesh represented as a graph: G = (V, E) ▶ Each vertex vi ∈ V → a node ▶ Each edge (vi , vj ) ∈ E → local mesh connectivity ▶ Vertex features: xi ∈ RFin ▶ Use both geometry and curvature: xi = (x, y, z, κ(vi ))T with Fin = 4 ▶ Curvature κ(vi ) captures local surface bending ⇒ correlates with perceptual saliency O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 11 / 27
15. Feature-Steered Convolution (FeaStConv) ▶ Core building block of SARMA (Verma et al., FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis, 2018) ▶ Dynamically learns filters based on input features ▶ For each vertex i: h(l) i = j∈N(i) w(l) ij · W(l)x(l) j ▶ N(i) → 1-ring neighbors ▶ W(l) is a learnable weight matrix at layer l that plays the role of classical convolution filters applied to neighbors, ▶ w(l) ij → learned attention weights via softmax over feature differences ▶ Adapts to local geometry → robust to irregular mesh structures O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 12 / 27
16. Residuals connection Problem: Deep GNNs suffer from oversmoothing, where node features become indistinguishable across layers, reducing model discriminability. A solution: Chen et al., Residual connections provably mitigate oversmoothing in graph neural networks, 2025 To fight oversmoothing, stabilize training and facilitate the learning of deep representations with our GNN, SARMA incorporates residual connections at each layer. x(l) = σ FeaStConv(l)(x(l−1), E) + r(l)(x(l−1)), where σ is an activation function and r(l) is a linear residual projection operator. O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 13 / 27
17. SARMA Architecture ▶ Three stacked FeaStConv layers with residual connections ▶ Residuals stabilize training and prevent over-smoothing ▶ Layer formulations: x(1) = ReLU(FeaStConv(1)(x)) + r(1)(x) x(2) = ReLU(FeaStConv(2)(x(1))) + x(1) x(3) = FeaStConv(3)(x(2)) + r(3)(x(2)) ▶ Final output: y = x(3) ∈ RN (saliency per vertex) O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 14 / 27
18. SARMA Architecture overview SARMA : Saliency Analysis with a Residual Mesh-based Architecture Curvature computation FeastConv Linear FeastConv Identity FeastConv Linear Prediction Ground Truth MSE Loss O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 15 / 27
19. Training Objective ▶ Supervised regression of vertex-level saliency ▶ Ground-truth saliency: ytrue ∈ [0, 1]N ▶ Loss function: Mean Squared Error (MSE) LMSE = 1 N N i=1 (yi − ytrue i )2 ▶ Input normalization: ▶ Coordinates centered and scaled to unit sphere ▶ Curvature standardized (zero-mean, unit variance) ▶ Output: continuous saliency values between 0 and 1 O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 16 / 27
20. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 17 / 27
21. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 18 / 27
22. The Schelling dataset ▶ The Schelling dataset (Chen et al., Schelling points on 3D surface meshes, ACM TOG, 2012) contains 400 meshes divided into 20 object categories ▶ Each mesh was annotated by humans with a collection of salient points ▶ A continuous saliency map was obtained by a Gaussian filtering of the discrete Schelling points O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 19 / 27
23. Results on the Schelling dataset Type of Method Method PLCC AUC Hancrafted features Multiscale Gaussian 0.2230 0.7000 Ranking patches 0.3010 0.7880 Spectral Processing 0.3240 0.8110 RPCA 0.3360 N/A CS2Point 0.4010 0.6930 Local to Global Saliency 0.4070 0.8560 Sparse metric-based 0.4303 0.8168 Cluster-based 0.4321 0.6072 Salient Regions 0.4370 0.8550 MF-M5P 0.4670 0.6144 View-based CNN Cfs-CNN 0.4550 0.8920 MIMO-GAN-CRF 0.4760 N/A PointNet-based Attention-embedding 0.4910 0.9120 GNN-based SARMA (Ours) 0.4952 0.7926 Table: PLCC and AUC results on the Schelling Dataset. O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 20 / 27
24. Comparison with the best SOTA approaches Cfs-CNN Attention Embedding SARMA GT O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 21 / 27
25. Ablation study Network components PLCC SARMA FeaStConv w/o curvature 0.323197 w/o residual connection SARMA FeaStConv w/o curvature 0.363045 SARMA FeaStConv w/o residual connection 0.389720 SARMA GCConv 0.444785 SARMA GAT 0.479774 SARMA FeaStConv 0.495245 Table: PLCC results on the Schelling Dataset without curvature at input, without residual connections, and with different convolution operators in our proposed SARMA architecture. O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 22 / 27
26. Visual results Figure: Results on the Schelling dataset. Right: reference. Left : prediction. O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 23 / 27
27. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 24 / 27
28. Outline 1. Introduction 2. SARMA (Saliency Analysis with a Residual Mesh-based Architecture) 3. Results 4. Conclusion O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 25 / 27
29. Conclusion ▶ Proposed SARMA: a novel GNN for saliency prediction on 3D meshes ▶ Uses Feature-Steered Convolutions (FeaStConv) for end-to-end learning ▶ Uses residual connections to prevent over-smoothing ▶ Combines geometric and topological cues with curvature-based features ▶ Outperforms SOTA methods on the Schelling dataset (highest PLCC, strong AUC) ▶ Ablation study confirms the key role of FeaStConv, curvature and residual connections in model accuracy O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 26 / 27
30. The End Code sooen available at : https://lezoray.users.greyc.fr/projects/VCIP2025 O. L´ ezoray et al.– VCIP 2025 Saliency Prediction on 3D Meshes Using Residual FeaStConv-Based Graph Neural Networks 27 / 27