$30 off During Our Annual Pro Sale. View Details »

ISPA 2023

ISPA 2023

Olivier Lézoray

September 20, 2023
Tweet

More Decks by Olivier Lézoray

Other Decks in Research

Transcript

  1. 1
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    No Reference 3D mesh quality assessment
    using deep convolutional features
    Zaineb Ibork(1,2), Anass Nouri(1), Olivier Lezoray(2), Christophe Charrier(2), Raja Touahni(1)
    (1)SETIME Laboratory, Information Processing and A.I Team, Faculty of Sciences, Ibn Tofail University,
    Kenitra, Morocco
    (2)Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France
    {zaineb.ibork,anass.nouri, touahni.raja}@uit.ac.ma {olivier.lezoray, christophe.charrier}@unicaen.fr
    le 18/09/2023

    View Slide

  2. 2
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 2
    Zaineb Ibork
    Plan Introduction and objectives
    01
    Methodology and Database
    construction
    02
    Experimentations and results
    04
    Proposed method
    03
    Conclusion
    05

    View Slide

  3. 3
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Problematic
    ➢ 3D Meshes can be altered by pre-processing steps as:
    acquisition, compression or denoising
    ➢ In this context, visual quality assessment algorithms can be used to quantify
    the amount of distortions that affect a 3D mesh and hence degrade its visual
    rendering.
    ➢ Objective visual quality assessment methods can be categorized based on the
    availability of a reference object:
    Full-Reference(FR), No-Reference(NR) and Reduced-Reference (RR) methods.
    ➢ We introduce a no-reference mesh quality assessment index based on deep
    convolutional features named:
    DCFQI (Deep Convolutional Features Quality Index)
    Objective
    ➢ Assess the visual quality of a 3D mesh without referring to its reference
    version.

    View Slide

  4. 4
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 4
    Zaineb Ibork
    Plan Introduction and objectives
    01
    Methodology and Database
    construction
    02
    Experimentations and results
    04
    Proposed method
    03
    Conclusion
    05

    View Slide

  5. 5
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Methodology
    ➢ We utilized a pre-trained CNN-based model with thirteen layers for
    feature extraction and an MLP with two layers for regression, considering
    2D representations of a mesh as input and generating a subjective quality
    score as the output.
    ➢ After training, the network became capable of predicting the quality of a
    mesh that was not part of the training phase.
    ➢ We constructed a base model using 'leave-one-model-out
    cross-validation'.
    ➢ Subsequently, we built another model using the same method, but it was
    initialized with weights from a model trained cumulatively on the
    database.

    View Slide

  6. 6
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Databases construction: 3D mesh rendering
    Armadillo’s 11 rendered views: the views in the first row are obtained by fixing θe = 0 and graduating θa by 60 degrees at
    a time. We switch this process in the second row

    View Slide

  7. 7
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Database construction: Transformation Process from View to Patches

    View Slide

  8. 8
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 8
    Zaineb Ibork
    Plan Introduction and objectives
    01
    Methodology and Database
    construction
    02
    Experimentations and results
    04
    Proposed method
    03
    Conclusion
    05

    View Slide

  9. 9
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    The pipeline of the proposed quality assessment index that estimate
    the quality of a rendered image (2D view or 2D view patch)

    View Slide

  10. 10
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 10
    Zaineb Ibork
    Plan Introduction and objectives
    01
    Methodology and Database
    construction
    02
    Experimentations and results
    04
    Proposed method
    03
    Conclusion
    05

    View Slide

  11. 11
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Description
    ➢ 88 models between 40K and 50K vertices generated from
    4 reference objects.
    ➢ Two types of distortion: noise addition and smoothing
    ➢ Distortions were applied with different strengths and at
    four locations: on the whole model, on smooth areas, on
    rough areas and on intermediate areas.
    ➢ Subjective evaluations were made at normal viewing
    distance, using a SSIS (Single Stimulus Impairment Scale)
    method with 12 observers.
    ➢ Each model is associated to a Mean Opinion Score after
    normalization and outlier removal.
    Mesh database: LIRIS/EPFL 3D Model General-Purpose
    Armadillo Dyno
    Venus RockerArm

    View Slide

  12. 12
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Mesh database: LIRIS/EPFL 3D Model General-Purpose
    Smoothed mesh
    Taubin15
    Noised mesh
    noise0015
    Original mesh

    View Slide

  13. 13
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Resulting neural networks issued from the leave-one-mesh-out
    cross-validation (LOMO-CV)

    View Slide

  14. 14
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Base Model
    SROOC VALUES FOR THE BASE MODEL TRAINED FOR 20 EPOCHS ON Bview OR Bpatch DATABASES

    View Slide

  15. 15
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Base Model
    SROOC VALUES FOR THE BASE MODEL TRAINED WITH AN EARLY STOPPING ON Bview AND Bpatch DATABASES

    View Slide

  16. 16
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Cumulative Model Training

    View Slide

  17. 17
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Cumulative Model results
    SROOC AND PCC CUMULATIVE MODEL (CM) VS RE-TRAINED CUMULATIVE MODEL (RCM) RESULTS
    TRAINED ON DATABASES BVIEW OR BPATCH

    View Slide

  18. 18
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Results versus SOTA
    COMPARISON OF OUR PROPOSED DCFQI VIEW/PATCH BASED BASE AND CUMULATIVE MODELS (DCFQI-VBM &
    DCFQI-VCM / DCFQI-PBM & DCFQI-PCM RESPECTIVELY) WITH THE STATE OF THE ART NO-REFERENCE METRICS.

    View Slide

  19. 19
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023 19
    Zaineb Ibork
    Plan Introduction and objectives
    01
    Methodology and Database
    construction
    02
    Experimentations and results
    04
    Proposed method
    03
    Conclusion
    05

    View Slide

  20. 20
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Conclusion
    Future works
    In this paper, we presented a no-reference mesh quality assessment approach:
    ➢ It renders the mesh in 2D views that can be subsequently divided in patches.
    ➢ From these images, deep features are extracted by the pre-trained VGG16
    CNN and fed into a MLP that performs quality prediction.
    ➢ This base model is competitive with the state-of-the-art, even at the view
    level.
    ➢ Finally a cumulative training has been proposed to obtain a single final model
    for prediction that goes beyond the state-of-the-art.
    Future works will consider
    ➢ Combining both view and patch predictions.
    ➢ The case of colored meshes.

    View Slide

  21. 21
    Zaineb Ibork 13th Int'l Symposium on Image and Signal Processing and Analysis ISPA 2023
    Thank you!
    Any questions?
    Special thanks: This work received funding from PHC TOUBKAL TBK/22/142-CAMPUS N°47259YH.

    View Slide