Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GNeRF: GAN-based Neural Radiance Field without Posed Camera

peisuke
December 12, 2021

GNeRF: GAN-based Neural Radiance Field without Posed Camera

第9回全日本コンピュータビジョン勉強会
ICCV2021論文読み回発表資料
GNeRF: GAN-based Neural Radiance Field without Posed Camera

peisuke

December 12, 2021
Tweet

More Decks by peisuke

Other Decks in Technology

Transcript

  1. GNeRF: GAN-based Neural Radiance
    Field without Posed Camera
    @peisuke
    9

    View Slide

  2. ABEJA
    Twitter @peisuke
    Github https://github.com/peisuke
    Qiita https://qiita.com/peisuke
    SlideShare https://www.slideshare.net/FujimotoKeisuke

    View Slide


  3. • GNeRF: GAN-based Neural Radiance Field without Posed Camera

    • NeRF


    • GAN Radiance Field
    • RF

    View Slide

  4. NeRF
    • / 5

    View Slide



  5. (x,y,z)T
    (φ1,θ1)T
    (φ2,θ2)T
    (x,y,z) φ2,θ2
    (x,y,z) φ1,θ1

    View Slide

  6. NeRF



    View Slide

  7. NeRF
    • NeRF in the Wild
    • NeRF
    • iNeRF

    iNeRF
    NeRF in the Wild BP

    View Slide

  8. 条件 iNeRF NeRF-- IDR GNeRF(提案⼿法)
    事前学習 必要 不要 不要 不要
    粗いカメラ座標 必要 必要 必要 不要
    前⽅のみの制約 不要 必要 不要 不要
    マスク 不要 不要 必要 不要


    View Slide





  9. • GAN Radiance Field
    • RF

    View Slide

  10. • Phase A Phase B

    View Slide

  11. RF
    • D
    E 3

    View Slide

  12. GAN Real

    • Discriminator Real
    • Dynamic Patch Sampling /
    16x16
    • Dynamic Patch Sampling

    View Slide

  13. GAN Fake

    • NeRF
    16x16
    • G

    View Slide

  14. • 6
    𝑹 =
    𝑟!!
    𝑟!"
    𝑟!#
    𝑟"!
    𝑟""
    𝑟"#
    𝑟#!
    𝑟#"
    𝑟##
    𝒓 = 𝑟!!
    𝑟!"
    𝑟!#
    𝑟"!
    𝑟""
    𝑟"#
    $
    • SO(3)
    NN
    • On the Continuity of Rotation Representations in Neural Networks

    View Slide

  15. • Inversion network
    • Static Patch Sampling
    • E
    • Static Patch Sampling

    View Slide

  16. G
    G
    D G

    View Slide


  17. • Inversion network
    • refinement

    View Slide


  18. Refinement

    View Slide





  19. • Photometric loss

    View Slide


  20. • Generator
    • NeRF
    • Discriminator
    • Instance Normalization Spectral Normalization
    • Inversion network
    • Vision Transformer

    • A
    • A B 1 iteration
    • B

    View Slide


  21. • COLMAP NeRF C+n
    • Refine C+r
    • Ground Truth NeRF G+n

    • Synthetic-NeRF image size: 400x400 DTU 500x400
    • 100

    • Peak Signal-to-Noise Ratio SSIM LPIPS
    • /
    • Single TAITAN GPU 30

    View Slide

  22. COLMAP

    View Slide

  23. • COLMAP
    Drums

    View Slide

  24. 難しいシーン
    特徴の少ない
    シーン

    View Slide

  25. IDR
    • 3D
    IDR
    • IDR
    • PNSR

    View Slide

  26. • COLMAP
    • Drums Lego COLMAP

    View Slide

  27. Ablation Study


    GAN InvNet. PhaseB

    • A B A ABABABAB B

    View Slide



  28. • COLMAP


    View Slide

  29. • NeRF

    • GAN RF
    • RF


    • COLMAP

    View Slide