NIPS2017reading_3Dreconstruction

Transcript

1. ̏࣍ݩ෮ݩʹؔͯ͠ Learning a Multi-View Stereo Machine NIPS2017࿦จಡΈձˏΫοΫύου 1 ಛʹදه͕ͳ͍ݶΓɺҎԼͷࢿྉ͔ΒҾ༻ https://arxiv.org/pdf/1708.05375.pdf
2. Learning a Multi-View Stereo Machine ▸ චऀ • Abhishek Kar, Christian Häne, Jitendra Malik ʢUC Berkeley) ▸ ֓ཁ • Multi View StereoʢMVSʣʹΑΔີͳ3࣍ݩ෮ݩΛDeep LearningͰEnd2Endʹֶश • MVSΛ”ֶशͰ͖Δ”ͷͰ͸ແ͍͔ͱ͍͏ٙ໰ʹ౴͑Δ 2
3. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ 2. Ϛονϯά 3. ̏࣍ݩ෮ݩ 4. Τϥʔͷআڈ 3
4. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ 2. Ϛονϯά 3. ̏࣍ݩ෮ݩ 4. Τϥʔͷআڈ ==> DeepԿ๭ͰશͯղܾͰ͖ͦ͏ 4
5. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ɹ← CNNͰ͍͚Δ 2. Ϛονϯά 3. ̏࣍ݩ෮ݩ 4. Τϥʔͷআڈ 5
6. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ 2. Ϛονϯάɹ← CNNͱRNNͰ͍͚Δ 3. ̏࣍ݩ෮ݩ 4. Τϥʔͷআڈ 6
7. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ 2. Ϛονϯά 3. ̏࣍ݩ෮ݩɹ← DeconvͰ͍͚Δ 4. Τϥʔͷআڈ 7
8. എܠ ▸ Multi View Stereoͱ͸ 1. ಛ௃఺நग़ 2. Ϛονϯά 3. ̏࣍ݩ෮ݩ 4. Τϥʔͷআڈɹ← Encoder-DecoderͰ͍͚Δ 8
9. DeepԿ๭Ͱࡾ࣍ݩ෮ݩ ▸ 3DR2N2(ECCV2016) • ෳ਺ը૾ΛΤϯίʔυ͠ɺLSTMͰϚονϯά 9 http://3d-r2n2.stanford.edu
10. DeepԿ๭Ͱࡾ࣍ݩ෮ݩ ▸ 3D Shape Reconstruction by Modeling 2.5D Sketch (NIPS2017) • ϦΞϧͷը૾͔Β2.5DͷεέονΛى͜͠ɺ2.5DεέονΛ΋ͱʹ 3DshapeਪఆΛEnd2EndֶशͰ͢Δ 10 https://arxiv.org/pdf/1711.03129.pdf
11. ࿩͢಺༰ ▸ શମ૾ ▸ ख๏ ▸ ࣮ݧ ▸ ·ͱΊ 11
12. શମ૾ 12 http://bair.berkeley.edu/blog/2017/09/05/unified-3d/
13. શମ૾ 13 Learnt Stereo Machines
14. ख๏ ▸ Image Encoder • Encoder-DecoderܕʢU-netʣͷ૚ઃܭ • Ϛονϯάʹ༻͍Δ̎Dͷಛ௃Ϛοϓ࡞੒ • ࣍ݩ2Dnಛ௃Ϛο 14
15. ख๏ ▸ Unplojection ▸ 2࣍ݩͷಛ௃Ϛοϓ͸3࣍ݩͷຊདྷ͋Δ΂͖ಛ௃Ϛοϓ͔ΒࣹӨ ▸ 3࣍ݩάϦουʹٯࣹӨ 15 http://bair.berkeley.edu/blog/2017/09/05/unified-3d/
16. ख๏ ▸ Unplojection ▸ 2࣍ݩͷಛ௃Ϛοϓ͸3࣍ݩͷຊདྷ͋Δ΂͖ಛ௃Ϛοϓ͔ΒࣹӨ ▸ 3࣍ݩάϦουʹٯࣹӨ 16 http://bair.berkeley.edu/blog/2017/09/05/unified-3d/
17. ख๏ ▸ Unplohection ▸ 2࣍ݩͷಛ௃Ϛοϓ͸3࣍ݩͷຊདྷ͋Δ΂͖ಛ௃Ϛοϓ͔ΒࣹӨ ▸ 3࣍ݩάϦουʹٯࣹӨ 17 http://bair.berkeley.edu/blog/2017/09/05/unified-3d/
18. ख๏ ▸ Unplohection ▸ 2࣍ݩͷಛ௃Ϛοϓ͸3࣍ݩͷຊདྷ͋Δ΂͖ಛ௃Ϛοϓ͔ΒࣹӨ ▸ 3࣍ݩάϦουʹٯࣹӨ 18 http://bair.berkeley.edu/blog/2017/09/05/unified-3d/
19. ख๏ ▸ Recurrent Grid Fusion • 3࣍ݩͷಛ௃ϚοϓͷϚονϯάΛGated Recurrent Unit(GRU)Ͱ • GRUʹ͍࣋ͬͯͨ͘Ίɺ3D convolutionΛ࢖༻ • ͜ͷաఔ͕MVSͷܭࢉϚονϯάΛ୲౰ • ֶशͷࡍ͸ը૾ͷೖྗॱΛϥϯμϜʹೖΕସ͑Δ 19
20. ख๏ ▸ 3D Grid Reasoning • GRUͰ̏࣍ݩάϦουʹͨ͠ΒϊΠζ͕ଟ͔ͬͨɻ • 3U-netͰEncode Decode͢ΔͱFilteringͰ͖Δ 20
21. ख๏ ▸ Differentiable Projection • Depthͷ෮ݩʹ͸L1 loss(high frequency informationͷͨΊ) • Voxelͷ෮ݩʹ͸voxel͝ͱͷcross entropy loss 21
22. ࣮ݧ ▸ σʔληοτ • ShapeNetσʔλΛར༻ • ̏࣍ݩCADϞσϧͷެ։σʔληοτ 22 https://shapenet.cs.stanford.edu/shrec17/
23. ࣮ݧ • ೖྗը૾ ▸ ShapeNetͷ3DϞσϧΛϨϯμϦϯάͯ͠224x224x3 ▸ ̍ࢹ఺͋ͨΓ̐ຕ ▸ Χϝϥϙʔζ • Ξ΢τϓοτ ▸ Depth: 224x224x3 ▸ Voxel: 32x32x32 23
24. ࣮ݧ ▸ ݁Ռ 24 3DR2N2ͱൺ΂ɺࡉ͔͍෮ݩ͕Մೳ
25. ࣮ݧ ▸ ݁Ռ 25 3DR2N2ͱൺ΂ɺগͳ͍ຕ਺Ͱ෮ݩ͕Մೳ ຕ਺૿͑Δͱੑೳ্͕͕Δ
26. ࣮ݧ ▸ ݁Ռ 26 stereo matchingͰ͸෮ݩ͠ͳ͍ ૭΋෮ݩՄೳ
27. ࣮ݧ ▸ ݁Ռ 27 stereo matchingʹൺ΂
 গͳ͍ຕ਺Ͱ΋෮ݩ͕Մೳ චऀᐌ͘ CNNͷίϯςΫετΛݟΔྗ͸
 ैདྷͷstereo matchingΛ͙྇ DepthMapͷਪఆ݁ՌΛෳ਺૊Έ߹Θͤͯ̏࣍ݩ෮ݩͨ͠
28. ·ͱΊ ▸ Learnt Stereo MachinesΛఏҊ ▸ ෳ਺ࢹ఺͔Βͷೖྗը૾Λݩʹɺ
 DepthMapͱVoxelͷਪఆ͕Մೳͱͳͬͨ ▸ ՝୊ • ग़ྗVoxel͕32x32x32ͱখ͍͞ 28