A Modular Framework for Image-based 3D Reconstruction

Talk about my Master's Thesis.

In this thesis I analyse the state of the art of Structure from Motion and
3D reconstruction algorithms to develop a framework, that improves the process of
working with these algorithms.
This framework allows reusing existing parts of a processing chain,
as well as developing new algorithms efficiently.
This is done by providing a data model and interfaces,
which allow inspection of intermediate results,
as well as reorganisation of the processing flow.

I first cover the basic principles of 3D reconstruction, and then review existing
implementations. Based on that the framework is developed to
allow the implementation of modular and flexible 3D reconstruction
processing chains.

I also show how this framework can be applied to existing problems
to quickly come to a working implementation, as it allows the user to focus
on the algorithm implementation details instead of the need to deal with
the overall processing chain design.

The implementation of the framework is released as Open Source software: https://github.com/uipf/

Transcript

1. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM A Modular Framework for Image-based 3D Reconstruction Carsten Brandt May 18, 2017 Master’s thesis Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 1 / 24
2. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM 3D Reconstruction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 2 / 24
3. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM 3D Reconstruction Structure from Motion (SfM) Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 2 / 24
4. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM 3D Reconstruction Structure from Motion (SfM) Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 2 / 24
5. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM 3D Reconstruction Structure from Motion (SfM) Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 2 / 24
6. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM 3D Reconstruction Structure from Motion (SfM) Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 2 / 24
7. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Table of Contents 1 Problem Definition 2 The Process of 3D Reconstruction 3 Framework Design and Implementation 4 Using the Framework for SfM Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 3 / 24
8. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Table of Contents 1 Problem Definition 2 The Process of 3D Reconstruction 3 Framework Design and Implementation 4 Using the Framework for SfM Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 4 / 24
9. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
10. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
11. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Keypoint Detection Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
12. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Keypoint Detection Image Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
13. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Keypoint Detection Image Matching Geometry Estimation Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
14. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Keypoint Detection Image Matching Geometry Estimation Triangulation Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
15. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition ? Keypoint Detection Image Matching Geometry Estimation Triangulation . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
16. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Problem Definition Keypoint Detection Image Matching Geometry Estimation Triangulation . . . Processing Chain Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 5 / 24
17. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
18. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
19. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
20. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
21. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
22. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Framework goals: Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
23. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Framework goals: flexible processing chain creation Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
24. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Framework goals: flexible processing chain creation applicable to large reconstruction problems Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
25. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Framework goals: flexible processing chain creation applicable to large reconstruction problems visualisation options Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
26. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Existing implementations Existing implementations: single program for everything not flexible for different use case hard to adjust/improve Framework goals: flexible processing chain creation applicable to large reconstruction problems visualisation options Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 6 / 24
27. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Table of Contents 1 Problem Definition 2 The Process of 3D Reconstruction 3 Framework Design and Implementation 4 Using the Framework for SfM Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 7 / 24
28. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Images Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
29. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Images Images +Metadata Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
30. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Images Images +Metadata Images +Metadata +Keypoints Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
31. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Images Images +Metadata Images +Metadata +Keypoints Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
32. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Images Images +Metadata Images +Metadata +Keypoints Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
33. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
34. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Keypoint Matching Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
35. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Keypoint Matching Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Graph Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
36. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
37. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Feature Extraction Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
38. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
39. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
40. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
41. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
42. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F 3D Geometry Estimation Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
43. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F 3D Geometry Estimation Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
44. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F 3D Geometry Estimation Images +P Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
45. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F 3D Geometry Estimation 3D Point Cloud Images +P Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
46. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Preprocessing Keypoint Detection Keypoint Description Image Clustering Keypoint Matching Match Filtering Images Images +Metadata Images +Metadata +Keypoints Images +Metadata +Keypoints Image Cluster Image Graph Image Pair +F 3D Geometry Estimation 3D Point Cloud Images +P Geometry Refinement Feature Extraction Feature Matching Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 8 / 24
47. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
48. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
49. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
50. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Sparse Point Cloud Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
51. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Sparse Point Cloud MVS Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
52. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Sparse Point Cloud MVS Dense Point Cloud Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
53. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Sparse Point Cloud MVS Dense Point Cloud Surface Reconstruction Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
54. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Structure from Motion Image-based 3D Reconstruction Highlevel overview of image-based 3D reconstruction Images SfM Images with P Image Graph Sparse Point Cloud MVS Dense Point Cloud Surface Reconstruction Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 9 / 24
55. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Table of Contents 1 Problem Definition 2 The Process of 3D Reconstruction 3 Framework Design and Implementation 4 Using the Framework for SfM Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 10 / 24
56. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
57. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm Data Image,String,Int,... Data type Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
58. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs Data Image,String,Int,... Data type Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
59. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs Data Image,String,Int,... Data type Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
60. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs parameters Data Image,String,Int,... Data type Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
61. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs parameters Data Image,String,Int,... Data type clear interface → reuse of code in different contexts Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
62. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs parameters Data Image,String,Int,... Data type clear interface → reuse of code in different contexts self-contained unit → parallelizable Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
63. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs parameters Data Image,String,Int,... Data type clear interface → reuse of code in different contexts self-contained unit → parallelizable common data type implementations → data flow control Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
64. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Module and Data Module Module Implementation of an Algorithm inputs outputs parameters Data Image,String,Int,... Data type clear interface → reuse of code in different contexts self-contained unit → parallelizable common data type implementations → data flow control processing chain is a dependency graph Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 11 / 24
65. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Processing Chain load module: load image params: filename: image.png resize module: resize image params: width: 1200px height: 900px store module: store image params: filename: resized.png Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 12 / 24
66. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Processing Chain load module: load image params: filename: image.png resize module: resize image params: width: 1200px height: 900px store module: store image params: filename: resized.png other1 module: ... params: ... Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 12 / 24
67. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Processing Chain load module: load image params: filename: image.png resize module: resize image params: width: 1200px height: 900px store module: store image params: filename: resized.png other1 module: ... params: ... other2 module: ... params: ... Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 12 / 24
68. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 1 #include <u i p f / data . hpp> 2 #include <u i p f / data / opencv . hpp> 3 4 using namespace u i p f ; 5 6 #define UIPF MODULE ID ” opencv . imgproc . r e s i z e ” 7 #define UIPF MODULE NAME ” Resize Image” 8 #define UIPF MODULE CATEGORY ” opencv ” 9 #define UIPF MODULE CLASS OpenCVResizeImage // . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
69. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 1 #include <u i p f / data . hpp> 2 #include <u i p f / data / opencv . hpp> 3 4 using namespace u i p f ; 5 6 #define UIPF MODULE ID ” opencv . imgproc . r e s i z e ” 7 #define UIPF MODULE NAME ” Resize Image” 8 #define UIPF MODULE CATEGORY ” opencv ” 9 #define UIPF MODULE CLASS OpenCVResizeImage // . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
70. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 1 #include <u i p f / data . hpp> 2 #include <u i p f / data / opencv . hpp> 3 4 using namespace u i p f ; 5 6 #define UIPF MODULE ID ” opencv . imgproc . r e s i z e ” 7 #define UIPF MODULE NAME ” Resize Image” 8 #define UIPF MODULE CATEGORY ” opencv ” 9 #define UIPF MODULE CLASS OpenCVResizeImage // . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
71. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 11 #define UIPF MODULE INPUTS \ 12 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 13 ” the input image . ” )} 14 15 #define UIPF MODULE OUTPUTS \ 16 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 17 ” the r e s i z e d image . ” )} 18 19 #define UIPF MODULE PARAMS \ 20 {” width ” , ParamDescription ( ”new width . ” ) } , \ 21 {” he ig h t ” , ParamDescription ( ”new he i g h t . ” ) } 22 23 #include <u i p f /Module . hpp> 24 25 void OpenCVResizeImage : : run () { 26 // Module Implementation goes here . 27 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
72. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 11 #define UIPF MODULE INPUTS \ 12 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 13 ” the input image . ” )} 14 15 #define UIPF MODULE OUTPUTS \ 16 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 17 ” the r e s i z e d image . ” )} 18 19 #define UIPF MODULE PARAMS \ 20 {” width ” , ParamDescription ( ”new width . ” ) } , \ 21 {” he ig h t ” , ParamDescription ( ”new he i g h t . ” ) } 22 23 #include <u i p f /Module . hpp> 24 25 void OpenCVResizeImage : : run () { 26 // Module Implementation goes here . 27 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
73. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 11 #define UIPF MODULE INPUTS \ 12 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 13 ” the input image . ” )} 14 15 #define UIPF MODULE OUTPUTS \ 16 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 17 ” the r e s i z e d image . ” )} 18 19 #define UIPF MODULE PARAMS \ 20 {” width ” , ParamDescription ( ”new width . ” ) } , \ 21 {” he ig h t ” , ParamDescription ( ”new he i g h t . ” ) } 22 23 #include <u i p f /Module . hpp> 24 25 void OpenCVResizeImage : : run () { 26 // Module Implementation goes here . 27 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
74. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 11 #define UIPF MODULE INPUTS \ 12 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 13 ” the input image . ” )} 14 15 #define UIPF MODULE OUTPUTS \ 16 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 17 ” the r e s i z e d image . ” )} 18 19 #define UIPF MODULE PARAMS \ 20 {” width ” , ParamDescription ( ”new width . ” ) } , \ 21 {” he ig h t ” , ParamDescription ( ”new he i g h t . ” ) } 22 23 #include <u i p f /Module . hpp> 24 25 void OpenCVResizeImage : : run () { 26 // Module Implementation goes here . 27 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
75. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module 11 #define UIPF MODULE INPUTS \ 12 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 13 ” the input image . ” )} 14 15 #define UIPF MODULE OUTPUTS \ 16 {”image” , DataDescription ( data : : OpenCVMat : : i d ( ) , \ 17 ” the r e s i z e d image . ” )} 18 19 #define UIPF MODULE PARAMS \ 20 {” width ” , ParamDescription ( ”new width . ” ) } , \ 21 {” he ig h t ” , ParamDescription ( ”new he i g h t . ” ) } 22 23 #include <u i p f /Module . hpp> 24 25 void OpenCVResizeImage : : run () { 26 // Module Implementation goes here . 27 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 13 / 24
76. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
77. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
78. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
79. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
80. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
81. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Implementation of a Module: Data Interface 1 void OpenCVResizeImage : : run () { 2 3 OpenCVMat : : ptr img = getInputData <OpenCVMat>(”image ” ) ; 4 i n t width = getParam<int >(” width ” , −1); 5 i n t h ei gh t = getParam<int >(” he i g h t ” , −1); 6 7 // do something here 8 9 OpenCVMat : : ptr newImage (new OpenCVMat(m) ) ; 10 newImage−>f i l e n a m e = image−>f i l e n a m e ; 11 newImage−>e x i f = image−>e x i f ; 12 setOutputData<OpenCVMat>(”image ” , newImage ) ; 13 14 } Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 14 / 24
82. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Data structure implementation Image,String,Int,... Data type Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 15 / 24
83. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Data structure implementation Image,String,Int,... Data type Data structure is a container Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 15 / 24
84. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Data structure implementation Image,String,Int,... Data type Data structure is a container Interface for visualisation and serialisation Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 15 / 24
85. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Data structure implementation Image,String,Int,... Data type Data structure is a container Interface for visualisation and serialisation Custom datatypes possible Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 15 / 24
86. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Framework overview Framework Library Runner Data Manager Logger Module Loader Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 16 / 24
87. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Framework overview Framework Library Runner Data Manager Logger Module Loader Module 1 Module n . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 16 / 24
88. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Framework overview Framework Library Processing Chain YAML File Runner Data Manager Logger Module Loader Module 1 Module n . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 16 / 24
89. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview Framework overview GUI CLI Framework Library Processing Chain YAML File Runner Data Manager Logger Module Loader Module 1 Module n . . . Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 16 / 24
90. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview GU Interface Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 17 / 24
91. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview CL Interface # run a p r o c e s s i n g chain u i p f −c processing −chain . yaml Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 18 / 24
92. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview CL Interface # run a p r o c e s s i n g chain u i p f −c processing −chain . yaml # run a s i n g l e module u i p f uipfsfm . keypoint −i image . jpg −o p o i n t s . t x t Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 18 / 24
93. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Module Data structure Framework overview CL Interface # run a p r o c e s s i n g chain u i p f −c processing −chain . yaml # run a s i n g l e module u i p f uipfsfm . keypoint −i image . jpg −o p o i n t s . t x t # l i s t and document modules u i p f −l u i p f −−i n f o uipfsfm . keypoint . s i f t Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 18 / 24
94. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Table of Contents 1 Problem Definition 2 The Process of 3D Reconstruction 3 Framework Design and Implementation 4 Using the Framework for SfM Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 19 / 24
95. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Dataset 79 images 2592 × 1728px “Der Hass” dataset by [Fuhrmann 2014] Simon Fuhrmann, Fabian Langguth and Michael Goesele. MVE - A Multi-View Reconstruction Environment. In GCH, pages 1118, 2014. Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 20 / 24
96. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Processing Chain Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 21 / 24
97. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Processing Chain Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 21 / 24
98. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Processing Chain Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 21 / 24
99. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Sparse Point Cloud Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 22 / 24
100. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Mesh Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 23 / 24
101. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Conclusion Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 24 / 24
102. Problem Definition The Process of 3D Reconstruction Framework Design and Implementation Using the Framework for SfM Dataset Processing Chain Results Conclusion Conclusion https://github.com/uipf/uipf Carsten Brandt A Modular Framework for Image-based 3D Reconstruction May 18, 2017 24 / 24