Video Propagation Networks

A presentation I gave about Video Propagation Networks at the 2018 edition of the "Advanced Methods in Computer Grpahics" Seminar at ETH Zurich.

See the original paper here: https://ps.is.tue.mpg.de/publications/vpn-cvpr17

Transcript

1. Beamerkalibrierungsuntersuchungstestvortragsfolie Quadrat Auflösung: 1920x1080 Stern in den Ecken zu sehen? Sowas wie 16:9? Nein? Soweit wie man es von Beamern erwarten kann? Quadrat quadratisch? Graustufen korrekt?
3. „Talk is cheap - Show me the video“ - Linus Torvalds, Mastermind behind Linux
4. Demo
14. Video Propagation Networks Marcel Neidinger nmarcel@student.ethz.ch 23/03/2018
15. Introduction Related Work and Background Video Propagation Networks Experimental Evaluation
16. Introduction Related Work and Background Video Propagation Networks Experimental Evaluation 00
17. Introduction
18. So what is the problem? 01
19. And what can I do with it? 02
20. And what can I do with it? 02
21. And what can I do with it? 02
22. And what can I do with it? 02
23. And what can I do with it? 03
24. And what can I do with it? 03
25. Duck Sea Background And what can I do with it? 03
26. Duck Sea Background And what can I do with it? 03
27. And what can I do with it? 04
28. And what can I do with it? 04
29. And what can I do with it? 04
30. And what can I do with it? 04
31. Summary: Introduction Recap 05
32. Summary: Introduction Recap 05
33. Summary: Introduction Video Segmentation Recap 05
34. Summary: Introduction Video Segmentation Recap Semantic Labeling 05
35. Summary: Introduction Video Segmentation Recap Semantic Labeling Color Propagation 05
36. Related Work and Background
37. General Propagation: Graphs 06
38. General Propagation: Graphs 06
39. General Propagation: Graphs 06
40. General Propagation: Graphs These Graphs can be Huge 07
41. General Propagation: Filtering 08
42. General Propagation: Filtering 08
43. General Propagation: Filtering 08
44. General Propagation: Filtering 08
45. Background: Bilateral Filtering Edge Preserving 09
46. Background: Bilateral Filtering Edge Preserving 09
47. Background: Bilateral Filtering 10
48. Background: Bilateral Filtering 10
49. Background: Bilateral Filtering ˆ vi = j∈n Wi,jvj 11
50. Background: Bilateral Filtering ˆ vi = j∈n Wi,jvj 11
51. Background: Bilateral Filtering ˆ vi = j∈n Wi,jvj g 11
52. Background: Bilateral Filtering x y 12
53. Background: Bilateral Filtering x y T 12
54. Bilateral Filtering: Splatting 13
55. Bilateral Filtering: Convolving B 14
56. Bilateral Filtering: slicing 15
57. Summary: Related Work Recap x y T B 16
58. Video Propagation Networks
59. The Task: A „formal“ Definition Black Box (VPN) 17
60. Architecture BNN CNN Bilateral Network Spatial Network 18
61. Bilateral Network: Bilateral Convolution Layer (BCL) 19
62. Bilateral Network: Bilateral Convolution Layer (BCL) 19
63. Bilateral Network: Bilateral Convolution Layer (BCL) 19
64. Bilateral Network: Bilateral Convolution Layer (BCL) x y T t 19
65. Bilateral Network: Bilateral Convolution Layer (BCL) x y T t 19
66. Bilateral Network: Bilateral Convolution Layer (BCL) x y T t 19
67. Bilateral Network: Bilateral Convolution Layer (BCL) B x y T t 19
68. Bilateral Network: Bilateral Convolution Layer (BCL) B x y T t 19
69. Bilateral Network: Bilateral Convolution Layer (BCL) B x y T t 19
70. Bilateral Network: Architecture Guidance BCLa BCLb || BCLa BCLb || C-1 C-3 C-3 C-3 || C-1 20
71. Summary: Video Propagation Networks Recap BNN CNN Bilateral Network Spatial Network 21 x y T t
72. Experimental Evaluation
73. Experiment: Video Object Segmentation 22
74. Experiment: Video Object Segmentation 22
75. Experiment: Video Object Segmentation 22
76. Video Object Segmentation: Dataset Densly Annotated VIdeo Segmentation 23
77. Video Object Segmentation: Evaluation 0 20 40 60 80 SEA JMP BNN-Identity VPN Stage 2 VPN-Deeplab IoU F T 24
78. Video Object Segmentation: Improvements 25
79. Video Object Segmentation: Improvements 68 67,5 67 66,5 66 0 500 1000 IoU Score Number of Points (in Thousand) 26
80. Video Object Segmentation: Improvements 12.000 SLIC Superpixel 27
81. Summary: Experimental Evaluation Recap -to- 28
82. Conclusion Three things to take away (i) VPNs use a bilateral and a spatial network (ii) Offer favorable runtime over current methods (iii) Perform better then current methods 1 1 And yes, this is the only textual slide you will see in this presentation
83. „The important thing is to never stop questioning“ - Albert Einstein, Mastermind of the Universe
84. Appendix • Definition of Performance Measures • Detailed Performance (Video Segmentation) • Detailed Performance (Semantic Labeling) • Detailed Performance (Color Propagation) • Literature Back to Overview • JumpCut (JMP) • SeamSeg (SEA)
85. Performance Measures Intersection over Union (IoU) Groundtruth G, Output segmentation M Contour Accuracy F Temporal Stability T Back to Overview
86. Detailed Performance (Video Segmentation) Back to Overview
87. Detailed Performance (Semantic Labeling) Back to Overview
88. Detailed Performance (Color Propagation) Back to Overview
89. Literature (besides the main paper) [1] S. Paris and F. Durand. A fast approximation of the bilateralfilter using a signal processing approach. In European Conference on Computer Vision, pages 568–580. Springer, 2006. [2] V. Jampani, M. Kiefel, and P. V. Gehler. Learning sparse high dimensional filters: Image filtering, dense CRFs and bilateral neural networks. In Computer Vision and Pattern Recognition, IEEE Conference on, June 2016. Back to Overview
90. JumpCut (JMP) Back to Overview • Key idea: Foreground and background exhibit different motions • Calculate two Nearest-Neighbor Fields to label target frame • Calculate Silhouette Edges (using the NNFs)
91. SeamSeg (SEA) Back to Overview • Key idea: Seams(Connected Paths of low energy) Energy formulation allows for temporal label propagation