Video Propagation Networks

A presentation I gave about Video Propagation Networks at the 2018 edition of the "Advanced Methods in Computer Grpahics" Seminar at ETH Zurich.

See the original paper here: https://ps.is.tue.mpg.de/publications/vpn-cvpr17

Transcript

1. Beamerkalibrierungsuntersuchungstestvortragsfolie
   Quadrat
   Auflösung: 1920x1080
   Stern in den Ecken zu sehen?
   Sowas wie 16:9? Nein? Soweit wie man es von Beamern erwarten kann?
   Quadrat quadratisch?
   Graustufen korrekt?
   

   View Slide

2. View Slide

3. „Talk is cheap - Show me the video“
   - Linus Torvalds,
   Mastermind behind Linux
   

   View Slide

4. Demo
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. Video Propagation Networks
    Marcel Neidinger
    nmarcel@student.ethz.ch
    23/03/2018
    

    View Slide

15. Introduction
    Related Work and Background
    Video Propagation Networks
    Experimental Evaluation
    

    View Slide

16. Introduction
    Related Work and Background
    Video Propagation Networks
    Experimental Evaluation
    00
    

    View Slide

17. Introduction
    

    View Slide

18. So what is the problem?
    01
    

    View Slide

19. And what can I do with it?
    02
    

    View Slide

20. And what can I do with it?
    02
    

    View Slide

21. And what can I do with it?
    02
    

    View Slide

22. And what can I do with it?
    02
    

    View Slide

23. And what can I do with it?
    03
    

    View Slide

24. And what can I do with it?
    03
    

    View Slide

25. Duck
    Sea
    Background
    And what can I do with it?
    03
    

    View Slide

26. Duck
    Sea
    Background
    And what can I do with it?
    03
    

    View Slide

27. And what can I do with it?
    04
    

    View Slide

28. And what can I do with it?
    04
    

    View Slide

29. And what can I do with it?
    04
    

    View Slide

30. And what can I do with it?
    04
    

    View Slide

31. Summary: Introduction Recap
    05
    

    View Slide

32. Summary: Introduction Recap
    05
    

    View Slide

33. Summary: Introduction
    Video Segmentation
    Recap
    05
    

    View Slide

34. Summary: Introduction
    Video Segmentation
    Recap
    Semantic Labeling
    05
    

    View Slide

35. Summary: Introduction
    Video Segmentation
    Recap
    Semantic Labeling
    Color Propagation
    05
    

    View Slide

36. Related Work and Background
    

    View Slide

37. General Propagation: Graphs
    06
    

    View Slide

38. General Propagation: Graphs
    06
    

    View Slide

39. General Propagation: Graphs
    06
    

    View Slide

40. General Propagation: Graphs
    These Graphs can be
    Huge
    07
    

    View Slide

41. General Propagation: Filtering
    08
    

    View Slide

42. General Propagation: Filtering
    08
    

    View Slide

43. General Propagation: Filtering
    08
    

    View Slide

44. General Propagation: Filtering
    08
    

    View Slide

45. Background: Bilateral Filtering
    Edge Preserving
    09
    

    View Slide

46. Background: Bilateral Filtering
    Edge Preserving
    09
    

    View Slide

47. Background: Bilateral Filtering
    10
    

    View Slide

48. Background: Bilateral Filtering
    10
    

    View Slide

49. Background: Bilateral Filtering
    ˆ
    vi =
    j∈n
    Wi,jvj
    11
    

    View Slide

50. Background: Bilateral Filtering
    ˆ
    vi =
    j∈n
    Wi,jvj
    11
    

    View Slide

51. Background: Bilateral Filtering
    ˆ
    vi =
    j∈n
    Wi,jvj
    g
    11
    

    View Slide

52. Background: Bilateral Filtering
    x y
    12
    

    View Slide

53. Background: Bilateral Filtering
    x y
    T
    12
    

    View Slide

54. Bilateral Filtering: Splatting
    13
    

    View Slide

55. Bilateral Filtering: Convolving
    B
    14
    

    View Slide

56. Bilateral Filtering: slicing
    15
    

    View Slide

57. Summary: Related Work Recap
    x y
    T
    B
    16
    

    View Slide

58. Video Propagation Networks
    

    View Slide

59. The Task: A „formal“ Definition
    Black Box
    (VPN)
    17
    

    View Slide

60. Architecture
    BNN CNN
    Bilateral Network Spatial Network
    18
    

    View Slide

61. Bilateral Network: Bilateral Convolution Layer (BCL)
    19
    

    View Slide

62. Bilateral Network: Bilateral Convolution Layer (BCL)
    19
    

    View Slide

63. Bilateral Network: Bilateral Convolution Layer (BCL)
    19
    

    View Slide

64. Bilateral Network: Bilateral Convolution Layer (BCL)
    x y
    T
    t
    19
    

    View Slide

65. Bilateral Network: Bilateral Convolution Layer (BCL)
    x y
    T
    t
    19
    

    View Slide

66. Bilateral Network: Bilateral Convolution Layer (BCL)
    x y
    T
    t
    19
    

    View Slide

67. Bilateral Network: Bilateral Convolution Layer (BCL)
    B
    x y
    T
    t
    19
    

    View Slide

68. Bilateral Network: Bilateral Convolution Layer (BCL)
    B
    x y
    T
    t
    19
    

    View Slide

69. Bilateral Network: Bilateral Convolution Layer (BCL)
    B
    x y
    T
    t
    19
    

    View Slide

70. Bilateral Network: Architecture
    Guidance
    BCLa
    BCLb
    ||
    BCLa
    BCLb
    || C-1 C-3 C-3 C-3 || C-1
    20
    

    View Slide

71. Summary: Video Propagation Networks
    Recap
    BNN CNN
    Bilateral Network Spatial Network
    21
    x y
    T
    t
    

    View Slide

72. Experimental Evaluation
    

    View Slide

73. Experiment: Video Object Segmentation
    22
    

    View Slide

74. Experiment: Video Object Segmentation
    22
    

    View Slide

75. Experiment: Video Object Segmentation
    22
    

    View Slide

76. Video Object Segmentation: Dataset
    Densly Annotated VIdeo Segmentation
    23
    

    View Slide

77. Video Object Segmentation: Evaluation
    0
    20
    40
    60
    80
    SEA JMP BNN-Identity VPN Stage 2 VPN-Deeplab
    IoU F T
    24
    

    View Slide

78. Video Object Segmentation: Improvements
    25
    

    View Slide

79. Video Object Segmentation: Improvements
    68
    67,5
    67
    66,5
    66
    0 500 1000
    IoU Score
    Number of Points
    
    (in Thousand)
    26
    

    View Slide

80. Video Object Segmentation: Improvements
    12.000
    
    SLIC Superpixel
    27
    

    View Slide

81. Summary: Experimental Evaluation
    Recap
    -to-
    28
    

    View Slide

82. Conclusion
    Three things to take away
    (i) VPNs use a bilateral and a spatial network
    
    (ii) Offer favorable runtime over current
    methods
    
    (iii) Perform better then current methods
    1
    1 And yes, this is the only textual slide you will see in this presentation
    

    View Slide

83. „The important thing is to never stop
    questioning“
    - Albert Einstein,
    Mastermind of the Universe
    

    View Slide

84. Appendix
    • Definition of Performance Measures
    • Detailed Performance (Video Segmentation)
    • Detailed Performance (Semantic Labeling)
    • Detailed Performance (Color Propagation)
    • Literature
    Back to Overview
    • JumpCut (JMP)
    • SeamSeg (SEA)
    

    View Slide

85. Performance Measures
    Intersection over Union (IoU)
    Groundtruth G, Output segmentation M
    Contour Accuracy F
    Temporal Stability T
    Back to Overview
    

    View Slide

86. Detailed Performance (Video Segmentation)
    Back to Overview
    

    View Slide

87. Detailed Performance (Semantic Labeling)
    Back to Overview
    

    View Slide

88. Detailed Performance (Color Propagation)
    Back to Overview
    

    View Slide

89. Literature (besides the main paper)
    [1] S. Paris and F. Durand. A fast approximation of the bilateralfilter using a signal
    processing approach.
    
    In European Conference on Computer Vision, pages 568–580. Springer, 2006.
    [2] V. Jampani, M. Kiefel, and P. V. Gehler. Learning sparse high dimensional filters:
    Image filtering, dense CRFs and bilateral neural networks.
    
    In Computer Vision and Pattern Recognition, IEEE Conference on, June 2016.
    Back to Overview
    

    View Slide

90. JumpCut (JMP)
    Back to Overview
    • Key idea: Foreground and background exhibit different motions
    • Calculate two Nearest-Neighbor Fields to label target frame
    • Calculate Silhouette Edges (using the NNFs)
    

    View Slide

91. SeamSeg (SEA)
    Back to Overview
    • Key idea: Seams(Connected Paths of low energy) Energy formulation
    allows for temporal label propagation
    

    View Slide