Max-Planck-Institut für Informatik Invited Talk

Transcript

1. Dynamic Graph-Cut's for Efficient Mobile
   Image and Video Segmentation
   Ozan Sener
   Middle East Technical University
   

   View Slide

2. Interaction Segmentation Mask Application
   Interactive Image and Video Segmentation
   Segmentation Mask
   for Rest of the Video
   Application
   

   View Slide

3. [Bounding Box + Color GMM + Min-Cut/Max-Flow] = [Rother, SIGGRAPH 05]
   [Approx. Bound. + Color and Spatial Distance +Dynamic Alg.] = [Li Y, SIGGRAPH 95]
   [Scribble + Color Histogram + Min-Cut/Max-Flow] = [Boykov Y, IJCV 06]
   Color GMM
   Path (Boundary) Cost
   Dynamic Algorithm
   Min-Cut / Max-Flow
   Interaction Model Defintion Minimization
   

   View Slide

4. Mobile Touch Screen Devices ?
   User Centered Interaction
   More Interaction Errors
   Low Computational Power
   

   View Slide

5. Coloring
   Gesture of Coloring a Color Book
   

   View Slide

6. Some Details of Method
   Color GMMs are used as models
   Iterative EM is used [Rother, SIGGRAPH 05]
   Image is initially over-segmented for efficiency via
   SLIC algorithm [Achanta, PAMI 2012]
   Min-cut/Max-flow is used for energy minimization
   [Boykov, PAMI 2004]
   

   View Slide

7. Min-Cut / Max-Flow Review
   Graph Structure Augmentin Path Algorithm
   

   View Slide

8. Dynamic ?
   If the graph structure is not changing, previous flows can be
   reused in the minimization [Kohli, PAMI 2007]
   Throughout the interaction graph structure does not change
   at all, but min-cut/max-flow is solved many times.
   Only problem which can arise is the edge weights and it
   can be solved via additional flow.
   

   View Slide

9. Temporally + Spatially Dynamic
   Can we extend this concept to spatial dimension ?
   At any stage only part of the whole graph containing
   foreground object is need to be solved. But, what is the size
   of this sub-graph ?
   If the external flow which can flow through edges of the
   subgraph can not change the solution, there is no need to
   enlarge it anymore ?. However, this is hard to achieve.
   Clustering supplied by GMM is generally confident;
   however, labeling can be wrong.
   

   View Slide

10. Temporally + Spatially Dynamic (cont'd)
    Our Claim: if the labels of the GMM clusters can not be
    changed via external flows, there is no need to enlarge the
    subgraph.
    Algorithm: Start with the bounding box of the interaction
    and enlarges it until
    is satisfied for all GMM clusers.
    

    View Slide

11. (a): Blue rectangle is bounding box of the current interaction
    Red rectangle is the computed bounding box
    (b): Result of Min-Cut/Max-Flow for blue rectangle in (a)
    (c): Result of Min-Cut/Max-Flow for red rectangle in (a)
    Dynamic Graph-Cut in Action
    

    View Slide

12. Computation Time
    

    View Slide

13. Segmentation Quality
    

    View Slide

14. Redefinition of Video Segmentation
    Assume MRF energy for the initial frame is
    known.
    MRF energy of any other frame is linearly
    dependent on previous frame. (All superpixels
    are model assumption)
    

    View Slide

15. Biexponential Filters
    Spatio-temporal distance metric should be used
    for robust video segmentation.
    Geodesic distance is a best candidate with high
    computational complexity -O(n^3)-.
    Bi-exponential smoother is used for high
    performance approximation -O(n)- [Unser M,TIP 2011]
    

    View Slide

16. Sample Propagation
    

    View Slide

17. Dynamic Graph-Cut for Linear Filtering
    

    View Slide

18. Segmentation Quality
    

    View Slide

19. Computation Time Improvement via
    Bilateral Graph-Cut
    

    View Slide

20. Time vs Performance
    On Segtrack [Tsai BMVC 2010] dataset:
    

    View Slide

21. Thank you for your attention.
    

    View Slide

22. BACKUP SLIDES
    

    View Slide

23. View Slide

24. Overall Picture
    

    View Slide

25. Error-Tolerance
    Energy minimization can tolerate some level of
    error if hard labels are replaced with soft labels.
    Question: Hard Labels vs Error Tolerance
    Solution: Solve errors before they occur.
    

    View Slide

26. Error-Tolerance Algorithm
    New
    Superpixel
    Idea is keeping a single RGB gaussian model for the color model of the currently interacted
    region. If new superpixel is not confirming the color model, wait for it to come back or accept the
    new region
    E
    ?
    GMM
    P ?
    Back To
    Previous
    GMM
    w/o Error
    GMM
    C ?
    Insert to
    Current
    GMM
    Discard P
    Create GMM
    w/ Error
    T
    F
    T
    T
    F
    F
    GMM
    C ?
    Discard P
    Create GMM
    w/o Error
    Find Path
    T
    F
    Find path means minimize
    

    View Slide

27. Notes:
    False Positives are handled via path finding.
    False Negatives requires a restart.
    Single Color
    True Positive
    Multi Color
    False Positive
    Multi Color
    True Positive
    Error Tolerance in Action
    

    View Slide

28. Error-Tolerance
    

    View Slide

29. Interaction Quality
    Performance Easiness Entertainment Overall
    Proposed Method 5:1:.45 4:0:.86 5:1:.74 4:1:.45
    GrabCut 3:2:.92 4:1:.75 2:1:.61 3:1:.75
    Intelligent Scissors 3:1:.51 2:1:.74 3:2:.89 2:1:.76
    15 Subjects (Undergraduate Level Engineering Students)
    4 Random images out of 10 images
    Grading in the level of 1-5 for 4 different metrics
    Results in the format of Median:IQR:STD
    P-Values (via dependent ANOVA test): 0.0005
    

    View Slide