Patch-based potentials for interactive contour extraction

Transcript

1. Patch-based potentials for interactive contour extraction Thoraya Ben Chattah1,2, S´ ebastien Bougleux1, Olivier L´ ezoray1, Atef Hamouda2 1Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France 2University of Tunis El Manar, Faculty of Siences of Tunis olivier.lezoray@unicaen.fr https://lezoray.users.greyc.fr
2. 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4 Conclusion 2 / 30
3. 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4 Conclusion 3 / 30
4. Introduction Interactive segmentation : accurate object extraction with user-provided inputs FG/BG region seeds Points on the contour 4 / 30
5. Introduction Example of contour-based methods : Intelligent scissors (implemented in The GIMP) 5 / 30
6. Minimal Paths Minimal paths for contour-based object extraction Find shortest path between two given endpoints given a pixelwise cost function Relevant cost : decreasing function of gradient magnitude short paths tend to follow edges 6 / 30
7. Minimal Paths Given an image f : Ω ⊂ R2 → Rn + and two seeds ps, pe ∈ Ω located on the contour of an object Objective: find a path γ : [0, 1] → Ω so that γ(0) = ps and γ(1) = pe that represents a piece of contour : L(γ) = 1 0 P(γ(s)) γ (s) ds. (1) P is a potential function derived from image f that should have low values for edges. L(γ) is smaller when it goes through low values of P → minimize L(γ) given a pair of starting and ending points. Objective : find the geodesic path L(γ∗) = min γ∈A(ps ,pe ) L(γ) (2) 7 / 30
8. Minimal Paths First step: compute the geodesic distance map U{ps } : Ω → R+ from a point ps to any other point p ∈ Ω U{ps } (p) = d(ps, p) = min γ∈A(ps ,p) L(γ) (3) This can be done efficiently with the Fast Marching front-propagation algorithm (similar to Dijkstra’s shortest path algorithm on a graph) Second step : the minimal path between ps and pe can be constructed by applying a back-propagation procedure (gradient descent) starting from pe along the gradient of the map Ups until arriving at ps 8 / 30
9. Livewire interactive contour extraction how is interaction done with the user in point-based minimal path interactive contour extraction ? Initialization f : Ω → [0, 255]3 potential P distance map Up1 add p1 to S (our proposed one) 9 / 30
10. Introduction Livewire interactive contour extraction Initialization add p2 to S update potential compute minimal path P[γ] = +∞ 10 / 30
11. Introduction Livewire interactive contour extraction Initialization add p3 to S add p4 to S and so on until the contour is closed 11 / 30
12. Introduction Livewire interactive contour extraction Initialization potential other contour final potential 12 / 30
13. 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4 Conclusion 13 / 30
14. Patch-based potentials Classical potentials are pixel-based and local potentials This does not take into account the surrounding regions apart from the delineated contour A contour is a curve Γ separating optimally two regions R+ and R− Each region has usually different features from the other one Our proposal : to design new potentials based on patches as texture descriptors Patch at pi is P(pi ) = (f (pi + t), ∀t ∈ [−w, w]2)T ∈ R3(2w+1)2 14 / 30
15. Patch-based potentials Pixel-surround potentials : A pixel can be considered as lying on an edge if the patch at its left is different from the patch at its right. dc dc dl dr P1 (p) = max θ dθ,1 c (p) PAF 2 (p) = max θ AF(dθ,1 s (p)) AF ∈ {min, max, mean} dθ,1 s = {dθ,1 l , dθ,1 c , dθ,1 r } with patch differences: centered on p: dθ,δ c (p) = d P(pl,θ,δ), P(pr,θ,δ) upwind left: dθ,δ l (p) = d P(p), P(pl,θ,δ) upwind right: dθ,δ r (p) = d P(p), P(pr,θ,δ) with d(P(pi ), P(pj )) = P(pi ) − P(pj ) 2 and θ ∈ {0◦, 45◦, 90◦, 135◦} and δ the distance between the pixels along the line. 15 / 30
16. Patch-based potentials Band-based potentials : operating on larger neighborhood to discriminate edge from non-edge pixels. extend P2 to a larger neighborhood obtained with δ > 1 based on a band of ns patches on each side of central pixel p avoid overlap with patch P(p) by considering pixels at a distance δ ∈ [ , + ns] PAF 3 (p) = max θ AF min δ∈[ , +ns] dθ,δ s (p), ∀s ∈ {l, c, r} dc dl dr
    
    16 / 30
17. Patch-based potentials Seed-dependent patch-based potential previous potentials do not take into account the previously positioned seeds given a seed ps , consider regions R+ θs (ps ) and R− θs (ps ) with angle: θs = arg min θ ∠(Ns , ps pl,θ,1 s ) normal to γs−1,s at ps : Ns = (γ∗ s−1,s (ps ))⊥/ γ∗ s−1,s (ps ) R+(p s ) R- (p s ps γ ) ⇒ should be similar to the other regions for the next points of the contour 17 / 30
18. Patch-based potentials Seed-dependent patch-based potential given a seed ps , consider regions R+ θs (ps ) and R− θs (ps ) ∀p ∈ Ω, compute all pairwise distances between R+ θs (ps ) and R+ θ (p), ∀θ P4(pi ) = min θ ( min q1×q2 q1∈R+ θ (p) q2∈R+ θs (ps ) d(P(q1), P(q2)) + min q1×q2 q1∈R− θ (p) q2∈R− θs (ps ) d(P(q1), P(q2))) with 5x5 patches P(·) contained within π/4 to the band defined by θ to the horizon R+(ps ) R-(ps ) ps R+(p) R-(p) p ! 18 / 30
19. Patch-based potentials Seed-dependent patch-based potential for the 1st seed p1 , no idea of the orientation find the angle maximizing differences between patches in R+ θ (p1 ) and R− θ (p1 ) θ1 = arg max θ max q1×q2 q1∈R+ θ (p1) q2∈R− θ (p1) d(P(q1 ), P(q2 )) Combined patch-based potential P5 = αPmax 3 + (1 − α)P4 ⇒ note that this must be computed at each step of the extraction (since this depends on the last seed) 19 / 30
20. Patch-based potentials Illustration of the potentials Original image Pmax 2 Pmax 3 P4 P5 20 / 30
21. 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4 Conclusion 21 / 30
22. Experiments User simulation for contour extraction evaluation simulate the human user behavior with an agent find the longest possible path with minimum error from a seed ps the path must be contained in a tube of radius r on both sides of the ground truth realized by a dichotomous search along the ground truth object contour p s p e 2r ground truth finding longest path zoom 22 / 30
23. Experiments User simulation for contour extraction evaluation and so on until the contour is closed ground truth finding longest path zoom after several steps finding longest path end 23 / 30
24. Experiments Dataset ISEG [Gulshan & al. Geodesic star convexity for interactive image segmentation. CVPR 2010] What is the best patch size ? We considered w ∈ {1, 2, 3} corresponding to 3x3, 5x5 and 7x7 patches w = 1 w = 2 w = 3 Number of times a potential gives the minimum % of seeds ⇒ Best results are mostly always obtained with band-based potential Pmax 3 and w = 2 24 / 30
25. Experiments Quantitative results on ISEG Mean Mean Potential Radius % seeds Jaccard index PE 2 4.58 0.96 3 2.96 0.95 4 2.31 0.94 Pmax 3 2 3.93 0.97 3 2.67 0.95 4 2.12 0.94 P4 2 4.42 0.95 3 3.13 0.93 4 2.52 0.91 P5 2 3.67 0.97 3 2.38 0.95 4 1.86 0.94 difference of 0.1 percent point corresponds to 1.5 seed PE : classical gradient-based potential (decreasing function of the gradient) 25 / 30
26. Experiments Qualitative results on ISEG PE , #seeds=28 Pmax 3 , #seeds=34 P4, #seeds=26 P5, #seeds=24 PE , #seeds=38 Pmax 3 , #seeds=53 P4, #seeds=37 P5, #seeds=32 26 / 30
27. Experiments Qualitative results on ISEG PE , #seeds=48 Pmax 3 , #seeds=26 P4, #seeds=45 P5, #seeds=32 PE , #seeds=47 Pmax 3 , #seeds=37 P4, #seeds=46 P5, #seeds=28 27 / 30
28. 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4 Conclusion 28 / 30
29. Conclusion We have proposed: new patch-based potentials for contour extraction quantitative experiments were simulated with a user agent enables to improve classical gradient-based potential enables to reduce the user interaction (in terms of required points for object extraction) 29 / 30
30. The end Publications available at : https://lezoray.users.greyc.fr 30 / 30