Olivier Lézoray
November 21, 2018

# Patch-based potentials for interactive contour extraction

#### Olivier Lézoray

November 21, 2018

## Transcript

1. ### Patch-based potentials for interactive contour extraction Thoraya Ben Chattah1,2, S´

ebastien Bougleux1, Olivier L´ ezoray1, Atef Hamouda2 1Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC, 14000 Caen, France 2University of Tunis El Manar, Faculty of Siences of Tunis olivier.lezoray@unicaen.fr https://lezoray.users.greyc.fr
2. ### 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4

Conclusion 2 / 30
3. ### 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4

Conclusion 3 / 30
4. ### Introduction Interactive segmentation : accurate object extraction with user-provided inputs

FG/BG region seeds Points on the contour 4 / 30
5. ### Introduction Example of contour-based methods : Intelligent scissors (implemented in

The GIMP) 5 / 30
6. ### Minimal Paths Minimal paths for contour-based object extraction Find shortest

path between two given endpoints given a pixelwise cost function Relevant cost : decreasing function of gradient magnitude short paths tend to follow edges 6 / 30
7. ### Minimal Paths Given an image f : Ω ⊂ R2

→ Rn + and two seeds ps, pe ∈ Ω located on the contour of an object Objective: ﬁnd a path γ : [0, 1] → Ω so that γ(0) = ps and γ(1) = pe that represents a piece of contour : L(γ) = 1 0 P(γ(s)) γ (s) ds. (1) P is a potential function derived from image f that should have low values for edges. L(γ) is smaller when it goes through low values of P → minimize L(γ) given a pair of starting and ending points. Objective : ﬁnd the geodesic path L(γ∗) = min γ∈A(ps ,pe ) L(γ) (2) 7 / 30
8. ### Minimal Paths First step: compute the geodesic distance map U{ps

} : Ω → R+ from a point ps to any other point p ∈ Ω U{ps } (p) = d(ps, p) = min γ∈A(ps ,p) L(γ) (3) This can be done eﬃciently with the Fast Marching front-propagation algorithm (similar to Dijkstra’s shortest path algorithm on a graph) Second step : the minimal path between ps and pe can be constructed by applying a back-propagation procedure (gradient descent) starting from pe along the gradient of the map Ups until arriving at ps 8 / 30
9. ### Livewire interactive contour extraction how is interaction done with the

user in point-based minimal path interactive contour extraction ? Initialization f : Ω → [0, 255]3 potential P distance map Up1 add p1 to S (our proposed one) 9 / 30
10. ### Introduction Livewire interactive contour extraction Initialization add p2 to S

update potential compute minimal path P[γ] = +∞ 10 / 30
11. ### Introduction Livewire interactive contour extraction Initialization add p3 to S

add p4 to S and so on until the contour is closed 11 / 30
12. ### Introduction Livewire interactive contour extraction Initialization potential other contour ﬁnal

potential 12 / 30
13. ### 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4

Conclusion 13 / 30
14. ### Patch-based potentials Classical potentials are pixel-based and local potentials This

does not take into account the surrounding regions apart from the delineated contour A contour is a curve Γ separating optimally two regions R+ and R− Each region has usually diﬀerent features from the other one Our proposal : to design new potentials based on patches as texture descriptors Patch at pi is P(pi ) = (f (pi + t), ∀t ∈ [−w, w]2)T ∈ R3(2w+1)2 14 / 30
15. ### Patch-based potentials Pixel-surround potentials : A pixel can be considered

as lying on an edge if the patch at its left is diﬀerent from the patch at its right. dc dc dl dr P1 (p) = max θ dθ,1 c (p) PAF 2 (p) = max θ AF(dθ,1 s (p)) AF ∈ {min, max, mean} dθ,1 s = {dθ,1 l , dθ,1 c , dθ,1 r } with patch diﬀerences: centered on p: dθ,δ c (p) = d P(pl,θ,δ), P(pr,θ,δ) upwind left: dθ,δ l (p) = d P(p), P(pl,θ,δ) upwind right: dθ,δ r (p) = d P(p), P(pr,θ,δ) with d(P(pi ), P(pj )) = P(pi ) − P(pj ) 2 and θ ∈ {0◦, 45◦, 90◦, 135◦} and δ the distance between the pixels along the line. 15 / 30
16. ### Patch-based potentials Band-based potentials : operating on larger neighborhood to

discriminate edge from non-edge pixels. extend P2 to a larger neighborhood obtained with δ > 1 based on a band of ns patches on each side of central pixel p avoid overlap with patch P(p) by considering pixels at a distance δ ∈ [ , + ns] PAF 3 (p) = max θ AF min δ∈[ , +ns] dθ,δ s (p), ∀s ∈ {l, c, r} dc dl dr   16 / 30
17. ### Patch-based potentials Seed-dependent patch-based potential previous potentials do not take

into account the previously positioned seeds given a seed ps , consider regions R+ θs (ps ) and R− θs (ps ) with angle: θs = arg min θ ∠(Ns , ps pl,θ,1 s ) normal to γs−1,s at ps : Ns = (γ∗ s−1,s (ps ))⊥/ γ∗ s−1,s (ps ) R+(p s ) R- (p s ps γ ) ⇒ should be similar to the other regions for the next points of the contour 17 / 30
18. ### Patch-based potentials Seed-dependent patch-based potential given a seed ps ,

consider regions R+ θs (ps ) and R− θs (ps ) ∀p ∈ Ω, compute all pairwise distances between R+ θs (ps ) and R+ θ (p), ∀θ P4(pi ) = min θ ( min q1×q2 q1∈R+ θ (p) q2∈R+ θs (ps ) d(P(q1), P(q2)) + min q1×q2 q1∈R− θ (p) q2∈R− θs (ps ) d(P(q1), P(q2))) with 5x5 patches P(·) contained within π/4 to the band deﬁned by θ to the horizon R+(ps ) R-(ps ) ps R+(p) R-(p) p ! 18 / 30
19. ### Patch-based potentials Seed-dependent patch-based potential for the 1st seed p1

, no idea of the orientation ﬁnd the angle maximizing diﬀerences between patches in R+ θ (p1 ) and R− θ (p1 ) θ1 = arg max θ max q1×q2 q1∈R+ θ (p1) q2∈R− θ (p1) d(P(q1 ), P(q2 )) Combined patch-based potential P5 = αPmax 3 + (1 − α)P4 ⇒ note that this must be computed at each step of the extraction (since this depends on the last seed) 19 / 30
20. ### Patch-based potentials Illustration of the potentials Original image Pmax 2

Pmax 3 P4 P5 20 / 30
21. ### 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4

Conclusion 21 / 30
22. ### Experiments User simulation for contour extraction evaluation simulate the human

user behavior with an agent ﬁnd the longest possible path with minimum error from a seed ps the path must be contained in a tube of radius r on both sides of the ground truth realized by a dichotomous search along the ground truth object contour p s p e 2r ground truth ﬁnding longest path zoom 22 / 30
23. ### Experiments User simulation for contour extraction evaluation and so on

until the contour is closed ground truth ﬁnding longest path zoom after several steps ﬁnding longest path end 23 / 30
24. ### Experiments Dataset ISEG [Gulshan & al. Geodesic star convexity for

interactive image segmentation. CVPR 2010] What is the best patch size ? We considered w ∈ {1, 2, 3} corresponding to 3x3, 5x5 and 7x7 patches w = 1 w = 2 w = 3 Number of times a potential gives the minimum % of seeds ⇒ Best results are mostly always obtained with band-based potential Pmax 3 and w = 2 24 / 30
25. ### Experiments Quantitative results on ISEG Mean Mean Potential Radius %

seeds Jaccard index PE 2 4.58 0.96 3 2.96 0.95 4 2.31 0.94 Pmax 3 2 3.93 0.97 3 2.67 0.95 4 2.12 0.94 P4 2 4.42 0.95 3 3.13 0.93 4 2.52 0.91 P5 2 3.67 0.97 3 2.38 0.95 4 1.86 0.94 diﬀerence of 0.1 percent point corresponds to 1.5 seed PE : classical gradient-based potential (decreasing function of the gradient) 25 / 30
26. ### Experiments Qualitative results on ISEG PE , #seeds=28 Pmax 3

, #seeds=34 P4, #seeds=26 P5, #seeds=24 PE , #seeds=38 Pmax 3 , #seeds=53 P4, #seeds=37 P5, #seeds=32 26 / 30
27. ### Experiments Qualitative results on ISEG PE , #seeds=48 Pmax 3

, #seeds=26 P4, #seeds=45 P5, #seeds=32 PE , #seeds=47 Pmax 3 , #seeds=37 P4, #seeds=46 P5, #seeds=28 27 / 30
28. ### 1 Introduction and Motivations 2 Patch-based potentials 3 Experiments 4

Conclusion 28 / 30
29. ### Conclusion We have proposed: new patch-based potentials for contour extraction

quantitative experiments were simulated with a user agent enables to improve classical gradient-based potential enables to reduce the user interaction (in terms of required points for object extraction) 29 / 30