Shape Recognition

Transcript

1. Shape Recognition with Constraint Programming Steven Chan
2. Goal
3. Goal
4. Goal
5. Motivation • Most existing apps require drawing shapes in one stroke. • We should let users draw the shape as naturally as they would with pen and paper. • Add new shapes by natural descriptions.
6. Outline of System 1. Preprocessing: turn raw (x,y) input into line segments. 2. Recognition: shape CSPs. 3. Beautification: replace with a perfect shape.
7. Preprocessing 1. Find corners from a list of (x,y) points. 2. Replace by line segments by connecting the end points and the corners.
8. Preprocessing 1. Find corners from a list of (x,y) points. 2. Replace by line segments by connecting the end points and the corners.
9. Preprocessing 1. Find corners from a list of (x,y) points. 2. Replace by line segments by connecting the end points and the corners.
10. Preprocessing 1. Find corners from a list of (x,y) points. 2. Replace by line segments by connecting the end points and the corners. 4 line segments here l1 l2 l3 l4
11. Rectangle CSP • Variables: {Left, Right, Top, Bottom} • Domains: all existing line segments: {l1, l2, l3, l4} 4 line segments here l1 l2 l3 l4
12. Rectangle CSP • Variables: {Left, Right, Top, Bottom} • Domains: all existing line segments: {l1, l2, l3, l4} • Constraints: • All-different(L, R, T, B) 4 line segments here l1 l2 l3 l4
13. Rectangle CSP • Variables: {Left, Right, Top, Bottom} • Domains: all existing line segments: {l1, l2, l3, l4} • Constraints: • All-different(L, R, T, B) • T and B are horizontal • L and R are vertical 4 line segments here l1 l2 l3 l4
14. Rectangle CSP • Variables: {Left, Right, Top, Bottom} • Domains: all existing line segments: {l1, l2, l3, l4} • Constraints: • All-different(L, R, T, B) • T and B are horizontal • L and R are vertical • T is above B 4 line segments here l1 l2 l3 l4
15. Clockwise-meet Constraint A B A clockwise-meet B if
16. Clockwise-meet Constraint A B A clockwise-meet B if 1. there is a common point, and
17. Clockwise-meet Constraint A B A clockwise-meet B if 1. there is a common point, and 2. A can be rotated to B around the common
 point in less than 180 degrees clockwise.
18. Rectangle CSP • Variables: {Left, Right, Top, Bottom} • Domains: all existing line segments: {l1, l2, l3, l4} • Constraints: • All-different(L, R, T, B) • T and B are horizontal • L and R are vertical • T is above B • L clockwise-meet B,
 B clockwise-meet R,
 R clockwise-meet T,
 T clockwise-meet L 4 line segments here l1 l2 l3 l4
19. Rectangle CSP DL = {l1, l2, l3, l4} DR = {l1, l2, l3, l4} DT = {l1, l2, l3, l4} DB = {l1, l2, l3, l4} 4 line segments here l1 l2 l3 l4
20. Rectangle CSP Enforce node-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2, l4} DB = {l2, l4} 4 line segments here l1 l2 l3 l4
21. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2, l4} DB = {l2, l4} Top is above Bottom: If T is assigned l4, there are no supports in DB. 4 line segments here l1 l2 l3 l4
22. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2} DB = {l2, l4} Top is above Bottom: If T is assigned l4, there are no supports in DB. 4 line segments here l1 l2 l3 l4
23. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2} DB = {l2, l4} Top is above Bottom: If B is assigned l2, there are no supports in DT. 4 line segments here l1 l2 l3 l4
24. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2} DB = {l4} Top is above Bottom: If B is assigned l2, there are no supports in DT. 4 line segments here l1 l2 l3 l4
25. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l1, l3} DT = {l2} DB = {l4} R clockwise-meet T: Remove l1 from DR 4 line segments here l1 l2 l3 l4
26. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l3} DT = {l2} DB = {l4} R clockwise-meet T: Remove l1 from DR 4 line segments here l1 l2 l3 l4
27. Rectangle CSP Enforce arc-consistency DL = {l1, l3} DR = {l3} DT = {l2} DB = {l4} R clockwise-meet T: Remove l1 from DR L clockwise-meet B: Remove l3 from DL 4 line segments here l1 l2 l3 l4
28. Rectangle CSP Enforce arc-consistency DL = {l1} DR = {l3} DT = {l2} DB = {l4} R clockwise-meet T: Remove l1 from DR L clockwise-meet B: Remove l3 from DL 4 line segments here l1 l2 l3 l4
29. Rectangle CSP Enforce arc-consistency DL = {l1} DR = {l3} DT = {l2} DB = {l4} All constraints are satisfied. Rectangle recognized. 4 line segments here l1 l2 l3 l4
30. Beautification L = l1, R = l3, T = l2, B = l4 L T R B
31. Beautification L = l1, R = l3, T = l2, B = l4 top left = average of top of L, and left of T. L T R B
32. Beautification L = l1, R = l3, T = l2, B = l4 top left = average of top of L, and left of T. bottom right = average of right of B, and bottom of R. L T R B
33. Beautification L = l1, R = l3, T = l2, B = l4 top left = average of top of L, and left of T. bottom right = average of right of B, and bottom of R. L T R B
34. Another Example: Arrow • All-different(A,B,C) A B C
35. Another Example: Arrow • All-different(A,B,C) • A and C have equal length A B C
36. Another Example: Arrow • All-different(A,B,C) • A and C have equal length • B is longer than C • B is longer than A A B C
37. Another Example: Arrow • All-different(A,B,C) • A and C have equal length • B is longer than C • B is longer than A • C clockwise-meet B (< 70°) • B clockwise-meet A (< 70°) • C clockwise-meet A (< 100°) A B C
38. Experiment • Implemented the constraints and a simple solver in Objective-C. • Tested the performance with up to ~350 line segments with four shapes.
39. Experiment
40. Limitation • Easy to under-specify a shape.
41. Limitation • Easy to under-specify a shape. • Example: • All-different(L, R, T, B) • T and B are horizontal • L and R are vertical • L clockwise-meet B,
 B clockwise-meet R,
 R clockwise-meet T,
 T clockwise-meet L L R T B
42. Limitation • Easy to under-specify a shape. • Example: • All-different(L, R, T, B) • T and B are horizontal • L and R are vertical • L clockwise-meet B,
 B clockwise-meet R,
 R clockwise-meet T,
 T clockwise-meet L L R T B Need constraint T above B
43. Future Work • Recognize shapes with other components.
 
 
 • Indexing and storing the line segments by their properties.
44. Q&A