Behind the Scenes with Auto Layout or How to Solve Constraints with the Cassowary Algorithm (iOSConfSG)

Auto Layout is the most powerful tool on iOS today, to support developers in creating adaptive user interfaces on iOS. The power of this great technology lies in how simple its foundations are. The engineers behind Auto Layout at Apple started to reveal more of the internals of their technology recently, to help us understand its principles better, in order to get the best performance out of our layout code. Let’s look more closely at how the Auto Layout engine solves constraints through exploring the Cassowary algorithm, that’s running under the hood, turning constraints into linear equations, and solving them. By doing the math ourselves, we’ll have a better understanding of what layouts are easier, or more difficult to satisfy. What coding practices can help us achieve best Auto Layout performance.

Transcript

1. BEHIND THE SCENES WITH AUTO LAYOUT or how to solve constraints with the Cassowary algorithm Agnes Vasarhelyi @vasarhelyia
2. WWDC '18 High Performance Auto Layout
3. Constraints Displayed content ?
4. Constraints Displayed content ? WHAT
5. Constraints Displayed content ? WHAT RESULT
6. Constraints Displayed content ? WHAT RESULT HOW
7. From developer to user Developer Auto Layout User Constraints Render Loop Adaptivity
8. Your input ! • Satisfiable, non-ambiguous layouts • Size and position of content • Constraints (either code, or IB)
9. The Render Loop Constraints 1. Layout 2. Display 3. layoutSubviews() setNeedsLayout() layoutIfNeeded() updateConstraints() setNeedsUpdateConstraints() updateConstraintsIfNeeded() draw(_ :) setNeedsDisplay()
10. The Render Loop Constraints 1. Layout 2.
11. Auto Layout Engine Window
12. Auto Layout Engine Window View
13. Auto Layout Engine Window AL Engine View
14. Auto Layout Engine Window AL Engine View Constraint
15. Auto Layout Engine Window AL Engine View Constraint Add
16. Auto Layout Engine Window AL Engine View Constraint = Equation Add
17. Auto Layout Engine Window AL Engine View Constraint = Equation Add Variables
18. Auto Layout Engine View Values AL Engine What are the values for this frame? Stores values
19. Auto Layout Engine View Values Value changed AL Engine What are the values for this frame? Stores values
20. Auto Layout Engine View Values Value changed AL Engine What are the values for this frame? Stores values setNeedsLayout() layoutSubviews()
21. –Ken Ferry, WWDC '18 “The engine is a layout cache and dependency tracker.”
22. WWDC 2018 • Layout performance scales linearly • Constraint inequalities are not expensive • Error minimization has a cost • Constraint solver algorithm: Simplex (Cassowary)
23. PlanGrid Papers • "Paper" club • One per week • You pick, you lead
24. Cassowary paper ‘97 • Based on Simplex (40's) linear programming algorithm • Constraint solving • UI applications • Incremental, optimized
25. Incremental, optimized
26. Incremental, optimized
27. Incremental, optimized
28. Incremental, optimized
29. Incremental, optimized
30. • Similar problems repeatedly • Add / remove / edit efficient Performance Incremental, optimized
31. Linear Programming • Maximizing, or minimizing of model • Requirements = linear equations/inequalities • Objective function = goal
32. Constraint solving LP problem • Objective function minimizing distance / error • Constraints linear equations / inequalities
33. Let's solve constraints! x b y a leading leading width
34. Let's solve constraints! x b y a 10 15 30
35. Let's solve constraints! x b y a xa = 15
36. Let's solve constraints! x b y a xa = 15 widtha = 30
37. Let's solve constraints! x b y a xa = 15 widtha = 30 xb = xa + widtha + 10
38. Let's solve constraints! x b y a xa = 15 widtha = 30 xb = xa + widtha + 10 xb = 15 + 30 + 10 = 55
39. Let's solve constraints! x b y a xa = 15 widtha = 30 xb = xa + widtha + 10 xb = 15 + 30 + 10 = 55
40. xleft ≤ xmiddle ≤ xright xleft ≥ 0 xright ≤ 100 ≤ ≤ x y xmiddle xright xleft
41. xmiddle = (xleft + xright ) 2 xleft + 10 ≤ xright ≤ ≤ x y xmiddle xright xleft
42. 1. Augmented Simplex Form 2. Basic Feasible Solved Form 3. Basic Feasible Solution 4. Simplex Optimization Cassowary
43. 1. Augmented Simplex Form • Convert inequalities to equalities • Separating out possibly negative variables
44. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100
45. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100
46. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr
47. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr
48. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr
49. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr xr + s2 = 100
50. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr xr + s2 = 100
51. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr 0 ≤ xl , s1 , s2 xr + s2 = 100
52. Converting to equalities 2xm = xl + xr xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 2xm = xl + xr xl + 10 + s1 = xr 0 ≤ xl , s1 , s2 xr + s2 = 100 slack variables * s1 , s2 :
53. minimize xm − xl
54. x y xm − xl xmiddle xright xleft
55. 2xm = xl + xr xl + 10 + s1 = xr 0 ≤ xl , s1 , s2 xr + s2 = 100 minimize xm − xl subject to
56. • Cu, unrestricted: possibly negative • Cs, Simplex: non-negative • Substituting in Cu to Cs Cu / Cs separation
57. 2xm = xl + xr xl + 10 + s1 = xr 0 ≤ xl , s1 , s2 xr + s2 = 100 Cu Cs Cu: unrestricted constraints, * CS: Simplex constraints
58. 2xm = xl + xr xl + 10 + s1 = xr 0 ≤ xl , s1 , s2 xr + s2 = 100 Cu Cs Cu: unrestricted constraints, * CS: Simplex constraints
59. 2xm = xl + 100 − s2 xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs Cu: unrestricted constraints, * CS: Simplex constraints
60. 2xm = xl + 100 − s2 xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs Cu: unrestricted constraints, * CS: Simplex constraints
61. xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs minimize subject to xm − xl 2xm = xl + 100 − s2 Cu: unrestricted constraints, * CS: Simplex constraints
62. xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs minimize subject to xm − xl Cu: unrestricted constraints, * CS: Simplex constraints
63. xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs minimize subject to xm = 50 + 1 2 xl − 1 2 s2 xm − xl Cu: unrestricted constraints, * CS: Simplex constraints
64. xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs minimize subject to xm = 50 + 1 2 xl − 1 2 s2 Cu: unrestricted constraints, * CS: Simplex constraints
65. xl + 10 + s1 = 100 − s2 0 ≤ xl , s1 , s2 xr = 100 − s2 Cu Cs minimize subject to xm = 50 + 1 2 xl − 1 2 s2 50 − 1 2 xl − 1 2 s2 Cu: unrestricted constraints, * CS: Simplex constraints
66. 2. Basic feasible solved form x0 = c + a1 x1 + . . . + an xn x0 is basic, x1 . . . xn are parameters (Cs) c ≥ 0
67. s1 = xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 50 90 −s2 + 1 2 xl − 1 2 s2 −xl −s2 2. Basic feasible solved form xm , xr , s1 are basic, * xl , s2 are parametric
68. s1 = xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 50 90 2. Basic feasible solved form xm , xr , s1 are basic, * xl , s2 are parametric
69. 3. Basic feasible solution xr 100, xm 50, s1 90, xl 0, s2 0
70. 3. Basic feasible solution xr 100, xm 50, s1 90, xl 0, s2 0 50 − 1 2 xl − 1 2 s2 Objective function
71. 3. Basic feasible solution xr 100, xm 50, s1 90, xl 0, s2 0 50 − 1 2 xl − 1 2 s2 Objective function 50
72. x y 50 100 0 xr 100 xm 50, xl 0, 50 A basic feasible solution xmiddle xright xleft
73. x y 50 100 0 xr 100 xm 50, xl 0, 50 xm = (xl + xr ) 2 xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 Original constraints A basic feasible solution xmiddle xright xleft
74. 4. Simplex Optimization 1. Find (adjacent) basic feasible solved form 2. Pivot: swap out basic and parametric vars fobj
75. 4. Simplex Optimization s1 xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 90 −s2 50 + 1 2 xl − 1 2 s2 − −s2 xl =
76. 4. Simplex Optimization s1 xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 90 −s2 50 + 1 2 xl − 1 2 s2 − −s2 xl =
77. 4. Simplex Optimization s1 xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 90 −s2 50 + 1 2 xl − 1 2 s2 − −s2 xl =
78. 4. Simplex Optimization s1 xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 90 −s2 − −s2 xl =
79. 4. Simplex Optimization s1 xr = minimize subject to xm = 50 − 1 2 xl − 1 2 s2 100 90 −s2 − −s2 xl = 95 − 1 2 s1 −s2
80. 4. Simplex Optimization s1 xr = minimize subject to xm = 100 90 −s2 − −s2 xl = 95 − 1 2 s1 −s2
81. 4. Simplex Optimization s1 xr = minimize subject to xm = 100 90 −s2 − −s2 xl = 5 + 1 2 s1 95 − 1 2 s1 −s2
82. 4. Simplex Optimization s1 xr = minimize subject to xm = 100 90 −s2 − −s2 xl = 5 + 1 2 s1 95 − 1 2 s1 −s2 xm , xr , xl are basic, * s1 , s2 are parametric
83. 4. Simplex Optimization s1 xr = minimize subject to xm = 100 90 −s2 − −s2 xl = 5 + 1 2 s1 95 − 1 2 s1 −s2 xm , xr , xl are basic, * s1 , s2 are parametric
84. 4. Simplex Optimization s1 xr = minimize subject to xm = 100 90 −s2 − −s2 xl = 5 + 1 2 s1 95 − 1 2 s1 −s2 5 xm , xr , xl are basic, * s1 , s2 are parametric
85. x y 90 100 0 90 95 xr 100 xm 95, xl 90, 5 Optimal solution xm xr xl
86. x y 90 100 0 90 95 xr 100 xm 95, xl 90, 5 Optimal solution xm = (xl + xr ) 2 xl + 10 ≤ xr xl ≥ 0 xr ≤ 100 Original constraints xm xr xl
88. Adding constraints • Convert to augmented simplex form • Substitute new constraint in using current table • Reach basic feasible solved form
89. Removing constraints • Marker variables to keep track of original constraints • Pivot until marker is basic • Remove row
90. Constraint priorities • Weights added to constraints • Makes objective function non-linear • Quasi-linear optimization
91. WWDC '18 • Performance scales linearly (in independent views) • Inequalities are not expensive (one extra variable) • Error minimization has a cost (constraint priorities)
92. WWDC '18 • Performance scales linearly (in independent views) • Inequalities are not expensive (one extra variable) • Error minimization has a cost (constraint priorities)
93. WWDC '18 • Performance scales linearly (in independent views) • Inequalities are not expensive (one extra variable) • Error minimization has a cost (constraint priorities)
94. WWDC '18 • Performance scales linearly (in independent views) • Inequalities are not expensive (one extra variable) • Error minimization has a cost (constraint priorities)
95. “You don’t pay for what you don’t use” –Ken Ferry, WWDC '18
96. Agnes Vasarhelyi @vasarhelyia Thank you!
97. Resources • Auto Layout Documentation • High Performance Auto Layout WWDC '18 • Cassowary paper '97 • The Simplex Method: An Example (UT Dallas)