Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
Cloud2Curve: Generation and Vectorization of
Parametric Sketches
Ayan Das1,2, Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2
1 SketchX, CVSSP, University of Surrey, United Kingdom
2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence
3 University of Edinburgh, United Kingdom
CVPR 2021 (Poster)
Slide 3
Slide 3 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 4
Slide 4 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 5
Slide 5 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 6
Slide 6 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
…
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 7
Slide 7 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 8
Slide 8 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
Slide 9
Slide 9 text
Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
[0,0]
p1
p2
~
ε
Require polyline format for
inference
Slide 10
Slide 10 text
Parametric representation
Slide 11
Slide 11 text
Parametric representation
Represent sketches with Bézier curves
Slide 12
Slide 12 text
Parametric representation
Represent sketches with Bézier curves
Degree ‘n’
No. of control points ‘n+1’
Slide 13
Slide 13 text
Parametric representation
Represent sketches with Bézier curves
t
Degree ‘n’
No. of control points ‘n+1’
Slide 14
Slide 14 text
Parametric representation
Represent sketches with Bézier curves
• Shorter in length
• Arbitrary resolution
• Scalable, High quality
t
Degree ‘n’
No. of control points ‘n+1’
Slide 15
Slide 15 text
Parametric representation
Represent sketches with Bézier curves
• Shorter in length
• Arbitrary resolution
• Scalable, High quality
t
Degree ‘n’
No. of control points ‘n+1’
Slide 16
Slide 16 text
A stroke-by-stroke autoregressive model
[BezierSketch, ECCV, Das et al. 2020]
Slide 17
Slide 17 text
A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
[BezierSketch, ECCV, Das et al. 2020]
Slide 18
Slide 18 text
A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
t
[BezierSketch, ECCV, Das et al. 2020]
Slide 19
Slide 19 text
A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
• Compare each with ground-truth strokes
t
[BezierSketch, ECCV, Das et al. 2020]
Slide 20
Slide 20 text
A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
• Compare each with ground-truth strokes
t
Each stroke must be represented by
Bezier curve of same degree
[BezierSketch, ECCV, Das et al. 2020]
Slide 21
Slide 21 text
Variable Degree Bezier Curve (Idea)
Slide 22
Slide 22 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
Slide 23
Slide 23 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
t
Slide 24
Slide 24 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
A maximum set of
control points
Value in [0, 1]
Slide 25
Slide 25 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
A maximum set of
control points
Value in [0, 1]
Slide 26
Slide 26 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
L( , )
A maximum set of
control points
Value in [0, 1]
Slide 27
Slide 27 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
• Fully differentiable w.r.t downstream loss
t
r
L( , )
A maximum set of
control points
Value in [0, 1]
Slide 28
Slide 28 text
Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
• Fully differentiable w.r.t downstream loss
t
r
L( , )
A maximum set of
control points
Notice how degree
changing while
optimizing
Value in [0, 1]
Slide 29
Slide 29 text
Variable Degree Bezier Curve (Formulation)
Slide 30
Slide 30 text
Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)
Slide 31
Slide 31 text
Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)
• Compute “Degree selector” R(r)
Refer to the paper for details
Slide 32
Slide 32 text
Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)
• Compute “Degree selector” R(r)
• Augmented each component
Refer to the paper for details
Slide 33
Slide 33 text
Variable degree stroke decoder
Slide 34
Slide 34 text
Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
• Outputs an extra “degree parameter” at each timestep
Slide 35
Slide 35 text
Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
Slide 36
Slide 36 text
Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train
Slide 37
Slide 37 text
Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train
A point-cloud based encoder
• We used set-transformer to encode sketch as
point cloud
ε
Full Cloud2Curve
Model
Slide 38
Slide 38 text
Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train
A point-cloud based encoder
• We used set-transformer to encode sketch as
point cloud
• Possible to infer from non-sequential modalities
(raster etc)
ε
Full Cloud2Curve
Model
Slide 39
Slide 39 text
Loss functions & Regularization
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
Slide 40
Slide 40 text
Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
Slide 41
Slide 41 text
Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
Slide 42
Slide 42 text
Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
• Degree regularizer:
❑ Enforcing the degree to be as low as
possible
Slide 43
Slide 43 text
Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
• Degree regularizer:
❑ Enforcing the degree to be as low as
possible
High value of degree regularizer enforces same stroke to acquire less degree
Slide 44
Slide 44 text
Experiments & Results (Qualitative)
Slide 45
Slide 45 text
Experiments & Results (Qualitative)
Given point-cloud based sketch, Cloud2Curve generates
parametric samples
(more results in the paper)
Slide 46
Slide 46 text
Experiments & Results (Qualitative)
Given point-cloud based sketch, Cloud2Curve generates
parametric samples
(more results in the paper)
Higher “degree regularizer” reduced overall complexity
of strokes
Slide 47
Slide 47 text
Experiments & Results (Quantitative)
Slide 48
Slide 48 text
Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
Stroke Histogram Sketch Histogram
Slide 49
Slide 49 text
Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
• Shows classification acc. of generated samples
• Shows FID of generated samples
Stroke Histogram Sketch Histogram
Classification score FID score
Slide 50
Slide 50 text
Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
• Shows classification acc. of generated samples
• Shows FID of generated samples
• Vectorization Model
• Same model without source of randomness
• Measure classification acc. and Test loss
Stroke Histogram Sketch Histogram
Classification score FID score
Slide 51
Slide 51 text
Thank You
Project page: https://ayandas.me/pubs/2021/03/01/pub-9.html