No content

Cloud2Curve: Generation and Vectorization of
Parametric Sketches
Ayan Das1,2, Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2
1 SketchX, CVSSP, University of Surrey, United Kingdom
2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence
3 University of Edinburgh, United Kingdom
CVPR 2021 (Poster)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
…
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generation models
• The state-of-the-art “SketchRNN”[1]
❑ Uses Polyline representation
❑ Autoregressive generation of points[2]
Δp1
[0,0]
Δpi
Δpi-1
…
[0,0]
p1
p2
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
[0,0]
p1
p2
~
ε
Require polyline format for
inference

Parametric representation

Parametric representation
Represent sketches with Bézier curves

Parametric representation
Represent sketches with Bézier curves
Degree ‘n’
No. of control points ‘n+1’

Parametric representation
Represent sketches with Bézier curves
t
Degree ‘n’
No. of control points ‘n+1’

Parametric representation
Represent sketches with Bézier curves
• Shorter in length
• Arbitrary resolution
• Scalable, High quality
t
Degree ‘n’
No. of control points ‘n+1’

Parametric representation
Represent sketches with Bézier curves
• Shorter in length
• Arbitrary resolution
• Scalable, High quality
t
Degree ‘n’
No. of control points ‘n+1’

A stroke-by-stroke autoregressive model
[BezierSketch, ECCV, Das et al. 2020]

A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
[BezierSketch, ECCV, Das et al. 2020]

A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
t
[BezierSketch, ECCV, Das et al. 2020]

A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
• Compare each with ground-truth strokes
t
[BezierSketch, ECCV, Das et al. 2020]

A stroke-by-stroke autoregressive model
{Pi
}
{Pi-1
} {Pi
}
{Pi+1
}
• Output one stroke (Bezier Curve) at a time
• Instantiate with predefine set of “t” values
• Compare each with ground-truth strokes
t
Each stroke must be represented by
Bezier curve of same degree
[BezierSketch, ECCV, Das et al. 2020]

Variable Degree Bezier Curve (Idea)

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
t

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
A maximum set of
control points
Value in [0, 1]

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
A maximum set of
control points
Value in [0, 1]

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
t
r
L( , )
A maximum set of
control points
Value in [0, 1]

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
• Fully differentiable w.r.t downstream loss
t
r
L( , )
A maximum set of
control points
Value in [0, 1]

Variable Degree Bezier Curve (Idea)
• Degree of Bezier curve is a discrete quantity, difficult to optimize
• We contribute a Bezier curve formulation with variable degree
• One scalar “degree parameter” is used
• Fully differentiable w.r.t downstream loss
t
r
L( , )
A maximum set of
control points
Notice how degree
changing while
optimizing
Value in [0, 1]

Variable Degree Bezier Curve (Formulation)

Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)

Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)
• Compute “Degree selector” R(r)
Refer to the paper for details

Variable Degree Bezier Curve (Formulation)
• The Bezier curve formula has three components (Time matrix,
Degree matrix and Control Points)
• Compute “Degree selector” R(r)
• Augmented each component
Refer to the paper for details

Variable degree stroke decoder

Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
• Outputs an extra “degree parameter” at each timestep

Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation

Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train

Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train
A point-cloud based encoder
• We used set-transformer to encode sketch as
point cloud
ε
Full Cloud2Curve
Model

Variable degree stroke decoder
{Pi
}, r {Pi+1
}, r
t
• Outputs an extra “degree parameter” at each timestep
• Instantiates using Variable degree Bezier formulation
• Compute loss and backpropagate to train
A point-cloud based encoder
• We used set-transformer to encode sketch as
point cloud
• Possible to infer from non-sequential modalities
(raster etc)
ε
Full Cloud2Curve
Model

Loss functions & Regularization
Predicted Bezier Curve (P)
Ground-Truth stroke (s)

Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
Predicted Bezier Curve (P)
Ground-Truth stroke (s)

Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)

Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
• Degree regularizer:
❑ Enforcing the degree to be as low as
possible

Loss functions & Regularization
• Set-based loss: Treat strokes as point-cloud
❑ Sliced Wasserstein Distance (SWD)[1]
• Sequential loss: Treat strokes as sequences
❑ Optional. Helps the model fit better.
❑ Standard point-to-point MSE loss
Predicted Bezier Curve (P)
Ground-Truth stroke (s)
• Degree regularizer:
❑ Enforcing the degree to be as low as
possible
High value of degree regularizer enforces same stroke to acquire less degree

Experiments & Results (Qualitative)

Experiments & Results (Qualitative)
Given point-cloud based sketch, Cloud2Curve generates
parametric samples
(more results in the paper)

Experiments & Results (Qualitative)
Given point-cloud based sketch, Cloud2Curve generates
parametric samples
(more results in the paper)
Higher “degree regularizer” reduced overall complexity
of strokes

Experiments & Results (Quantitative)

Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
Stroke Histogram Sketch Histogram

Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
• Shows classification acc. of generated samples
• Shows FID of generated samples
Stroke Histogram Sketch Histogram
Classification score FID score

Experiments & Results (Quantitative)
• Shows the length histogram of strokes
• Shows the length histogram of sketches
• Shows classification acc. of generated samples
• Shows FID of generated samples
• Vectorization Model
• Same model without source of randomness
• Measure classification acc. and Test loss
Stroke Histogram Sketch Histogram
Classification score FID score

Thank You
Project page: https://ayandas.me/pubs/2021/03/01/pub-9.html