No content

BézierSketch: A generative model for
scalable vector sketches
Ayan Das1,2, Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2
1 SketchX, CVSSP, University of Surrey, United Kingdom
2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence
3 University of Edinburgh, United Kingdom
ECCV 2020 (Poster)

Traditional Sketch generator
➢ Prior works:
❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles
❑ Handwriting generation by A. Graves[2]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generator
➢ Prior works:
❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles
❑ Handwriting generation by A. Graves[2]
[1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018)
[2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
➢ Generates explicit waypoints, i.e., sampled coordinates
from the continuous drawing trajectory
❑ Relatively longer representation
❑ More temporal correlation
❑ Fixed resolution (depends on sampling rate)
❑ Bad quality, at least for artistic applications

➢ Prior works:
❑ Learning to synthesize parametric curves[1]
(unsupervised)
❑ Font generation[2] (supervised)
[1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018)
[2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019)
Vectorized Sketch generator

➢ Prior works:
❑ Learning to synthesize parametric curves[1]
(unsupervised)
❑ Font generation[2] (supervised)
[1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018)
[2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019)
➢ Generates parametric curve specifications, i.e.,
control points of Bézier curves, B-Splines, etc.
❑ Relatively shorter representation
❑ Inherently arbitrary resolution
❑ High quality for artistic uses.
Vectorized Sketch generator

➢ Prior works:
❑ Learning to synthesize parametric curves[1]
(unsupervised)
❑ Font generation[2] (supervised)
[1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018)
[2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019)
➢ Generates parametric curve specifications, i.e.,
control points of Bézier curves, B-Splines, etc.
❑ Relatively shorter representation
❑ Inherently arbitrary resolution
❑ High quality for artistic uses.
Vectorized Sketch generator

❑ Converts traditional waypoint-based representation into Bézier curves[1]
❑ The model is learnable, high-quality and faster than standard fitting-based algorithms
BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations
[1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007)
Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1]
❑ The model is learnable, high-quality and faster than standard fitting-based algorithms
BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations
[1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007)
Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1]
❑ The model is learnable, high-quality and faster than standard fitting-based algorithms
BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations
[1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007)
Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1]
❑ The model is learnable, high-quality and faster than standard fitting-based algorithms
BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations
[1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007)
Softmax+
Bézier curve formulation

❑ Converts traditional waypoint-based representation into Bézier curves[1]
❑ The model is learnable, high-quality and faster than standard fitting-based algorithms
BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations
[1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007)
Softmax+
Bézier curve formulation Non-parametric Decoder
Parametric Encoder

BézierEncoder: Training
➢ We simply minimize reconstruction loss :

BézierEncoder: Training
➢ We simply minimize reconstruction loss :
➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different
degrees of Bézier curves

BézierEncoder: Training
➢ We simply minimize reconstruction loss :
➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different
degrees of Bézier curves
➢ Restricting consecutive control points to be closer has regularizing effect

Waypoint based representation Bézier curve representation
BézierEncoder: Inference
BézierEncoder

Waypoint based representation Bézier curve representation
BézierEncoder: Inference
BézierEncoder

BézierSketch: Control Point mode
ΔP1
[0,0]
[0,0]
Pi-1
❑ At each time-step, it predicts difference vector of
successive control points.
ΔP1

BézierSketch: Control Point mode
ΔP1
[0,0]
ΔPi
Δpi-1
[0,0]
ΔPi
Pi-1
Pi
…
ΔPi
❑ At each time-step, it predicts difference vector of
successive control points.
ΔP1

BézierSketch: Control Point mode
ΔP1
[0,0]
ΔPi
Δpi-1
[0,0]
ΔPi
Pi-1
Pi
…
ΔPi
❑ At each time-step, it predicts difference vector of
successive control points.
ΔP1
❑ Also predicts “stroke-end” and “sketch-
end” bit, just like Sketch-RNN

BézierSketch: Control Point mode
ΔP1
[0,0]
ΔPi
Δpi-1
[0,0]
ΔPi
Pi-1
Pi
…
ΔPi
❑ At each time-step, it predicts difference vector of
successive control points.
ΔP1
❑ Also predicts “stroke-end” and “sketch-
end” bit, just like Sketch-RNN
❑ Average length goes down by 33%
Stroke lengths Sketch lengths

S1
Si
Si-1
…
S0
BézierSketch: Stroke mode
❑ At each time-step, it predicts the next stroke

S1
Si
Si-1
…
S0
Si
BézierSketch: Stroke mode
❑ At each time-step, it predicts the next stroke

S1
Si
Si-1
…
S0
Si
BézierSketch: Stroke mode
❑ At each time-step, it predicts the next stroke
❑ Fixed-degree Bézier representation used. Predicts 2N
real numbers (N being the degree of Bézier curve)

S1
Si
Si-1
…
S0
Si
BézierSketch: Stroke mode
❑ At each time-step, it predicts the next stroke
❑ Fixed-degree Bézier representation used. Predicts 2N
real numbers (N being the degree of Bézier curve)
❑ RNN time-steps is the number of strokes in a sketch

Qualitative results
Unconditional “Control point mode” generation Unconditional “Stroke mode” generation
Conditional “Control point mode” generation Conditional “Stroke mode” generation

Future works

Future works
❑ Use more sophisticated parametric curves, like BSpline

Future works
❑ Use more sophisticated parametric curves, like BSpline
❑ Build encoder that predicts parametric curves directly from raster images

Thank You