No content

BézierSketch: A generative model for scalable vector sketches Ayan Das1,2, Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom 2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence 3 University of Edinburgh, United Kingdom ECCV 2020 (Poster)

Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)

Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013) ➢ Generates explicit waypoints, i.e., sampled coordinates from the continuous drawing trajectory ❑ Relatively longer representation ❑ More temporal correlation ❑ Fixed resolution (depends on sampling rate) ❑ Bad quality, at least for artistic applications

➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) Vectorized Sketch generator

➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator

➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation

❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation Non-parametric Decoder Parametric Encoder

BézierEncoder: Training ➢ We simply minimize reconstruction loss :

BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves

BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves ➢ Restricting consecutive control points to be closer has regularizing effect

Waypoint based representation Bézier curve representation BézierEncoder: Inference BézierEncoder

Waypoint based representation Bézier curve representation BézierEncoder: Inference BézierEncoder

BézierSketch: Control Point mode ΔP1 [0,0] [0,0] Pi-1 ❑ At each time-step, it predicts difference vector of successive control points. ΔP1

BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1

BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN

BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN ❑ Average length goes down by 33% Stroke lengths Sketch lengths

S1 Si Si-1 … S0 BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke

S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke

S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve)

S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve) ❑ RNN time-steps is the number of strokes in a sketch

Qualitative results Unconditional “Control point mode” generation Unconditional “Stroke mode” generation Conditional “Control point mode” generation Conditional “Stroke mode” generation

Future works

Future works ❑ Use more sophisticated parametric curves, like BSpline

Future works ❑ Use more sophisticated parametric curves, like BSpline ❑ Build encoder that predicts parametric curves directly from raster images

Thank You