BézierSketch: A generative model for scalable vector sketches

The study of neural generative models of human sketches is a fascinating contemporary modeling problem due to the links between sketch image generation and the human drawing process. The landmark SketchRNN provided breakthrough by sequentially generating sketches as a sequence of waypoints. However this leads to low-resolution image generation, and failure to model long sketches. In this paper we present BézierSketch, a novel generative model for fully vector sketches that are automatically scalable and high-resolution. To this end, we first introduce a novel inverse graphics approach to stroke embedding that trains an encoder to embed each stroke to its best fit Bézier curve. This enables us to treat sketches as short sequences of paramaterized strokes and thus train a recurrent sketch generator with greater capacity for longer sketches, while producing scalable high-resolution results. We report qualitative and quantitative results on the Quick, Draw! benchmark.

Transcript

2. BézierSketch: A generative model for scalable vector sketches Ayan Das1,2, Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom 2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence 3 University of Edinburgh, United Kingdom ECCV 2020 (Poster)
3. Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
4. Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013) ➢ Generates explicit waypoints, i.e., sampled coordinates from the continuous drawing trajectory ❑ Relatively longer representation ❑ More temporal correlation ❑ Fixed resolution (depends on sampling rate) ❑ Bad quality, at least for artistic applications
5. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) Vectorized Sketch generator
6. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator
7. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised) ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator
8. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
9. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
10. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
11. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
12. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
13. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
14. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation
15. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation Non-parametric Decoder Parametric Encoder
16. BézierEncoder: Training ➢ We simply minimize reconstruction loss :
17. BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves
18. BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢ Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves ➢ Restricting consecutive control points to be closer has regularizing effect
19. Waypoint based representation Bézier curve representation BézierEncoder: Inference BézierEncoder
20. Waypoint based representation Bézier curve representation BézierEncoder: Inference BézierEncoder
21. BézierSketch: Control Point mode ΔP1 [0,0] [0,0] Pi-1 ❑ At each time-step, it predicts difference vector of successive control points. ΔP1
22. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1
23. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN
24. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN ❑ Average length goes down by 33% Stroke lengths Sketch lengths
25. S1 Si Si-1 … S0 BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke
26. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke
27. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve)
28. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑ At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve) ❑ RNN time-steps is the number of strokes in a sketch
29. Qualitative results Unconditional “Control point mode” generation Unconditional “Stroke mode” generation Conditional “Control point mode” generation Conditional “Stroke mode” generation
30. Future works
31. Future works ❑ Use more sophisticated parametric curves, like BSpline
32. Future works ❑ Use more sophisticated parametric curves, like BSpline ❑ Build encoder that predicts parametric curves directly from raster images
33. Thank You