Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BézierSketch: A generative model for scalable vector sketches

BézierSketch: A generative model for scalable vector sketches

The study of neural generative models of human sketches is a fascinating contemporary modeling problem due to the links between sketch image generation and the human drawing process. The landmark SketchRNN provided breakthrough by sequentially generating sketches as a sequence of waypoints. However this leads to low-resolution image generation, and failure to model long sketches. In this paper we present BézierSketch, a novel generative model for fully vector sketches that are automatically scalable and high-resolution. To this end, we first introduce a novel inverse graphics approach to stroke embedding that trains an encoder to embed each stroke to its best fit Bézier curve. This enables us to treat sketches as short sequences of paramaterized strokes and thus train a recurrent sketch generator with greater capacity for longer sketches, while producing scalable high-resolution results. We report qualitative and quantitative results on the Quick, Draw! benchmark.

Ayan Das

May 13, 2021
Tweet

More Decks by Ayan Das

Other Decks in Research

Transcript

  1. BézierSketch: A generative model for scalable vector sketches Ayan Das1,2,

    Yongxin Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom 2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence 3 University of Edinburgh, United Kingdom ECCV 2020 (Poster)
  2. Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha

    et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  3. Traditional Sketch generator ➢ Prior works: ❑ Sketch-RNN[1] by Ha

    et. al. for free-hand doodles ❑ Handwriting generation by A. Graves[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013) ➢ Generates explicit waypoints, i.e., sampled coordinates from the continuous drawing trajectory ❑ Relatively longer representation ❑ More temporal correlation ❑ Fixed resolution (depends on sampling rate) ❑ Bad quality, at least for artistic applications
  4. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised)

    ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) Vectorized Sketch generator
  5. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised)

    ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator
  6. ➢ Prior works: ❑ Learning to synthesize parametric curves[1] (unsupervised)

    ❑ Font generation[2] (supervised) [1] Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S.M.A., Vinyals, O.: Synthesizing programs for images using reinforced adversarial learning. In: ICML (2018) [2] Lopes, R.G., Ha, D., Eck, D., Shlens, J.: A learned representation for scalable vector graphics. In: ICCV (2019) ➢ Generates parametric curve specifications, i.e., control points of Bézier curves, B-Splines, etc. ❑ Relatively shorter representation ❑ Inherently arbitrary resolution ❑ High quality for artistic uses. Vectorized Sketch generator
  7. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  8. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  9. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  10. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  11. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  12. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+
  13. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation
  14. ❑ Converts traditional waypoint-based representation into Bézier curves[1] ❑ The

    model is learnable, high-quality and faster than standard fitting-based algorithms BézierEncoder: An inverse-graphics based encoder for Bézierrepresentations [1] Salomon, D.: Curves and surfaces for computer graphics. Springer Science & Business Media (2007) Softmax+ Bézier curve formulation Non-parametric Decoder Parametric Encoder
  15. BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢

    Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves
  16. BézierEncoder: Training ➢ We simply minimize reconstruction loss : ➢

    Furthermore, we extend this simple formulation by considering an ensemble of losses with different degrees of Bézier curves ➢ Restricting consecutive control points to be closer has regularizing effect
  17. BézierSketch: Control Point mode ΔP1 [0,0] [0,0] Pi-1 ❑ At

    each time-step, it predicts difference vector of successive control points. ΔP1
  18. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi

    Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1
  19. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi

    Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN
  20. BézierSketch: Control Point mode ΔP1 [0,0] ΔPi Δpi-1 [0,0] ΔPi

    Pi-1 Pi … ΔPi ❑ At each time-step, it predicts difference vector of successive control points. ΔP1 ❑ Also predicts “stroke-end” and “sketch- end” bit, just like Sketch-RNN ❑ Average length goes down by 33% Stroke lengths Sketch lengths
  21. S1 Si Si-1 … S0 BézierSketch: Stroke mode ❑ At

    each time-step, it predicts the next stroke
  22. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑

    At each time-step, it predicts the next stroke
  23. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑

    At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve)
  24. S1 Si Si-1 … S0 Si BézierSketch: Stroke mode ❑

    At each time-step, it predicts the next stroke ❑ Fixed-degree Bézier representation used. Predicts 2N real numbers (N being the degree of Bézier curve) ❑ RNN time-steps is the number of strokes in a sketch
  25. Qualitative results Unconditional “Control point mode” generation Unconditional “Stroke mode”

    generation Conditional “Control point mode” generation Conditional “Stroke mode” generation
  26. Future works ❑ Use more sophisticated parametric curves, like BSpline

    ❑ Build encoder that predicts parametric curves directly from raster images