Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Cloud2Curve: Generation and Vectorization of Pa...

Ayan Das
May 13, 2021

Cloud2Curve: Generation and Vectorization of Parametric Sketches

Analysis of human sketches in deep learning has advanced immensely through the use of waypoint-sequences rather than raster-graphic representations. We further aim to model sketches as a sequence of low-dimensional parametric curves. To this end, we propose an inverse graphics framework capable of approximating a raster or waypoint based stroke encoded as a point-cloud with a variable-degree Bezier curve. Building on this module, we present Cloud2Curve, a generative model for scalable high-resolution vector sketches that can be trained end-to-end using point-cloud data alone. As a consequence, our model is also capable of deterministic vectorization which
can map novel raster or waypoint based sketches to their corresponding high-resolution scalable Bezier equivalent. We evaluate the generation and vectorization capabilities of our model on Quick, Draw! and K-MNIST datasets.

Ayan Das

May 13, 2021
Tweet

More Decks by Ayan Das

Other Decks in Research

Transcript

  1. Cloud2Curve: Generation and Vectorization of Parametric Sketches Ayan Das1,2, Yongxin

    Yang1,2, Timothy Hospedales1,3, Tao Xiang1,2 and Yi-Zhe Song1,2 1 SketchX, CVSSP, University of Surrey, United Kingdom 2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence 3 University of Edinburgh, United Kingdom CVPR 2021 (Poster)
  2. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] [1] Ha,

    D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  3. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  4. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation ❑ Autoregressive generation of points[2] [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  5. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation ❑ Autoregressive generation of points[2] … [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  6. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation ❑ Autoregressive generation of points[2] Δp1 [0,0] Δpi Δpi-1 … [0,0] p1 p2 [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  7. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation ❑ Autoregressive generation of points[2] Δp1 [0,0] Δpi Δpi-1 … [0,0] p1 p2 [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013)
  8. Traditional Sketch generation models • The state-of-the-art “SketchRNN”[1] ❑ Uses

    Polyline representation ❑ Autoregressive generation of points[2] Δp1 [0,0] Δpi Δpi-1 … [0,0] p1 p2 [1] Ha, D., Eck, D.: A neural representation of sketch drawings. In: ICLR (2018) [2] Graves, A.: Generating Sequences With Recurrent Neural Networks. (2013) [0,0] p1 p2 ~ ε Require polyline format for inference
  9. Parametric representation Represent sketches with Bézier curves • Shorter in

    length • Arbitrary resolution • Scalable, High quality t Degree ‘n’ No. of control points ‘n+1’
  10. Parametric representation Represent sketches with Bézier curves • Shorter in

    length • Arbitrary resolution • Scalable, High quality t Degree ‘n’ No. of control points ‘n+1’
  11. A stroke-by-stroke autoregressive model {Pi } {Pi-1 } {Pi }

    {Pi+1 } • Output one stroke (Bezier Curve) at a time [BezierSketch, ECCV, Das et al. 2020]
  12. A stroke-by-stroke autoregressive model {Pi } {Pi-1 } {Pi }

    {Pi+1 } • Output one stroke (Bezier Curve) at a time • Instantiate with predefine set of “t” values t [BezierSketch, ECCV, Das et al. 2020]
  13. A stroke-by-stroke autoregressive model {Pi } {Pi-1 } {Pi }

    {Pi+1 } • Output one stroke (Bezier Curve) at a time • Instantiate with predefine set of “t” values • Compare each with ground-truth strokes t [BezierSketch, ECCV, Das et al. 2020]
  14. A stroke-by-stroke autoregressive model {Pi } {Pi-1 } {Pi }

    {Pi+1 } • Output one stroke (Bezier Curve) at a time • Instantiate with predefine set of “t” values • Compare each with ground-truth strokes t Each stroke must be represented by Bezier curve of same degree [BezierSketch, ECCV, Das et al. 2020]
  15. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize
  16. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree t
  17. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree • One scalar “degree parameter” is used t r A maximum set of control points Value in [0, 1]
  18. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree • One scalar “degree parameter” is used t r A maximum set of control points Value in [0, 1]
  19. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree • One scalar “degree parameter” is used t r L( , ) A maximum set of control points Value in [0, 1]
  20. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree • One scalar “degree parameter” is used • Fully differentiable w.r.t downstream loss t r L( , ) A maximum set of control points Value in [0, 1]
  21. Variable Degree Bezier Curve (Idea) • Degree of Bezier curve

    is a discrete quantity, difficult to optimize • We contribute a Bezier curve formulation with variable degree • One scalar “degree parameter” is used • Fully differentiable w.r.t downstream loss t r L( , ) A maximum set of control points Notice how degree changing while optimizing Value in [0, 1]
  22. Variable Degree Bezier Curve (Formulation) • The Bezier curve formula

    has three components (Time matrix, Degree matrix and Control Points)
  23. Variable Degree Bezier Curve (Formulation) • The Bezier curve formula

    has three components (Time matrix, Degree matrix and Control Points) • Compute “Degree selector” R(r) Refer to the paper for details
  24. Variable Degree Bezier Curve (Formulation) • The Bezier curve formula

    has three components (Time matrix, Degree matrix and Control Points) • Compute “Degree selector” R(r) • Augmented each component Refer to the paper for details
  25. Variable degree stroke decoder {Pi }, r {Pi+1 }, r

    • Outputs an extra “degree parameter” at each timestep
  26. Variable degree stroke decoder {Pi }, r {Pi+1 }, r

    t • Outputs an extra “degree parameter” at each timestep • Instantiates using Variable degree Bezier formulation
  27. Variable degree stroke decoder {Pi }, r {Pi+1 }, r

    t • Outputs an extra “degree parameter” at each timestep • Instantiates using Variable degree Bezier formulation • Compute loss and backpropagate to train
  28. Variable degree stroke decoder {Pi }, r {Pi+1 }, r

    t • Outputs an extra “degree parameter” at each timestep • Instantiates using Variable degree Bezier formulation • Compute loss and backpropagate to train A point-cloud based encoder • We used set-transformer to encode sketch as point cloud ε Full Cloud2Curve Model
  29. Variable degree stroke decoder {Pi }, r {Pi+1 }, r

    t • Outputs an extra “degree parameter” at each timestep • Instantiates using Variable degree Bezier formulation • Compute loss and backpropagate to train A point-cloud based encoder • We used set-transformer to encode sketch as point cloud • Possible to infer from non-sequential modalities (raster etc) ε Full Cloud2Curve Model
  30. Loss functions & Regularization • Set-based loss: Treat strokes as

    point-cloud ❑ Sliced Wasserstein Distance (SWD)[1] Predicted Bezier Curve (P) Ground-Truth stroke (s)
  31. Loss functions & Regularization • Set-based loss: Treat strokes as

    point-cloud ❑ Sliced Wasserstein Distance (SWD)[1] • Sequential loss: Treat strokes as sequences ❑ Optional. Helps the model fit better. ❑ Standard point-to-point MSE loss Predicted Bezier Curve (P) Ground-Truth stroke (s)
  32. Loss functions & Regularization • Set-based loss: Treat strokes as

    point-cloud ❑ Sliced Wasserstein Distance (SWD)[1] • Sequential loss: Treat strokes as sequences ❑ Optional. Helps the model fit better. ❑ Standard point-to-point MSE loss Predicted Bezier Curve (P) Ground-Truth stroke (s) • Degree regularizer: ❑ Enforcing the degree to be as low as possible
  33. Loss functions & Regularization • Set-based loss: Treat strokes as

    point-cloud ❑ Sliced Wasserstein Distance (SWD)[1] • Sequential loss: Treat strokes as sequences ❑ Optional. Helps the model fit better. ❑ Standard point-to-point MSE loss Predicted Bezier Curve (P) Ground-Truth stroke (s) • Degree regularizer: ❑ Enforcing the degree to be as low as possible High value of degree regularizer enforces same stroke to acquire less degree
  34. Experiments & Results (Qualitative) Given point-cloud based sketch, Cloud2Curve generates

    parametric samples (more results in the paper) Higher “degree regularizer” reduced overall complexity of strokes
  35. Experiments & Results (Quantitative) • Shows the length histogram of

    strokes • Shows the length histogram of sketches Stroke Histogram Sketch Histogram
  36. Experiments & Results (Quantitative) • Shows the length histogram of

    strokes • Shows the length histogram of sketches • Shows classification acc. of generated samples • Shows FID of generated samples Stroke Histogram Sketch Histogram Classification score FID score
  37. Experiments & Results (Quantitative) • Shows the length histogram of

    strokes • Shows the length histogram of sketches • Shows classification acc. of generated samples • Shows FID of generated samples • Vectorization Model • Same model without source of randomness • Measure classification acc. and Test loss Stroke Histogram Sketch Histogram Classification score FID score