Blending Texture Features from Multiple Reference Images for Style Transfer - SIGGRAPH ASIA 2016 Technical Brief

本スライドはSIGGRAPH ASIA 2016 Technical Briefでの発表スライドになります。

本研究では、与えられた画像集合から画像集合中に共通する画風を学習し、その画風を別の写真などの画像へ転写する方法を提案します。本研究では、画風とは一つの作品によって特徴付けられるのではなく、同じ画風で描かれた複数の作品に共通する特徴によって定められると仮定することで、従来法に比べて元画像の色分布を保存しながら画風を転写しています。
画像集合の取得のため、本研究では多くの付加情報が付与された500,000枚以上のデジタルイラストを含む新たなデータセットを構築しました。

Transcript

1. BLENDING TEXTURE FEATURES FROM MULTIPLE REFERENCE IMAGES FOR STYLE TRANSFER Hikaru Ikuta*1,*2, Keisuke Ogaki*2, Yuri Odagiri*2 *1The University of Tokyo *2Dwango, Co. ltd.
2. OUR SYSTEM “Watercolor” Objective: Obtain an image painted in the texture of a given set of images Input: Content Image and Texture Image Set Tested with a novel dataset “nico-illust” containing 500,000 digital paintings $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF
3. Original Video Our Results The input style image set (50 images)
4. MOTIVATION Gatys, et al. [1] $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
5. MOTIVATION Gatys, et al. [1] $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR The color depends heavily on the input texture image The color correspondence of the original image is often lost Uses only one image, so…
6. MOTIVATION Gatys, et al. [1] $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR Is this really a “style” transfer? The color depends heavily on the input texture image The color correspondence of the original image is often lost Uses only one image, so…
7. MOTIVATION Gatys, et al. [1] $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR Is this really a “style” transfer? The color depends heavily on the input texture image The color correspondence of the original image is often lost Uses only one image, so… The color of one artwork The “style” 6=
8. MOTIVATION Gatys, et al. [1] $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR Is this really a “style” transfer? The color depends heavily on the input texture image The color correspondence of the original image is often lost Uses only one image, so…
9. MOTIVATION Gatys, et al. [1] Our Method $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF $POUFOU
   *NBHF 5FYUVSF
   *NBHF
   4FU 0VUQVU
   *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
10. MOTIVATION Gatys, et al. [1] Our Method $POUFOU
    *NBHF 5FYUVSF
    *NBHF
    4FU 0VUQVU
    *NBHF $POUFOU
    *NBHF 5FYUVSF
    *NBHF
    4FU 0VUQVU
    *NBHF [1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR Our method… Preserves the original color, while extracting the texture of the texture image set Therefore, it can transfer the same style onto different images
11. THE METHOD
12. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF 5FYUVSF
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 5FYUVSF
    'FBUVSF SYSTEM OVERVIEW (FORMER) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
13. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF 5FYUVSF
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 5FYUVSF
    'FBUVSF The output image tries to be close to the:
 - Content image in the “Content Feature” space
 - Texture image in the “Texture Feature” space SYSTEM OVERVIEW (FORMER) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
14. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF 5FYUVSF
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 5FYUVSF
    'FBUVSF SYSTEM OVERVIEW (FORMER) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
15. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF 5FYUVSF
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 5FYUVSF
    'FBUVSF SYSTEM OVERVIEW (FORMER) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5 F G $POUFOU
    'FBUVSF F G G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 G G (Image of arbitrary size) Rany⇥any RN texture (Fixed size vector) The Texture Feature function : G 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5 2 6 6 6 6 6 4 0.5 7.7 2.1 5.6 . . . 3 7 7 7 7 7 5
16. G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Rany⇥any RN texture Dimension: Number of filters in the CNN squared … correlation matrix Middle layer channels of the CNN Vectorize [ ]
17. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF 5FYUVSF
    *NBHF $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F G 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 5FYUVSF
    'FBUVSF SYSTEM OVERVIEW (FORMER) Joint Distance Minimization 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
18. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF Novelty of our system SYSTEM OVERVIEW (PROPOSED) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
19. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF Novelty of our system Generate a better texture feature that - Preserves the colors of the content - Represents the author/genre’s “style” SYSTEM OVERVIEW (PROPOSED) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
20. Neural Style Transfer is a joint distance minimization problem $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF Novelty of our system Generate a better texture feature that - Preserves the colors of the content - Represents the author/genre’s “style” How? We use one key observation… SYSTEM OVERVIEW (PROPOSED) 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
21. KEY OBSERVATION Assume that: a concatenated image of some style belongs to the same style
22. KEY OBSERVATION G ( ) G ( ) ( ) Assume that: a concatenated image of some style belongs to the same style
23. KEY OBSERVATION G ( ) G ( ) ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) + + ⇡ G ( ) It is the linear combination of the texture features of each image Approximation is due to padding at boundaries in the CNN Derived from the properties of : Proof shown in paper Assume that: a concatenated image of some style belongs to the same style
24. KEY OBSERVATION G ( ) G ( ) ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) + + ⇡ G ( ) It is the linear combination of the texture features of each image Approximation is due to padding at boundaries in the CNN Derived from the properties of : Proof shown in paper Assume that: a concatenated image of some style belongs to the same style Linear combinations of texture features from a certain style,
 is again a texture feature of that style
25. KEY OBSERVATION G ( ) G ( ) ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) 1 3 G G ( ) G ( ) + + ⇡ G ( ) It is the linear combination of the texture features of each image Approximation is due to padding at boundaries in the CNN Derived from the properties of : Proof shown in paper Assume that: a concatenated image of some style belongs to the same style Linear combinations of texture features from a certain style,
 is again a texture feature of that style Let us understand it geometrically…
26. BLENDING IN THE “STYLE SPACE”
27. Dimensions in the Texture Feature . G BLENDING IN THE “STYLE SPACE”
28. 㻌 㲍 㻌 㲍 㻌 㲍 BLENDING IN THE “STYLE SPACE”
29. 㻌 㲍 㻌 㲍 㻌 㲍 - Weights are positive and sum up to one - Number of images < Number of feature dimensions Represents a simplex BLENDING IN THE “STYLE SPACE”
30. 㻌 㲍 㻌 㲍 㻌 㲍 BLENDING IN THE “STYLE SPACE”
31. 㻌 㲍 㻌 㲍 㻌 㲍 The Watercolor “Style Space” BLENDING IN THE “STYLE SPACE”
32. 㻌 㲍 㻌 㲍 㻌 㲍 Content Image (not watercolor) BLENDING IN THE “STYLE SPACE”
33. 㻌 㲍 㻌 㲍 㻌 㲍 Our method: find the closest point
 within the watercolor style space BLENDING IN THE “STYLE SPACE”
34. 㻌 㲍 㻌 㲍 㻌 㲍 arg min r X l,i,j Gl ij (I content ) K X k r k Gl ij (I k ) !2 s.t. K X k r k = 1, 0  r k  1 (k = 1, · · · , K) Find the optimal weights rk BLENDING IN THE “STYLE SPACE”
35. 㻌 㲍 㻌 㲍 㻌 㲍 arg min r X l,i,j Gl ij (I content ) K X k r k Gl ij (I k ) !2 s.t. K X k r k = 1, 0  r k  1 (k = 1, · · · , K) Find the optimal weights rk ˜ Gl ij = K X k rkGl ij (Ik) Optimal Texture Feature BLENDING IN THE “STYLE SPACE”
36. PIPELINE Now that we have chosen a texture feature, for the rest of the style transfer, we use Gatys, et al.[1] $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
37. PIPELINE Now that we have chosen a texture feature, for the rest of the style transfer, we use Gatys, et al.[1] $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
38. PIPELINE Now that we have chosen a texture feature, for the rest of the style transfer, we use Gatys, et al.[1] $POUFOU
    *NBHF Joint Distance Minimization $POUFOU
    'FBUVSF 0VUQVU
    *NBHF F 2 6 6 6 6 6 4 3.1 4.1 5.9 2.6 . . . 3 7 7 7 7 7 5 Optimal Blending 0QUJNBM
    5FYUVSF
    'FBUVSF A large number of images Annotations of the image’s style (“watercolor,” etc…) We need a large image dataset with… 2 6 6 6 6 6 4 2.7 1.8 4.2 8.4 . . . 3 7 7 7 7 7 5
39. EXPERIMENTS
40. Novel dataset: “nico-illust”[4] [4] https://nico-opendata.jp/en/ 500,000 images, largely consisting of digital paintings, with rich annotations: Tags, user comments, Number of favorites, etc.
 (Tags include motif name, style name, etc.)
41. Author: Kariwo (ID:33341043) (50 images) Author: Morin (ID:10195867) (4 images) Author: Last Hunter (ID:23607472) (50 images) Author: Niichi (ID:13762409) (50 images) Tag: Watercolor (50 images) Tag: Pixel Art (10 images)
43. Author: Kariwo (ID:33341043) (50 images) Author: Last Hunter (ID:23607472) (50 images)
44. Author: Morin (ID:10195867) (4 images) Author: Niichi (ID:13762409) (50 images)
45. Tag: Watercolor (50 images) Tag: Pixel Art (10 images)
46. Tag: Watercolor (50 images) Tag: Pixel Art (10 images) Automatically captures the use of perpendicular lines in the style of pixel art
47. Original Video “Watercolor” Style Transfer Weights of each texture image OUR SYSTEM
48. Texture image weights for each frame
49. CONCLUSION
50. CONCLUSION We proved that the Texture Features proposed in Gatys, et al.[1] can be linearly combined to construct a valid texture feature We proposed a novel dataset “nico-illust” that contains digital paintings We proposed a neural style transfer method that preserves the color of the original input image, by taking a collection of images to represent a given style