Slide 1
Slide 1 text
BLENDING TEXTURE FEATURES FROM
MULTIPLE REFERENCE IMAGES FOR STYLE TRANSFER
Hikaru Ikuta*1,*2, Keisuke Ogaki*2, Yuri Odagiri*2
*1The University of Tokyo *2Dwango, Co. ltd.
Slide 2
Slide 2 text
OUR SYSTEM
“Watercolor”
Objective: Obtain an image painted in the texture of a given set of images
Input: Content Image and Texture Image Set
Tested with a novel dataset “nico-illust” containing 500,000 digital paintings
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
Slide 3
Slide 3 text
Original Video
Our Results
The input style image set (50 images)
Slide 4
Slide 4 text
MOTIVATION
Gatys, et al. [1]
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Slide 5
Slide 5 text
MOTIVATION
Gatys, et al. [1]
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
The color depends heavily on the input texture image
The color correspondence of the original image is often lost
Uses only one image, so…
Slide 6
Slide 6 text
MOTIVATION
Gatys, et al. [1]
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Is this really a “style” transfer?
The color depends heavily on the input texture image
The color correspondence of the original image is often lost
Uses only one image, so…
Slide 7
Slide 7 text
MOTIVATION
Gatys, et al. [1]
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Is this really a “style” transfer?
The color depends heavily on the input texture image
The color correspondence of the original image is often lost
Uses only one image, so…
The color of one artwork The “style”
6=
Slide 8
Slide 8 text
MOTIVATION
Gatys, et al. [1]
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Is this really a “style” transfer?
The color depends heavily on the input texture image
The color correspondence of the original image is often lost
Uses only one image, so…
Slide 9
Slide 9 text
MOTIVATION
Gatys, et al. [1]
Our Method
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Slide 10
Slide 10 text
MOTIVATION
Gatys, et al. [1]
Our Method
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
$POUFOU*NBHF 5FYUVSF*NBHF4FU 0VUQVU*NBHF
[1] Gatys, L. A., et al. 2016. Image style transfer using convolutional neural networks. CVPR
Our method…
Preserves the original color, while extracting the texture of the
texture image set
Therefore, it can transfer the same style onto different images
Slide 11
Slide 11 text
THE METHOD
Slide 12
Slide 12 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
5FYUVSF*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
5FYUVSF
'FBUVSF
SYSTEM OVERVIEW (FORMER)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 13
Slide 13 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
5FYUVSF*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
5FYUVSF
'FBUVSF
The output image tries to be close to the:
- Content image in the “Content Feature” space
- Texture image in the “Texture Feature” space
SYSTEM OVERVIEW (FORMER)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 14
Slide 14 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
5FYUVSF*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
5FYUVSF
'FBUVSF
SYSTEM OVERVIEW (FORMER)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 15
Slide 15 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
5FYUVSF*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
5FYUVSF
'FBUVSF
SYSTEM OVERVIEW (FORMER)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
F
G
$POUFOU
'FBUVSF
F
G
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
G
G
(Image of arbitrary size)
Rany⇥any RN
texture
(Fixed size vector)
The Texture Feature function :
G
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
2
6
6
6
6
6
4
0.5
7.7
2.1
5.6
.
.
.
3
7
7
7
7
7
5
Slide 16
Slide 16 text
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Rany⇥any
RN
texture
Dimension:
Number of filters
in the CNN
squared
…
correlation
matrix
Middle layer channels of
the CNN
Vectorize
[ ]
Slide 17
Slide 17 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
5FYUVSF*NBHF
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
G
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
5FYUVSF
'FBUVSF
SYSTEM OVERVIEW (FORMER)
Joint
Distance
Minimization
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 18
Slide 18 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
Novelty of our system
SYSTEM OVERVIEW (PROPOSED)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 19
Slide 19 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
Novelty of our system
Generate a better texture feature that
- Preserves the colors of the content
- Represents the author/genre’s “style”
SYSTEM OVERVIEW (PROPOSED)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 20
Slide 20 text
Neural Style Transfer is a joint distance
minimization problem
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
Novelty of our system
Generate a better texture feature that
- Preserves the colors of the content
- Represents the author/genre’s “style”
How? We use one key observation…
SYSTEM OVERVIEW (PROPOSED)
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 21
Slide 21 text
KEY OBSERVATION
Assume that: a concatenated image of some style belongs to the same style
Slide 22
Slide 22 text
KEY OBSERVATION
G ( )
G ( )
( )
Assume that: a concatenated image of some style belongs to the same style
Slide 23
Slide 23 text
KEY OBSERVATION
G ( )
G ( )
( ) 1
3
G
G ( )
G ( ) 1
3
G
G ( )
G ( )
1
3
G
G ( )
G ( ) + +
⇡
G ( )
It is the linear combination of the texture features of each image
Approximation is due to padding at boundaries in the CNN
Derived from the properties of : Proof shown in paper
Assume that: a concatenated image of some style belongs to the same style
Slide 24
Slide 24 text
KEY OBSERVATION
G ( )
G ( )
( ) 1
3
G
G ( )
G ( ) 1
3
G
G ( )
G ( )
1
3
G
G ( )
G ( ) + +
⇡
G ( )
It is the linear combination of the texture features of each image
Approximation is due to padding at boundaries in the CNN
Derived from the properties of : Proof shown in paper
Assume that: a concatenated image of some style belongs to the same style
Linear combinations of texture features from a certain style,
is again a texture feature of that style
Slide 25
Slide 25 text
KEY OBSERVATION
G ( )
G ( )
( ) 1
3
G
G ( )
G ( ) 1
3
G
G ( )
G ( )
1
3
G
G ( )
G ( ) + +
⇡
G ( )
It is the linear combination of the texture features of each image
Approximation is due to padding at boundaries in the CNN
Derived from the properties of : Proof shown in paper
Assume that: a concatenated image of some style belongs to the same style
Linear combinations of texture features from a certain style,
is again a texture feature of that style
Let us understand it geometrically…
Slide 26
Slide 26 text
BLENDING IN THE “STYLE SPACE”
Slide 27
Slide 27 text
Dimensions
in the
Texture
Feature .
G
BLENDING IN THE “STYLE SPACE”
Slide 28
Slide 28 text
㻌
㲍
㻌
㲍
㻌
㲍
BLENDING IN THE “STYLE SPACE”
Slide 29
Slide 29 text
㻌
㲍
㻌
㲍
㻌
㲍
- Weights are positive and sum up to one
- Number of images < Number of feature dimensions
Represents a simplex
BLENDING IN THE “STYLE SPACE”
Slide 30
Slide 30 text
㻌
㲍
㻌
㲍
㻌
㲍
BLENDING IN THE “STYLE SPACE”
Slide 31
Slide 31 text
㻌
㲍
㻌
㲍
㻌
㲍
The Watercolor “Style Space”
BLENDING IN THE “STYLE SPACE”
Slide 32
Slide 32 text
㻌
㲍
㻌
㲍
㻌
㲍
Content Image (not watercolor)
BLENDING IN THE “STYLE SPACE”
Slide 33
Slide 33 text
㻌
㲍
㻌
㲍
㻌
㲍
Our method: find the closest point
within the watercolor style space
BLENDING IN THE “STYLE SPACE”
Slide 34
Slide 34 text
㻌
㲍
㻌
㲍
㻌
㲍
arg min
r
X
l,i,j
Gl
ij
(I
content
)
K
X
k
r
k
Gl
ij
(I
k
)
!2
s.t.
K
X
k
r
k
= 1, 0 r
k
1 (k = 1, · · · , K)
Find the optimal weights rk
BLENDING IN THE “STYLE SPACE”
Slide 35
Slide 35 text
㻌
㲍
㻌
㲍
㻌
㲍
arg min
r
X
l,i,j
Gl
ij
(I
content
)
K
X
k
r
k
Gl
ij
(I
k
)
!2
s.t.
K
X
k
r
k
= 1, 0 r
k
1 (k = 1, · · · , K)
Find the optimal weights rk
˜
Gl
ij
=
K
X
k
rkGl
ij
(Ik)
Optimal Texture Feature
BLENDING IN THE “STYLE SPACE”
Slide 36
Slide 36 text
PIPELINE
Now that we have chosen a texture feature,
for the rest of the style transfer, we use Gatys, et al.[1]
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 37
Slide 37 text
PIPELINE
Now that we have chosen a texture feature,
for the rest of the style transfer, we use Gatys, et al.[1]
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 38
Slide 38 text
PIPELINE
Now that we have chosen a texture feature,
for the rest of the style transfer, we use Gatys, et al.[1]
$POUFOU*NBHF
Joint
Distance
Minimization
$POUFOU
'FBUVSF
0VUQVU*NBHF
F
2
6
6
6
6
6
4
3.1
4.1
5.9
2.6
.
.
.
3
7
7
7
7
7
5
Optimal
Blending 0QUJNBM
5FYUVSF
'FBUVSF
A large number of images
Annotations of the image’s style (“watercolor,” etc…)
We need a large image dataset with…
2
6
6
6
6
6
4
2.7
1.8
4.2
8.4
.
.
.
3
7
7
7
7
7
5
Slide 39
Slide 39 text
EXPERIMENTS
Slide 40
Slide 40 text
Novel dataset: “nico-illust”[4]
[4] https://nico-opendata.jp/en/
500,000 images, largely consisting of digital paintings,
with rich annotations: Tags, user comments, Number of favorites, etc.
(Tags include motif name, style name, etc.)
Slide 41
Slide 41 text
Author: Kariwo
(ID:33341043)
(50 images)
Author: Morin
(ID:10195867)
(4 images)
Author: Last Hunter
(ID:23607472)
(50 images)
Author: Niichi
(ID:13762409)
(50 images)
Tag: Watercolor
(50 images)
Tag: Pixel Art
(10 images)
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
Author: Kariwo
(ID:33341043)
(50 images)
Author: Last Hunter
(ID:23607472)
(50 images)
Slide 44
Slide 44 text
Author: Morin
(ID:10195867)
(4 images)
Author: Niichi
(ID:13762409)
(50 images)
Slide 45
Slide 45 text
Tag: Watercolor
(50 images)
Tag: Pixel Art
(10 images)
Slide 46
Slide 46 text
Tag: Watercolor
(50 images)
Tag: Pixel Art
(10 images)
Automatically captures the use of
perpendicular lines in the style of pixel art
Slide 47
Slide 47 text
Original Video “Watercolor” Style Transfer
Weights of each texture image
OUR SYSTEM
Slide 48
Slide 48 text
Texture image weights for each frame
Slide 49
Slide 49 text
CONCLUSION
Slide 50
Slide 50 text
CONCLUSION
We proved that the Texture Features proposed in Gatys, et al.[1]
can be linearly combined to construct a valid texture feature
We proposed a novel dataset “nico-illust” that contains digital
paintings
We proposed a neural style transfer method that preserves the
color of the original input image, by taking a collection of
images to represent a given style