Slide 1
Slide 1 text
Variational Methods for Virtual Soccer Game Replays
Vicent Caselles1,2, Nicolas Papadakis2,4
Antonio Baeza2, Aurélie Bugeau2, Olivier D’Hondt2,Pau Gargallo2,3
Xavi Armangué3, Ignasi Rius3, Sergi Sagàs3
1Universitat Pompeu Fabra, 2Barcelona Media, 3Media Pro
4 Institut de Mathématiques de Bordeaux
Mathematics for Imaging: the Legacy of Vicent Caselles
SIAM IS
May 12th 2014
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.)
Slide 2
Slide 2 text
Vicent Caselles and Image Processing
What people knows:
Active contours
Inpainting
Color enhancement
Total Variation
Cheeger Sets
...
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 1/40
Slide 3
Slide 3 text
Vicent Caselles and Industrial Research
What people may ignore:
BarcelonaMedia: a non-profit foundation dedicated to applied research and the
transfer of knowledge and technology
Since 2007, Vicent was also head of the image group in BarcelonaMedia
He has dedicated a lot of his time to a lot of very applied projects
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 2/40
Slide 4
Slide 4 text
Virtual Soccer Game Replays
Context
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 3/40
Slide 5
Slide 5 text
Virtual Soccer Game Replays
Context
FC Barcelona was the best soccer team in Europ (it is now Madrid)
The company MediaPro wanted professional tools for soccer diffusion
Objective: synthetize realistic novel views of soccer games from few real
cameras
Camera 1 Camera 2 Camera 3 Camera 4
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 4/40
Slide 6
Slide 6 text
Existing products
Digital Air: matrix effet, 360◦ with 360 cameras...
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 5/40
Slide 7
Slide 7 text
Existing products
BBC research, Liberovision: planar approximation of players
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 6/40
Slide 8
Slide 8 text
Vicent Caselles and Industrial Research
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 7/40
Slide 9
Slide 9 text
Overview
The image approach: project i3media
The 3D approach: project Fine
A focus on variational transfer of colors
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 8/40
Slide 10
Slide 10 text
Overview
The image approach: project i3media
The 3D approach: project Fine
A focus on variational transfer of colors
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 8/40
Slide 11
Slide 11 text
The image approach
Pre-processing
Color correction
Camera calibration
Player segmentation (Active Contours)
Processing
Depth estimation (Total Variation)
Virtual image synthesis from a given point of view
Post-processing
Inpainting of occluded areas
Temporal filtering
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
Slide 12
Slide 12 text
The image approach
Pre-processing
Color correction
Camera calibration
Player segmentation (Active Contours)
Processing
Depth estimation (Total Variation)
Virtual image synthesis from a given point of view
Post-processing
Inpainting of occluded areas
Temporal filtering
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
Slide 13
Slide 13 text
The image approach
Pre-processing
Color correction
Camera calibration
Player segmentation (Active Contours)
Processing
Depth estimation (Total Variation)
Virtual image synthesis from a given point of view
Post-processing
Inpainting of occluded areas
Temporal filtering
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
Slide 14
Slide 14 text
Pre-processing
Color correction
Raw images
Corrected images with a linear transformation
⇒ Camera calibration
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 10/40
Slide 15
Slide 15 text
Pre-processing
Color correction
Raw images
Corrected images with a linear transformation
⇒ Camera calibration
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 10/40
Slide 16
Slide 16 text
Pre-processing
Player segmentation
Background learning on each camera
Background substraction
Regularization of the masks (Active Contours)
Camera 2 Camera 3
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 11/40
Slide 17
Slide 17 text
Processing
Depth estimation
Discretization of depth: multi-label problem
Homographies between cameras for each depth plane from calibration
Independant estimation on each camera:
Convexification of the multi-label problem [Pock et al, 2008]
Acceleration with a narrow band approach to deal with a large number of labels
[Baeza, Caselles, Gargallo and Papadakis, 2010]
Joint estimation of depth for all cameras:
Visibility constraint using Graph Cuts [Kolmogorov et al., 2002]
Acceleration with multi-resolution [Papadakis and Caselles, 2010]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
Slide 18
Slide 18 text
Processing
Depth estimation
Discretization of depth: multi-label problem
Homographies between cameras for each depth plane from calibration
Independant estimation on each camera:
Convexification of the multi-label problem [Pock et al, 2008]
Acceleration with a narrow band approach to deal with a large number of labels
[Baeza, Caselles, Gargallo and Papadakis, 2010]
Joint estimation of depth for all cameras:
Visibility constraint using Graph Cuts [Kolmogorov et al., 2002]
Acceleration with multi-resolution [Papadakis and Caselles, 2010]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
Slide 19
Slide 19 text
Processing
Depth estimation
Discretization of depth: multi-label problem
Homographies between cameras for each depth plane from calibration
Independant estimation on each camera:
Convexification of the multi-label problem [Pock et al, 2008]
Acceleration with a narrow band approach to deal with a large number of labels
[Baeza, Caselles, Gargallo and Papadakis, 2010]
Joint estimation of depth for all cameras:
Visibility constraint using Graph Cuts [Kolmogorov et al., 2002]
Acceleration with multi-resolution [Papadakis and Caselles, 2010]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
Slide 20
Slide 20 text
Processing
Depth estimation with the given calibration
Camera 2 Camera 3
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 13/40
Slide 21
Slide 21 text
Image synthesis from a given point of view
Virtual Depth synthesis
Transfer the depth map to a virtual camera
Real Camera 2 Real Camera 3
Virtual Camera 1 Virtual Camera 2
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 14/40
Slide 22
Slide 22 text
Image synthesis from a given point of view
Texture reconstruction by color projection
Transfer the known texture using homographies
Real Camera 2 Real Camera 3
Virtual Camera 1 Virtual Camera 2
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 15/40
Slide 23
Slide 23 text
Post-processing
Inpainting of occluded areas
Estimation of the inpainting mask from the occluded areas and the virtual depth
discontinuities
Synthetic image Virtual depth map Areas to inpaint
Exemplar-based Image inpainting [Bugeau, Bertamío, Caselles, Sapiro, 2010]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 16/40
Slide 24
Slide 24 text
Post-processing
Inpainting of occluded areas
Virtual image Mask Inpainted image
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 17/40
Slide 25
Slide 25 text
Post-processing
Temporal filtering
Estimation of the optical flow on the virtual sequence [Papadakis, Baeza,
Gargallo and Caselles, 2010]:
Discretization of velocity components, independant convexifications
Temporal consistency: filtering of color over Lagrangian trajectories [Bugeau,
Gargallo, D’Hondt, Hervieu, Papadakis, Caselles 2010; Facciolo, Sadek,
Bugeau, Caselles, 2011]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 18/40
Slide 26
Slide 26 text
Illustration: Matrix effect
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 19/40
Slide 27
Slide 27 text
Illustration: moving virtual camera
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 20/40
Slide 28
Slide 28 text
The image approach
Limitations
A lot of post-processing were needed for each novel view synthesis
The 3D estimation was not possible from the available calibration
Calibration from the lines and circles available in the playground: Patent
[Alvarez and Caselles, 2011]
The direct 3D estimation of the scene could be done
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 21/40
Slide 29
Slide 29 text
The image approach
Limitations
A lot of post-processing were needed for each novel view synthesis
The 3D estimation was not possible from the available calibration
Calibration from the lines and circles available in the playground: Patent
[Alvarez and Caselles, 2011]
The direct 3D estimation of the scene could be done
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 21/40
Slide 30
Slide 30 text
Overview
The image approach
The 3D approach
A focus on variational transfer of colors
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 22/40
Slide 31
Slide 31 text
The 3D approach
Pre-processing
Color correction
Camera calibration
Player segmentation (Active Contours)
Processing
Independant depth estimation with plane sweep [Zach et al. 2008]
3D reconstruction with Total Variation
Texture reconstruction
Inpainting of occluded areas
Image synthesis of a given point of view
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 23/40
Slide 32
Slide 32 text
The 3D approach
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 24/40
Slide 33
Slide 33 text
The 3D approach
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 25/40
Slide 34
Slide 34 text
Other industrial applications:
Urban reconstruction
Stereo inpainting
Logo detection
and more to come in the next presentations
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 26/40
Slide 35
Slide 35 text
Overview
The image approach
The 3D approach
A focus on variational transfer of colors
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 27/40
Slide 36
Slide 36 text
Variational transfer of colors
Color equalization
Soccer: a simple and fast parametric transformation of colors is sufficient
Color transfer between images
f0 and f1 are the color histograms of 2 given images
Transfer the colors of one image to the other
Find an intermediate color palette
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 28/40
Slide 37
Slide 37 text
Variational transfer of colors
Color equalization
Soccer: a simple and fast parametric transformation of colors is sufficient
Color transfer between images
f0 and f1 are the color histograms of 2 given images
Transfer the colors of one image to the other
Find an intermediate color palette
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 28/40
Slide 38
Slide 38 text
Optimal transport and Color transfer
Optimal transport
Literature: [Delon 2004, Pitié et al. 2007, Rabin et al. 2010...]
The transport cost W(f0
, f1
): shortest path to move f0 to f1
The mapping T between densities: f0
(T(x)) = f1
(x)
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 29/40
Slide 39
Slide 39 text
Optimal transport and Color transfer
Grayscale images: Explicit computation
Two grayscale images I0 and I1: Ω → [0; 255]
Two gray level histograms f0 and f1: [0, 255] → [0; 1]
Cumulative histograms:
F0
(λ) =
λ
0
f0
(t)dt =
1
|Ω|
|{x ∈ Ω, s.t I0
(x) ≤ λ}|
F is non-decreasing and can be inverted
The L2 optimal transportation reads:
W(f0
, f1
)2 =
1
0
||F−1
0
(λ) − F−1
1
(λ)||2dλ
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 30/40
Slide 40
Slide 40 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 41
Slide 41 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 42
Slide 42 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 43
Slide 43 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 44
Slide 44 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 45
Slide 45 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 46
Slide 46 text
Optimal transport and color interpolation
Can not be extended to higher dimensions f(x) with x ∈ Rd
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
Slide 47
Slide 47 text
Optimal transport in 3D
Fast approximations of the Optimal transport problem can be considered [Pitié et al.
2007, Rabin et al. 2010...]
[Rabin et al. 2010]
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 32/40
Slide 48
Slide 48 text
Variational transfer of colors
Limitations
Working on color space does not deal with local content of images: creation of
artifacts
Post-processing of the image may be nedded
Extension to more than 2 images (possible with Wasserstein barycenters
[Ferradans, Peyré, Rabin])
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 33/40
Slide 49
Slide 49 text
Variational transfer of colors
The approach (Joint work with Edoardo Provenzi)
Represent the color distribution of an image I as a function of the pixels of the
image F(I)
Define a metric between color distributions of two images I and J:
d(F(I), F(J)).
Use this measure in a variational model where the unknown is the image itself
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 34/40
Slide 50
Slide 50 text
Variational transfer of colors
A simple but nice idea
The histogram of an image I is fI
: [0; 255]N → [0; 1]
The cumulative histogram for N=1 reads:
FI
(λ) =
1
|Ω|
|{x ∈ Ω, s.t I(x) ≤ λ}| =
1
|Ω|
x∈Ω
χ[0;λ]
(I(x))
F is non linear but can be differentiated with respect to I:
∂I
FI
= −
1
|Ω|
x∈Ω
δ(I(x) − λ)
It can be extended to N > 1:
FI
(λ) =
1
|Ω|
|{x ∈ Ω, s.t I1
(x) ≤ λ1
· · · IN
(x) ≤ λN
}| =
1
|Ω|
x∈Ω
N
i=1
χ[0;λi]
(Ii
(x))
with λ = (λ1
, . . . , λN
) ∈ [0, 1]N : N-dimensional intensity level
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 35/40
Slide 51
Slide 51 text
Variational transfer of colors
A simple but nice idea
The histogram of an image I is fI
: [0; 255]N → [0; 1]
The cumulative histogram for N=1 reads:
FI
(λ) =
1
|Ω|
|{x ∈ Ω, s.t I(x) ≤ λ}| =
1
|Ω|
x∈Ω
χ[0;λ]
(I(x))
F is non linear but can be differentiated with respect to I:
∂I
FI
= −
1
|Ω|
x∈Ω
δ(I(x) − λ)
It can be extended to N > 1:
FI
(λ) =
1
|Ω|
|{x ∈ Ω, s.t I1
(x) ≤ λ1
· · · IN
(x) ≤ λN
}| =
1
|Ω|
x∈Ω
N
i=1
χ[0;λi]
(Ii
(x))
with λ = (λ1
, . . . , λN
) ∈ [0, 1]N : N-dimensional intensity level
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 35/40
Slide 52
Slide 52 text
Variational transfer of colors
Distance between cumulative histograms
J is a reference image and I = I0 is the input image
Distance between cumulative histograms to transfer the colors of J:
E1
(I) = ||FI
− FJ
||2
The differentation with respect to I gives:
∇I
E1
(I) = −
1
|Ω|
(FI
(I(x)) − FJ
(I(x)))
Aditional terms to preserve the geometry of I:
E2
(I) = ||I − I0
||2
E3
(I) = ||∇I − ∇I0
||2
E4
(I) = |∇I0
| − θ(I), ∇I0
with θ(I) = ∇I/||∇I||
...
Optimize the non convex functional n
i=1
Ei
(I)
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 36/40
Slide 53
Slide 53 text
Variational transfer of colors
Distance between cumulative histograms
J is a reference image and I = I0 is the input image
Distance between cumulative histograms to transfer the colors of J:
E1
(I) = ||FI
− FJ
||2
The differentation with respect to I gives:
∇I
E1
(I) = −
1
|Ω|
(FI
(I(x)) − FJ
(I(x)))
Aditional terms to preserve the geometry of I:
E2
(I) = ||I − I0
||2
E3
(I) = ||∇I − ∇I0
||2
E4
(I) = |∇I0
| − θ(I), ∇I0
with θ(I) = ∇I/||∇I||
...
Optimize the non convex functional n
i=1
Ei
(I)
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 36/40
Slide 54
Slide 54 text
Illustration
Reference image Original image [Pitié et al, 2007] Our method
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 37/40
Slide 55
Slide 55 text
Extension to the equalization of several images
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 38/40
Slide 56
Slide 56 text
Extension to the equalization of several images
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 38/40
Slide 57
Slide 57 text
Extension to the equalization of several images
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
Slide 58
Slide 58 text
Extension to the equalization of several images
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
Slide 59
Slide 59 text
Extension to the equalization of several images
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
Slide 60
Slide 60 text
Variational transfer of colors
Limitations
Make obvious the jpeg blocks created by compression (unless over
regularization)
Discretization of the color space
Conclusion
Nice way to start with the concepts of optimal transport
More sophisticated models: Julien Rabin (Wednesday, MS09)
Vicent: be interested in every details
Mathematics ⇔ Applications
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40
Slide 61
Slide 61 text
Variational transfer of colors
Limitations
Make obvious the jpeg blocks created by compression (unless over
regularization)
Discretization of the color space
Conclusion
Nice way to start with the concepts of optimal transport
More sophisticated models: Julien Rabin (Wednesday, MS09)
Vicent: be interested in every details
Mathematics ⇔ Applications
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40
Slide 62
Slide 62 text
Variational transfer of colors
Limitations
Make obvious the jpeg blocks created by compression (unless over
regularization)
Discretization of the color space
Conclusion
Nice way to start with the concepts of optimal transport
More sophisticated models: Julien Rabin (Wednesday, MS09)
Vicent: be interested in every details
Mathematics ⇔ Applications
Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40