Variational Methods for Virtual Soccer Game Replays

Transcript

1. Variational Methods for Virtual Soccer Game Replays Vicent Caselles1,2, Nicolas Papadakis2,4 Antonio Baeza2, Aurélie Bugeau2, Olivier D’Hondt2,Pau Gargallo2,3 Xavi Armangué3, Ignasi Rius3, Sergi Sagàs3 1Universitat Pompeu Fabra, 2Barcelona Media, 3Media Pro 4 Institut de Mathématiques de Bordeaux Mathematics for Imaging: the Legacy of Vicent Caselles SIAM IS May 12th 2014 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.)
2. Vicent Caselles and Image Processing What people knows: Active contours Inpainting Color enhancement Total Variation Cheeger Sets ... Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 1/40
3. Vicent Caselles and Industrial Research What people may ignore: BarcelonaMedia: a non-profit foundation dedicated to applied research and the transfer of knowledge and technology Since 2007, Vicent was also head of the image group in BarcelonaMedia He has dedicated a lot of his time to a lot of very applied projects Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 2/40
4. Virtual Soccer Game Replays Context Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 3/40
5. Virtual Soccer Game Replays Context FC Barcelona was the best soccer team in Europ (it is now Madrid) The company MediaPro wanted professional tools for soccer diffusion Objective: synthetize realistic novel views of soccer games from few real cameras Camera 1 Camera 2 Camera 3 Camera 4 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 4/40
6. Existing products Digital Air: matrix effet, 360◦ with 360 cameras... Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 5/40
7. Existing products BBC research, Liberovision: planar approximation of players Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 6/40
8. Vicent Caselles and Industrial Research Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 7/40
9. Overview The image approach: project i3media The 3D approach: project Fine A focus on variational transfer of colors Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 8/40
10. Overview The image approach: project i3media The 3D approach: project Fine A focus on variational transfer of colors Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 8/40
11. The image approach Pre-processing Color correction Camera calibration Player segmentation (Active Contours) Processing Depth estimation (Total Variation) Virtual image synthesis from a given point of view Post-processing Inpainting of occluded areas Temporal filtering Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
12. The image approach Pre-processing Color correction Camera calibration Player segmentation (Active Contours) Processing Depth estimation (Total Variation) Virtual image synthesis from a given point of view Post-processing Inpainting of occluded areas Temporal filtering Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
13. The image approach Pre-processing Color correction Camera calibration Player segmentation (Active Contours) Processing Depth estimation (Total Variation) Virtual image synthesis from a given point of view Post-processing Inpainting of occluded areas Temporal filtering Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 9/40
14. Pre-processing Color correction Raw images Corrected images with a linear transformation ⇒ Camera calibration Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 10/40
15. Pre-processing Color correction Raw images Corrected images with a linear transformation ⇒ Camera calibration Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 10/40
16. Pre-processing Player segmentation Background learning on each camera Background substraction Regularization of the masks (Active Contours) Camera 2 Camera 3 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 11/40
17. Processing Depth estimation Discretization of depth: multi-label problem Homographies between cameras for each depth plane from calibration Independant estimation on each camera: Convexification of the multi-label problem [Pock et al, 2008] Acceleration with a narrow band approach to deal with a large number of labels [Baeza, Caselles, Gargallo and Papadakis, 2010] Joint estimation of depth for all cameras: Visibility constraint using Graph Cuts [Kolmogorov et al., 2002] Acceleration with multi-resolution [Papadakis and Caselles, 2010] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
18. Processing Depth estimation Discretization of depth: multi-label problem Homographies between cameras for each depth plane from calibration Independant estimation on each camera: Convexification of the multi-label problem [Pock et al, 2008] Acceleration with a narrow band approach to deal with a large number of labels [Baeza, Caselles, Gargallo and Papadakis, 2010] Joint estimation of depth for all cameras: Visibility constraint using Graph Cuts [Kolmogorov et al., 2002] Acceleration with multi-resolution [Papadakis and Caselles, 2010] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
19. Processing Depth estimation Discretization of depth: multi-label problem Homographies between cameras for each depth plane from calibration Independant estimation on each camera: Convexification of the multi-label problem [Pock et al, 2008] Acceleration with a narrow band approach to deal with a large number of labels [Baeza, Caselles, Gargallo and Papadakis, 2010] Joint estimation of depth for all cameras: Visibility constraint using Graph Cuts [Kolmogorov et al., 2002] Acceleration with multi-resolution [Papadakis and Caselles, 2010] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 12/40
20. Processing Depth estimation with the given calibration Camera 2 Camera 3 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 13/40
21. Image synthesis from a given point of view Virtual Depth synthesis Transfer the depth map to a virtual camera Real Camera 2 Real Camera 3 Virtual Camera 1 Virtual Camera 2 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 14/40
22. Image synthesis from a given point of view Texture reconstruction by color projection Transfer the known texture using homographies Real Camera 2 Real Camera 3 Virtual Camera 1 Virtual Camera 2 Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 15/40
23. Post-processing Inpainting of occluded areas Estimation of the inpainting mask from the occluded areas and the virtual depth discontinuities Synthetic image Virtual depth map Areas to inpaint Exemplar-based Image inpainting [Bugeau, Bertamío, Caselles, Sapiro, 2010] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 16/40
24. Post-processing Inpainting of occluded areas Virtual image Mask Inpainted image Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 17/40
25. Post-processing Temporal filtering Estimation of the optical flow on the virtual sequence [Papadakis, Baeza, Gargallo and Caselles, 2010]: Discretization of velocity components, independant convexifications Temporal consistency: filtering of color over Lagrangian trajectories [Bugeau, Gargallo, D’Hondt, Hervieu, Papadakis, Caselles 2010; Facciolo, Sadek, Bugeau, Caselles, 2011] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 18/40
26. Illustration: Matrix effect Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 19/40
27. Illustration: moving virtual camera Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 20/40
28. The image approach Limitations A lot of post-processing were needed for each novel view synthesis The 3D estimation was not possible from the available calibration Calibration from the lines and circles available in the playground: Patent [Alvarez and Caselles, 2011] The direct 3D estimation of the scene could be done Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 21/40
29. The image approach Limitations A lot of post-processing were needed for each novel view synthesis The 3D estimation was not possible from the available calibration Calibration from the lines and circles available in the playground: Patent [Alvarez and Caselles, 2011] The direct 3D estimation of the scene could be done Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 21/40
30. Overview The image approach The 3D approach A focus on variational transfer of colors Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 22/40
31. The 3D approach Pre-processing Color correction Camera calibration Player segmentation (Active Contours) Processing Independant depth estimation with plane sweep [Zach et al. 2008] 3D reconstruction with Total Variation Texture reconstruction Inpainting of occluded areas Image synthesis of a given point of view Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 23/40
32. The 3D approach Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 24/40
33. The 3D approach Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 25/40
34. Other industrial applications: Urban reconstruction Stereo inpainting Logo detection and more to come in the next presentations Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 26/40
35. Overview The image approach The 3D approach A focus on variational transfer of colors Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 27/40
36. Variational transfer of colors Color equalization Soccer: a simple and fast parametric transformation of colors is sufficient Color transfer between images f0 and f1 are the color histograms of 2 given images Transfer the colors of one image to the other Find an intermediate color palette Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 28/40
37. Variational transfer of colors Color equalization Soccer: a simple and fast parametric transformation of colors is sufficient Color transfer between images f0 and f1 are the color histograms of 2 given images Transfer the colors of one image to the other Find an intermediate color palette Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 28/40
38. Optimal transport and Color transfer Optimal transport Literature: [Delon 2004, Pitié et al. 2007, Rabin et al. 2010...] The transport cost W(f0 , f1 ): shortest path to move f0 to f1 The mapping T between densities: f0 (T(x)) = f1 (x) Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 29/40
39. Optimal transport and Color transfer Grayscale images: Explicit computation Two grayscale images I0 and I1: Ω → [0; 255] Two gray level histograms f0 and f1: [0, 255] → [0; 1] Cumulative histograms: F0 (λ) = λ 0 f0 (t)dt = 1 |Ω| |{x ∈ Ω, s.t I0 (x) ≤ λ}| F is non-decreasing and can be inverted The L2 optimal transportation reads: W(f0 , f1 )2 = 1 0 ||F−1 0 (λ) − F−1 1 (λ)||2dλ Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 30/40
40. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
41. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
42. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
43. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
44. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
45. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
46. Optimal transport and color interpolation Can not be extended to higher dimensions f(x) with x ∈ Rd Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 31/40
47. Optimal transport in 3D Fast approximations of the Optimal transport problem can be considered [Pitié et al. 2007, Rabin et al. 2010...] [Rabin et al. 2010] Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 32/40
48. Variational transfer of colors Limitations Working on color space does not deal with local content of images: creation of artifacts Post-processing of the image may be nedded Extension to more than 2 images (possible with Wasserstein barycenters [Ferradans, Peyré, Rabin]) Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 33/40
49. Variational transfer of colors The approach (Joint work with Edoardo Provenzi) Represent the color distribution of an image I as a function of the pixels of the image F(I) Define a metric between color distributions of two images I and J: d(F(I), F(J)). Use this measure in a variational model where the unknown is the image itself Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 34/40
50. Variational transfer of colors A simple but nice idea The histogram of an image I is fI : [0; 255]N → [0; 1] The cumulative histogram for N=1 reads: FI (λ) = 1 |Ω| |{x ∈ Ω, s.t I(x) ≤ λ}| = 1 |Ω| x∈Ω χ[0;λ] (I(x)) F is non linear but can be differentiated with respect to I: ∂I FI = − 1 |Ω| x∈Ω δ(I(x) − λ) It can be extended to N > 1: FI (λ) = 1 |Ω| |{x ∈ Ω, s.t I1 (x) ≤ λ1 · · · IN (x) ≤ λN }| = 1 |Ω| x∈Ω N i=1 χ[0;λi] (Ii (x)) with λ = (λ1 , . . . , λN ) ∈ [0, 1]N : N-dimensional intensity level Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 35/40
51. Variational transfer of colors A simple but nice idea The histogram of an image I is fI : [0; 255]N → [0; 1] The cumulative histogram for N=1 reads: FI (λ) = 1 |Ω| |{x ∈ Ω, s.t I(x) ≤ λ}| = 1 |Ω| x∈Ω χ[0;λ] (I(x)) F is non linear but can be differentiated with respect to I: ∂I FI = − 1 |Ω| x∈Ω δ(I(x) − λ) It can be extended to N > 1: FI (λ) = 1 |Ω| |{x ∈ Ω, s.t I1 (x) ≤ λ1 · · · IN (x) ≤ λN }| = 1 |Ω| x∈Ω N i=1 χ[0;λi] (Ii (x)) with λ = (λ1 , . . . , λN ) ∈ [0, 1]N : N-dimensional intensity level Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 35/40
52. Variational transfer of colors Distance between cumulative histograms J is a reference image and I = I0 is the input image Distance between cumulative histograms to transfer the colors of J: E1 (I) = ||FI − FJ ||2 The differentation with respect to I gives: ∇I E1 (I) = − 1 |Ω| (FI (I(x)) − FJ (I(x))) Aditional terms to preserve the geometry of I: E2 (I) = ||I − I0 ||2 E3 (I) = ||∇I − ∇I0 ||2 E4 (I) = |∇I0 | − θ(I), ∇I0 with θ(I) = ∇I/||∇I|| ... Optimize the non convex functional n i=1 Ei (I) Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 36/40
53. Variational transfer of colors Distance between cumulative histograms J is a reference image and I = I0 is the input image Distance between cumulative histograms to transfer the colors of J: E1 (I) = ||FI − FJ ||2 The differentation with respect to I gives: ∇I E1 (I) = − 1 |Ω| (FI (I(x)) − FJ (I(x))) Aditional terms to preserve the geometry of I: E2 (I) = ||I − I0 ||2 E3 (I) = ||∇I − ∇I0 ||2 E4 (I) = |∇I0 | − θ(I), ∇I0 with θ(I) = ∇I/||∇I|| ... Optimize the non convex functional n i=1 Ei (I) Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 36/40
54. Illustration Reference image Original image [Pitié et al, 2007] Our method Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 37/40
55. Extension to the equalization of several images Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 38/40
56. Extension to the equalization of several images Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 38/40
57. Extension to the equalization of several images Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
58. Extension to the equalization of several images Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
59. Extension to the equalization of several images Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 39/40
60. Variational transfer of colors Limitations Make obvious the jpeg blocks created by compression (unless over regularization) Discretization of the color space Conclusion Nice way to start with the concepts of optimal transport More sophisticated models: Julien Rabin (Wednesday, MS09) Vicent: be interested in every details Mathematics ⇔ Applications Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40
61. Variational transfer of colors Limitations Make obvious the jpeg blocks created by compression (unless over regularization) Discretization of the color space Conclusion Nice way to start with the concepts of optimal transport More sophisticated models: Julien Rabin (Wednesday, MS09) Vicent: be interested in every details Mathematics ⇔ Applications Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40
62. Variational transfer of colors Limitations Make obvious the jpeg blocks created by compression (unless over regularization) Discretization of the color space Conclusion Nice way to start with the concepts of optimal transport More sophisticated models: Julien Rabin (Wednesday, MS09) Vicent: be interested in every details Mathematics ⇔ Applications Virtual Soccer Game Replays (V. Caselles, N. Papadakis et al.) 40/40