Slide 1

Slide 1 text

. Inverse problems in signal and image processing and Bayesian inference framework: from basic to advanced Bayesian computation Ali Mohammad-Djafari Laboratoire des Signaux et Syst` emes (L2S) UMR8506 CNRS-CentraleSup´ elec-UNIV PARIS SUD SUPELEC, 91192 Gif-sur-Yvette, France http://lss.centralesupelec.fr Email: [email protected] http://djafari.free.fr http://publicationslist.org/djafari A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 1/77

Slide 2

Slide 2 text

Contents 1. Signal and Image Processing: Classical/Inverse problems approaches 2. Inverse problems examples Instrumentation Imaging systems to see outside of a body Imaging systems to see inside of a body Other imaging systems (Acoustics, Radar, SAR,...) 3. Analytical/Algebraic methods 4. Deterministic regularization methods and their limitations 5. Bayesian approach 6. Two main steps: Priors and Computational aspects 7. Case studies: Instrumentation, X ray Computed Tomography, Microwave imaging, Acoustic source localisation, Ultrasound imaging, Satellite image restoration, etc. A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 2/77

Slide 3

Slide 3 text

Signal and Image Processing: Classical/Inverse problems approach Classical: You have given a signal or an image, process it. Examples: Signal: Detect periodicities, changes, Model it for prediction, ... AR, MA, ARMA modeling,... Parameter estimation,... Image: Enhancement, Restoration, Segmentation, Contour detection, Compression, ... Model based or Inverse problem approach: What represent the observed signal or image? How they are related to the desired unknowns? Forward modelling / Inversion Examples: Deconvolution, Image restoration, Image reconstruction in Computed Tomography (CT), ... PCA, ICA / Blind source Separation, Compressed Sensing / L1 Regularization, Bayesian sparsity enforcing A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 3/77

Slide 4

Slide 4 text

Inverse Problems examples Example 1: Instrumentation: Measuring the temperature with a thermometer Deconvolution f (t) input of the instrument g(t) output of the instrument Example 2: Seeing outside of a body: Making an image using a camera, a microscope or a telescope: Image restoration f (x, y) real scene g(x, y) observed image Example 3: Seeing inside of a body: Computed Tomography usng X rays, US, Microwave, etc.: Image reconstruction f (x, y) a section of a real 3D body f (x, y, z) gφ (r) a line of observed radiographe gφ (r, z) Example 4: Seeing differently: MRI, Radar, SAR, Infrared, etc.: Fourier Synthesis f (x, y) a section of body or a scene g(u, v) partial data in the Fourier domain A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 4/77

Slide 5

Slide 5 text

Measuring variation of temperature with a therometer f (t) variation of temperature over time g(t) variation of length of the liquid in thermometer Forward model: Convolution g(t) = f (t ) h(t − t ) dt + (t) h(t): impulse response of the measurement system Inverse problem: Deconvolution Given the forward model H (impulse response h(t))) and a set of data g(ti ), i = 1, · · · , M find f (t) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 5/77

Slide 6

Slide 6 text

Measuring variation of temperature with a therometer Forward model: Convolution g(t) = f (t ) h(t − t ) dt + (t) f (t)−→ Thermometer h(t) −→ g(t) Inversion: Deconvolution f (t) g(t) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 6/77

Slide 7

Slide 7 text

Seeing outside of a body: Making an image with a camera, a microscope or a telescope f (x, y) real scene g(x, y) observed image Forward model: Convolution g(x, y) = f (x , y ) h(x − x , y − y ) dx dy + (x, y) h(x, y): Point Spread Function (PSF) of the imaging system Inverse problem: Image restoration Given the forward model H (PSF h(x, y))) and a set of data g(xi , yi ), i = 1, · · · , M find f (x, y) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 7/77

Slide 8

Slide 8 text

Making an image with an unfocused camera Forward model: 2D Convolution g(x, y) = f (x , y ) h(x − x , y − y ) dx dy + (x, y) f (x, y) E h(x, y) E + c (x, y) E g(x, y) Inversion: Image Deconvolution or Restoration ? ⇐= A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 8/77

Slide 9

Slide 9 text

Seeing inside of a body: Computed Tomography f (x, y) a section of a real 3D body f (x, y, z) gφ(r) a line of observed radiography gφ(r, z) Forward model: Line integrals or Radon Transform gφ(r) = Lr,φ f (x, y) dl + φ(r) = f (x, y) δ(r − x cos φ − y sin φ) dx dy + φ(r) Inverse problem: Image reconstruction Given the forward model H (Radon Transform) and a set of data gφi (r), i = 1, · · · , M find f (x, y) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 9/77

Slide 10

Slide 10 text

2D and 3D Computed Tomography 3D 2D gφ(r1, r2) = Lr1,r2,φ f (x, y, z) dl gφ(r) = Lr,φ f (x, y) dl Forward probelm: f (x, y) or f (x, y, z) −→ gφ(r) or gφ(r1, r2) Inverse problem: gφ(r) or gφ(r1, r2) −→ f (x, y) or f (x, y, z) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 10/77

Slide 11

Slide 11 text

Computed Tomography: Radon Transform Forward: f (x, y) −→ g(r, φ) Inverse: f (x, y) ←− g(r, φ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 11/77

Slide 12

Slide 12 text

Microwave or ultrasound imaging Measures: diffracted wave by the object g(ri ) Unknown quantity: f (r) = k2 0 (n2(r) − 1) Intermediate quantity : φ(r) g(ri ) = D Gm(ri , r )φ(r ) f (r ) dr , ri ∈ S φ(r) = φ0(r) + D Go(r, r )φ(r ) f (r ) dr , r ∈ D Born approximation (φ(r ) φ0(r )) ): g(ri ) = D Gm(ri , r )φ0(r ) f (r ) dr , ri ∈ S Discretization: g = GmF φ φ= φ0 + GoF φ −→    g = H(f) with F = diag(f) H(f) = GmF (I − GoF )−1φ0 r r r r r r r r r r r r r r r E D D i i e e 7 7 — — v v 3 3 φ0 (φ, f ) g A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 12/77

Slide 13

Slide 13 text

Fourier Synthesis in X ray Tomography g(r, φ) = f (x, y) δ(r − x cos φ − y sin φ) dx dy G(Ω, φ) = g(r, φ) exp [−jΩr] dr F(u, y) = f (x, y) exp [−jvx, yy] dx dy F(v, y) = G(Ω, φ) for u = Ω cos φ and v = Ω sin φ f (x, y) φ g(r, φ)–FT–G(Ω, φ) E x T y r d d d d d d d d d d d d s s ¡ ¡ ¡ d d d d d r r r φ E u T v Ω d d d d d d d d d d d d s α F(ωx , ωy ) φ A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 13/77

Slide 14

Slide 14 text

Fourier Synthesis in X ray tomography G(u, v) = f (x, y) exp [−j (ux + vy)] dx dy ? =⇒ Forward problem: Given f (x, y) compute G(u, v) Inverse problem: Given G(u, v) on those lines estimate f (x, y) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 14/77

Slide 15

Slide 15 text

Fourier Synthesis in Diffraction tomography A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 15/77

Slide 16

Slide 16 text

Fourier Synthesis in Diffraction tomography G(u, v) = f (x, y) exp [−j (ux + vy)] dx dy ? =⇒ Forward problem: Given f (x, y) compute G(u, v) Inverse problem : Given G(u, v) on those semi cercles estimate f (x, y) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 16/77

Slide 17

Slide 17 text

Fourier Synthesis in different imaging systems G(u, v) = f (x, y) exp [−j (ux + vy)] dx dy X ray Tomography Diffraction Eddy current SAR & Radar Forward problem: Given f (x, y) compute G(u, v) Inverse problem : Given G(u, v) on those algebraic lines, cercles or curves, estimate f (x, y) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 17/77

Slide 18

Slide 18 text

Linear inverse problems Deconvolution g(t) = f (τ)h(t − τ) dτ Image restoration g(x, y) = f (x , y )h(x − x , y − y ) dx dy Image reconstruction in X ray CT g(r, φ) = f (x, y)δ(r − x cos φ − y sin φ) dx dy Fourier synthesis g(u, v) = f (x, y) exp [−j(ux + vy)] dx dy Unified linear relation g(s) = f (r) h(s, r) dr A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 18/77

Slide 19

Slide 19 text

Linear Inverse Problems g(si ) = h(si , r) f (r) dr + (si ), i = 1, · · · , M f (r) is assumed to be well approximated by f (r) N j=1 fj φj (r) with {φj (r)} a basis or any other set of known functions g(si ) = gi N j=1 fj h(si , r) φj (r) dr, i = 1, · · · , M g = Hf + with Hij = h(si , r) φj (r) dr H is huge dimensional 1D: 103 × 103, 2D: 106 × 106, 3D: 109 × 109 Due to ill-posedness of the inverse problems, Least squares (LS) methods: f = arg minf {J(f)} with J(f) = g − Hf 2 do not give satisfactory result. Need for regularization methods: J(f) = g − Hf 2 + λ f 2 A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 19/77

Slide 20

Slide 20 text

Regularization theory Inverse problems = Ill posed problems −→ Need for prior information Functional space (Tikhonov): g = H(f ) + J(f ) = ||g − H(f )||2 2 + λ||Df ||2 2 Finite dimensional space (Philips & Towmey): g = Hf + J(f) = g − Hf 2 + λ f 2 • Minimum norme LS (MNLS): J(f) = ||g − H(f)||2 + λ||f||2 • Classical regularization: J(f) = ||g − H(f)||2 + λ||Df||2 • More general regularization: J(f) = Q(g − H(f)) + λΩ(Df) or J(f) = ∆1(g, H(f)) + λ∆2(Df, f0) Limitations: • Errors are considered implicitly white and Gaussian • Limited prior information on the solution • Lack of tools for the determination of the hyperparameters A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 20/77

Slide 21

Slide 21 text

Inversion: Probabilistic methods Taking account of errors and uncertainties −→ Probability theory Maximum Likelihood (ML) Minimum Inaccuracy (MI) Probability Distribution Matching (PDM) Maximum Entropy (ME) and Information Theory (IT) Bayesian Inference (Bayes) Advantages: Explicit account of the errors and noise A large class of priors via explicit or implicit modeling A coherent approach to combine information content of the data and priors Limitations: Practical implementation and cost of calculation A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 21/77

Slide 22

Slide 22 text

Bayesian estimation approach M : g = Hf + Observation model M + Hypothesis on the noise −→ p(g|f; M) = p (g − Hf) A priori information p(f|M) Bayes : p(f|g; M) = p(g|f; M) p(f|M) p(g|M) Link with regularization : Maximum A Posteriori (MAP) : f = arg max f {p(f|g)} = arg max f {p(g|f) p(f)} = arg min f {J(f) = − ln p(g|f) − ln p(f)} Regularization: f = arg min f {J(f) = Q(g, Hf) + λΩ(f)} with Q(g, Hf) = − ln p(g|f) and λΩ(f) = − ln p(f) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 22/77

Slide 23

Slide 23 text

Case of linear models and Gaussian priors g = Hf + Prior knowledge on the noise: ∼ N(0, σ2I) → p(g|f) ∝ exp − 1 2σ2 g − Hf 2 Prior knowledge on f: f ∼ N(0, σ2 f (D D)−1) → p(f) ∝ exp − 1 2σ2 f Df 2 A posteriori: p(f|g) ∝ exp − 1 2σ2 g − Hf 2 − 1 2σ2 f Df 2 MAP : f = arg maxf {p(f|g)} = arg minf {J(f)} with J(f) = g − Hf 2 + λ Df 2, λ = σ2 σ2 f Advantage : characterization of the solution p(f|g) = N(f, P ) with f = P H g, P = H H + λD D −1 A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 23/77

Slide 24

Slide 24 text

MAP estimation with other priors: f = arg min f {J(f)} with J(f) = g − Hf 2 + λΩ(f) Separable priors: Gaussian: p(fj ) ∝ exp −α|fj |2 −→ Ω(f) = α j |fj |2 Gamma: p(fj ) ∝ f α j exp [−βfj ] −→ Ω(f) = α j ln fj + βfj Beta: p(fj ) ∝ f α j (1 − fj )β −→ Ω(f) = α j ln fj + β j ln(1 − fj ) Generalized Gaussian: p(fj ) ∝ exp [−α|fj |p] , 1 < p < 2 −→ Ω(f) = α j |fj |p, Markovian models: p(fj |f) ∝ exp  −α i∈Nj φ(fj , fi )   −→ Ω(f) = α j i∈Nj φ(fj , fi ), A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 24/77

Slide 25

Slide 25 text

MAP estimation with markovian priors: f = arg min f {J(f)} with J(f) = g − Hf 2 + λΩ(f) Ω(f) = j φ(fj − fj−1) with φ(t) : Convex functions: |t|α, 1 + t2 − 1, log(cosh(t)), t2 |t| ≤ T 2T|t| − T2 |t| > T or Non convex functions: log(1 + t2), t2 1 + t2 , arctan(t2), t2 |t| ≤ T T2 |t| > T A great number of methods, optimization algorithms,... A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 25/77

Slide 26

Slide 26 text

Main advantages of the Bayesian approach MAP = Regularization Posterior mean ? Marginal MAP ? More information in the posterior law than only its mode or its mean Tools for estimating hyper parameters Tools for model selection More specific and specialized priors, particularly through the hidden variables and hierarchical models More computational tools: Expectation-Maximization for computing the maximum likelihood parameters MCMC for posterior exploration Variational Bayes for analytical computation of the posterior marginals ... A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 26/77

Slide 27

Slide 27 text

Bayesian Estimation: Simple priors Linear model: g = Hf + Gaussian case: p(g|f, θ1) = N(Hf, θ1I) p(f|θ2) = N(0, θ2I) −→ p(f|g, θ) = N(f, P ) with P = (H H + λI)−1, λ = θ1 θ2 f = P H g f = arg min f {J(f)} with J(f) = g − Hf 2 2 + λ f 2 2 Generalized Gaussian prior & MAP: f = arg min f {J(f)} with J(f) = g − Hf 2 2 + λ f β Double Exponential (β = 1): f = arg min f {J(f)} with J(f) = g − Hf 2 2 + λ f 1 A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 27/77

Slide 28

Slide 28 text

Full (Unsupervised) Bayesian approach M : g = Hf + Forward & errors model: −→ p(g|f, θ1; M) Prior models −→ p(f|θ2; M) Hyperparameters θ = (θ1, θ2) −→ p(θ|M) Bayes: −→ p(f, θ|g; M) = p(g|f,θ;M) p(f|θ;M) p(θ|M) p(g|M) Joint MAP: (f, θ) = arg max (f,θ) {p(f, θ|g; M)} Marginalization: p(f|g; M) = p(f, θ|g; M) dθ p(θ|g; M) = p(f, θ|g; M) df Posterior means: f = f p(f, θ|g; M) dθ df θ = θ p(f, θ|g; M) df dθ Evidence of the model: p(g|M) = p(g|f, θ; M)p(f|θ; M)p(θ|M) df dθ A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 28/77

Slide 29

Slide 29 text

Two main steps in the Bayesian approach Prior modeling Separable: Gaussian, Gamma, Sparsity enforcing: Generalized Gaussian, mixture of Gaussians, mixture of Gammas, ... Markovian: Gauss-Markov, GGM, ... Markovian with hidden variables (contours, region labels) Choice of the estimator and computational aspects MAP, Posterior mean, Marginal MAP MAP needs optimization algorithms Posterior mean needs integration methods Marginal MAP and Hyperparameter estimation need integration and optimization Approximations: Gaussian approximation (Laplace) Numerical exploration MCMC Variational Bayes (Separable approximation) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 29/77

Slide 30

Slide 30 text

Different prior models for signals and images: Separable Gaussian Generalized Gaussian p(fj ) ∝ exp −α|fj |2 p(fj ) ∝ exp [−α|fj |p] , 1 ≤ p ≤ 2 Gamma Beta p(fj ) ∝ f α j exp [−βfj ] p(fj ) ∝ f α j (1 − fj )β A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 30/77

Slide 31

Slide 31 text

Sparsity enforcing prior models Sparse signals: Direct sparsity Sparse signals: Sparsity in a Transform domain A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 31/77

Slide 32

Slide 32 text

Sparsity enforcing prior models Simple heavy tailed models: Generalized Gaussian, Double Exponential Symmetric Weibull, Symmetric Rayleigh Student-t, Cauchy Generalized hyperbolic Elastic net Hierarchical mixture models: Mixture of Gaussians Bernoulli-Gaussian Mixture of Gammas Bernoulli-Gamma Mixture of Dirichlet Bernoulli-Multinomial A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 32/77

Slide 33

Slide 33 text

Which images I am looking for? A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 33/77

Slide 34

Slide 34 text

Which image I am looking for? Gauss-Markov Generalized GM Piecewize Gaussian Mixture of GM A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 34/77

Slide 35

Slide 35 text

Different prior models for signals and images: Separable Simple Gaussian, Gamma, Generalized Gaussian p(f) ∝ exp   j φ(f j )   Simple Markovian models: Gauss-Markov, Generalized Gauss-Markov p(f) ∝ exp   j j∈N(i) φ(f j − f i )   Hierarchical models with hidden variables: Bernouilli-Gaussian, Gaussian-Gamma p(f|z) ∝ exp   j p(f j |zj )   and p(z) ∝ exp   j p(zj )   with different choices for p(f j |zj ) and p(zj ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 35/77

Slide 36

Slide 36 text

Hierarchical models and hidden variables Student-t model St(f |ν) ∝ exp − ν + 1 2 log 1 + f 2/ν Infinite Scaled Gaussian Mixture (ISGM) equivalence St(f |ν) ∝= ∞ 0 N(f |, 0, 1/z) G(z|α, β) dz, with α = β = ν/2              p(f|z) = j p(fj |zj ) = j N(fj |0, 1/zj ) ∝ exp −1 2 j zj f 2 j p(z|α, β) = j G(zj |α, β) ∝ j zj (α−1) exp [−βzj ] ∝ exp j (α − 1) ln zj − βzj p(f, z|α, β) ∝ exp −1 2 j zj f 2 j + (α − 1) ln zj − βzj A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 36/77

Slide 37

Slide 37 text

Gauss-Markov-Potts prior models for images f (r) z(r) c(r) = 1 − δ(z(r) − z(r )) p(f (r)|z(r) = k, mk , vk ) = N(mk , vk ) p(f (r)) = k P(z(r) = k) N(mk , vk ) Mixture of Gaussians Separable iid hidden variables: p(z) = r p(z(r)) Markovian hidden variables: p(z) Potts-Markov: p(z(r)|z(r ), r ∈ V(r)) ∝ exp  γ r ∈V(r) δ(z(r) − z(r ))   p(z) ∝ exp  γ r∈R r ∈V(r) δ(z(r) − z(r ))   A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 37/77

Slide 38

Slide 38 text

Four different cases To each pixel of the image is associated 2 variables f (r) and z(r) f|z Gaussian iid, z iid : Mixture of Gaussians f|z Gauss-Markov, z iid : Mixture of Gauss-Markov f|z Gaussian iid, z Potts-Markov : Mixture of Independent Gaussians (MIG with Hidden Potts) f|z Markov, z Potts-Markov : Mixture of Gauss-Markov (MGM with hidden Potts) f (r) z(r) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 38/77

Slide 39

Slide 39 text

Bayesian Computation and Algorithms Joint posterior probability law of all the unknowns f, z, θ p(f, z, θ|g) ∝ p(g|f, θ1) p(f|z, θ2) p(z|θ3) p(θ) Often, the expression of p(f, z, θ|g) is complex. Its optimization (for Joint MAP) or its marginalization or integration (for Marginal MAP or PM) is not easy Two main techniques: MCMC and Variational Bayesian Approximation (VBA) MCMC: Needs the expressions of the conditionals p(f|z, θ, g), p(z|f, θ, g), and p(θ|f, z, g) VBA: Approximate p(f, z, θ|g) by a separable one q(f, z, θ|g) = q1(f) q2(z) q3(θ) and do any computations with these separable ones. A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 39/77

Slide 40

Slide 40 text

Hierarchical models Simple case (1 layer): θ2 c f c H g θ1 c © g = Hf + p(f|g, θ) ∝ p(g|f, θ1) p(f|θ2) Objective: Infer on f MAP: f = arg maxf {p(f|g, θ)} Posterior Mean (PM): f = p(f|g, θ) df Unsupervised case (2 layers): β0 c θ2 c α0 c θ1 c f c H g © p(f, θ|g) ∝ p(g|f, θ1) p(f|θ2) p(θ) Objective: Infer on f, θ JMAP: (f, θ) = arg max(f,θ) {p(f, θ|g)} Marginalization: p(θ|g) = p(f, θ|g) df VBA: Approximate p(f, θ|g) by q1(f) q2(θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 40/77

Slide 41

Slide 41 text

Hierarchical models (3 layers) γ0 c θ3 c z c α0 c θ1 c β0 c θ2 d d ‚ f © c H g p(f, z, θ|g) ∝ p(g|f, θ1) p(f|z, θ2) p(z|θ3) p(θ) p(θ) = p(θ1|α0) p(θ2|β0) p(θ3|γ0) Objective: Infer on f, z, θ JMAP: (f, z, θ) = arg max(f,z,θ) {p(f, z, θ|g)} Marginalization: p(z, θ|g) = p(f, z, θ|g) df p(θ|g) = p(z, θ|g) dz or p(f|g) = p(f, z, θ|g) dz dθ VBA: Approximate p(f, z, θ|g) by q1(f) q2(z) q3(θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 41/77

Slide 42

Slide 42 text

MCMC based algorithm p(f, z, θ|g) ∝ p(g|f, z, θ1) p(f|z, θ2) p(z|θ3) p(θ) General scheme: f ∼ p(f|z, θ, g) −→ z ∼ p(z|f, θ, g) −→ θ ∼ (θ|f, z, g) Estimate f using p(f|z, θ, g) ∝ p(g|f, θ) p(f|z, θ) When Gaussian, can be done via optimisation of a quadratic criterion. Estimate z using p(z|f, θ, g) ∝ p(g|f, z, θ) p(z) Often needs sampling (hidden discrete variable) Estimate θ using p(θ|f, z, g) ∝ p(g|f, σ2I) p(f|z, (mk, vk)) p(θ) Use of Conjugate priors −→ analytical expressions. A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 42/77

Slide 43

Slide 43 text

Variational Bayesian Approximation Approximate p(f, θ|g) by q(f, θ|g) = q1(f|g) q2(θ|g) and then continue computations. Criterion KL(q(f, θ|g) : p(f, θ|g)) KL(q : p) = q ln q/p = q1q2 ln q1q2 p = q1 ln q1 + q2 ln q2 − q ln p = −H(q1) − H(q2)− < ln p >q Iterative algorithm q1 −→ q2 −→ q1 −→ q2, · · ·    q1(f) ∝ exp ln p(g, f, θ; M) q2(θ) q2(θ) ∝ exp ln p(g, f, θ; M) q1(f) p(f, θ|g) −→ Variational Bayesian Approximation −→ q1(f) −→ f −→ q2(θ) −→ θ A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 43/77

Slide 44

Slide 44 text

JMAP (f, θ) = arg max (f,θ) {p(p(f, θ|g)} Alternate optimization: θ = arg minθ {p(f, θ|g)} f = arg minf {p(f, θ|g)} Main drawbacks: Convergence Uncertainties in each step are not accounted for A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 44/77

Slide 45

Slide 45 text

Marginalization Marginal MAP: θ = arg maxθ {p(θ|g)} where p(θ|g) = p(f, θ|g) df ∝ p(g|θ) p(θ) and then f = arg maxf p(f|θ, g) or Posterior Mean: f = f p(f|θ, g) df Main drawback: Needs the expression of the Likelihood: p(g|θ) = p(g|f, θ1) p(f|θ2) df Not always analytically available −→ EM, SEM and GEM algorithms A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 45/77

Slide 46

Slide 46 text

EM and GEM algorithms EM and GEM Algorithms: f as hidden variable, g as incomplete data, (g, f) as complete data ln p(g|θ) incomplete data log-likelihood ln p(g, f|θ) complete data log-likelihood Iterative algorithm:    E-step: Q(θ, θ (k) ) = E p(f|g,θ (k) ) {ln p(g, f|θ)} M-step: θ (k) = arg maxθ Q(θ, θ (k−1) ) GEM (Bayesian) algorithm:    E-step: Q(θ, θ (k) ) = E p(f|g,θ (k) ) {ln p(g, f|θ) + ln p(θ)} M-step: θ (k) = arg maxθ Q(θ, θ (k−1) ) p(f, θ|g) −→ EM, GEM −→ θ −→ p(f|θ, g) −→ f A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 46/77

Slide 47

Slide 47 text

JMAP, Marginalization, VBA JMAP: p(f, θ|g) optimization −→ f −→ θ Marginalization p(f, θ|g) Joint Posterior −→ p(θ|g) Marginalize over f −→ θ −→ p(f|θ, g) −→ f Variational Bayesian Approximation p(f, θ|g) −→ Variational Bayesian Approximation −→ q1(f) −→ f −→ q2(θ) −→ θ A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 47/77

Slide 48

Slide 48 text

Variational Bayesian Approximation Approximate p(f, θ|g) by q(f, θ) = q1(f) q2(θ) and then use them for any inferences on f and θ respectively. Criterion KL(q(f, θ|g) : p(f, θ|g)) KL(q : p) = q ln q p = q1q2 ln q1q2 p Iterative algorithm q1 −→ q2 −→ q1 −→ q2, · · ·    q1(f) ∝ exp ln p(g, f, θ; M) q2(θ) q2(θ) ∝ exp ln p(g, f, θ; M) q1(f) p(f, θ|g) −→ Variational Bayesian Approximation −→ q1(f) −→ f −→ q2(θ) −→ θ A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 48/77

Slide 49

Slide 49 text

Variational Bayesian Approximation p(f, θ|g, M) = p(f, θ, g|, M) p(g|M) p(f, θ, g|M) = p(g|f, θ, M) p(f|θ, M) p(θ|M) KL(q : p) = q(f, θ) ln p(f, θ|g; M) q(f, θ) df dθ p(g|M) = q(f, θ) p(g, f, θ|M) q(f, θ) df dθ ≥ q(f, θ) ln p(g, f, θ|M) q(f, θ) df dθ. Free energy: F(q) = q(f, θ) ln p(g, f, θ|M) q(f, θ) df dθ Evidence of the model M: p(g|M) = F(q) + KL(q : p) Minimizing KL(q : p) = Maximaizing F(q) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 49/77

Slide 50

Slide 50 text

VBA: Separable Approximation p(g|M) = F(q) + KL(q : p) q(f, θ) = q1(f) q2(θ) Minimizing KL(q : p) = Maximizing F(q) (q1, q2) = arg min (q1,q2) {KL(q1q2 : p)} = arg max (q1,q2) {F(q1q2)} KL(q1q2 : p) is convexe wrt q1 when q2 is fixed and vise versa: q1 = arg minq1 {KL(q1q2 : p)} = arg maxq1 {F(q1q2)} q2 = arg minq2 {KL(q1q2 : p)} = arg maxq2 {F(q1q2)}    q1(f) ∝ exp ln p(g, f, θ; M) q2(θ) q2(θ) ∝ exp ln p(g, f, θ; M) q1(f) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 50/77

Slide 51

Slide 51 text

BVA: Choice of family of laws q1 and q2 Case 1 : −→ Joint MAP q1(f|f) = δ(f − f) q2(θ|θ) = δ(θ − θ) −→    f= arg maxf p(f, θ|g; M) θ= arg maxθ p(f, θ|g; M) Case 2 : −→ EM q1(f) ∝ p(f|θ, g) q2(θ|θ) = δ(θ − θ) −→    Q(θ, θ)= ln p(f, θ|g; M) q1(f|θ) θ = arg maxθ Q(θ, θ) Appropriate choice for inverse problems q1(f) ∝ p(f|θ, g; M) q2(θ) ∝ p(θ|f, g; M) −→ Accounts for the uncertainties of θ for f and vise versa. Exponential families, Conjugate priors A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 51/77

Slide 52

Slide 52 text

JMAP, EM and VBA JMAP Alternate optimization Algorithm: θ(0) −→ θ−→ f = arg maxf p(f, θ|g) −→f −→ f ↑ ↓ θ ←− θ←− θ = arg maxθ p(f, θ|g) ←− f EM: θ(0) −→ θ−→ q1 (f) = p(f|θ, g) −→q1 (f) −→ f ↑ ↓ θ ←− θ←− Q(θ, θ) = ln p(f, θ|g) q1(f) θ = arg maxθ Q(θ, θ) ←− q1 (f) VBA: θ(0) −→ q2 (θ)−→ q1 (f) ∝ exp ln p(f, θ|g) q2(θ) −→q1 (f) −→ f ↑ ↓ θ ←− q2 (θ)←− q2 (θ) ∝ exp ln p(f, θ|g) q1(f) ←−q1 (f) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 52/77

Slide 53

Slide 53 text

Computed Tomography: Discretization fN f1 fj gi Hij           g(r, φ) = L f (x, y) dl f (x, y) = j fj bj (x, y) bj (x, y) = 1 if (x, y) ∈ pixel j 0 else gi = N j=1 Hij fj + i g = Hf + f (x, y) E x T y ¡ ¡ d d d r r Case study: Reconstruction from 2 projections g1(x) = f (x, y) dy, g2(y) = f (x, y) dx Very ill-posed inverse problem f (x, y) = g1(x) g2(y) Ω(x, y) Ω(x, y) is a Copula: Ω(x, y) dx = 1 Ω(x, y) dy = 1 A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 53/77

Slide 54

Slide 54 text

Simple example 1 3 2 4 4 6 3 7 ? ? ? ? 4 6 3 7 f1 f3 f2 f4 g3 g4 g1 g2 1 -1 -1 1 0 0 0 0 -1 1 1 -1 0 0 0 0     g1 g2 g3 g4     =     1 1 0 0 0 0 1 1 1 0 1 0 0 1 0 1         f1 f2 f3 f4     f1 f4 f7 f2 f5 f8 f3 f6 f9 g4 g5 g6 g1 g2 g3 Hf = g −→ f = H−1g if H invertible. H is rank deficient: rank(H) = 3 Problem has infinite number of solutions. How to find all those solutions ? Which one is the good one? Needs prior information. To find an unique solution, one needs either more data or prior information. A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 54/77

Slide 55

Slide 55 text

Application in CT: Reconstruction from 2 projections g|f f|z z c g = Hf + iid Gaussian iid q(r) ∈ {0, 1} g|f ∼ N(Hf, σ2I) or or 1 − δ(z(r) − z(r )) Gaussian Gauss-Markov Potts binary p(f, z, θ|g) ∝ p(g|f, θ1) p(f|z, θ2) p(z|θ3) p(θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 55/77

Slide 56

Slide 56 text

Proposed algorithms p(f, z, θ|g) ∝ p(g|f, θ1) p(f|z, θ2) p(z|θ3) p(θ) • MCMC based general scheme: f ∼ p(f|z, θ, g) −→ z ∼ p(z|f, θ, g) −→ θ ∼ (θ|f, z, g) Iterative algorithme: Estimate f using p(f|z, θ, g) ∝ p(g|f, θ) p(f|z, θ) Needs optimisation of a quadratic criterion. Estimate z using p(z|f, θ, g) ∝ p(g|f, z, θ) p(z) Needs sampling of a Potts Markov field. Estimate θ using p(θ|f, z, g) ∝ p(g|f, σ2I) p(f|z, (mk, vk)) p(θ) Conjugate priors −→ analytical expressions. • Variational Bayesian Approximation Approximate p(f, z, θ|g) by q1(f) q2(z) q3(θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 56/77

Slide 57

Slide 57 text

Results Original Backprojection Filtered BP LS Gauss-Markov+pos GM+Line process GM+Label process c z c A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 57/77

Slide 58

Slide 58 text

Application in Acoustic source localization (Ning Chu et al.) x (m) y (m) −1.2 −1 −0.8 −0.6 −0.4 −0.2 0 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 −10 −8 −6 −4 −2 0 2 x (m) y (m) −1.2 −1 −0.8 −0.6 −0.4 −0.2 0 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 −4 −2 0 2 4 6 8 10 Source powers Beamforming powers x (m) y (m) −1.2 −1 −0.8 −0.6 −0.4 −0.2 0 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 −12 −10 −8 −6 −4 −2 0 2 x (m) y (m) −1.2 −1 −0.8 −0.6 −0.4 −0.2 0 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 −12 −10 −8 −6 −4 −2 0 2 Bayesian MAP inversion Proposed VBA inversion A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 58/77

Slide 59

Slide 59 text

Application in Acoustic source localization (Ning Chu et al.) Wind tunnel Beamforming DAMAS Proposed VBA inference A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 59/77

Slide 60

Slide 60 text

Application in Microwave imaging g(ω) = f (r) exp [−j(ω.r)] dr + (ω) g(u, v) = f (x, y) exp [−j(ux + vy)] dx dy + (u, v) g = Hf + f (x, y) g(u, v) f IFT f Proposed method A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 60/77

Slide 61

Slide 61 text

Microwave Imaging for Breast Cancer detection (L. Gharsalli et al.) 2 cm ( D₄ ) ( D₃ ) 12.2 cm receivers 10 cm 7.5 cm D₁ D breast S tumor skin ( D₂ ) source A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 61/77

Slide 62

Slide 62 text

Microwave Imaging for Breast Cancer detection CSI: Contrast Source Inversion, VBA: Variational Bayesian Approach, MGI: Independent Gaussian mixture, MGM: Gauss-Markov mixture A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 62/77

Slide 63

Slide 63 text

Images fusion and joint segmentation (with O. F´ eron)    gi (r) = fi (r) + i (r) p(fi (r)|z(r) = k) = N(mi k , σ2 i k ) p(f|z) = i p(fi |z) g1 g2 −→ f1 f2 z A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 63/77

Slide 64

Slide 64 text

Data fusion in medical imaging (with O. F´ eron)    gi (r) = fi (r) + i (r) p(fi (r)|z(r) = k) = N(mi k , σ2 i k ) p(f|z) = i p(fi |z) g1 g2 −→ f1 f2 z A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 64/77

Slide 65

Slide 65 text

Joint segmentation of hyper-spectral images (with N. Bali & A. Mohammadpour)        gi (r) = fi (r) + i (r) p(fi (r)|z(r) = k) = N(mi k , σ2 i k ), k = 1, · · · , K p(f|z) = i p(fi |z) mi k follow a Markovian model along the index i A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 65/77

Slide 66

Slide 66 text

Segmentation of a video sequence of images (with P. Brault)        gi (r) = fi (r) + i (r) p(fi (r)|zi (r) = k) = N(mi k , σ2 i k ), k = 1, · · · , K p(f|z) = i p(fi |zi ) zi (r) follow a Markovian model along the index i A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 66/77

Slide 67

Slide 67 text

Image separation in Sattelite imaging (with H. Snoussi & M. Ichir)            gi (r) = N j=1 Aij fj (r) + i (r) p(fj (r)|zj (r) = k) = N(mj k , σ2 j k ) p(Aij ) = N(A0ij , σ2 0ij ) f g f z A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 67/77

Slide 68

Slide 68 text

Conclusions Inverse problems arise in many science and engineering applications Deterministic Algorithms: Optimization of a two terms criterion, penalty term, regularization term Probabilistic: Bayesian approach Hierarchical prior model with hidden variables are very powerful tools for Bayesian approach to inverse problems. Gauss-Markov-Potts models for images incorporating hidden regions and contours Main Bayesian computation tools: JMAP, MCMC and VBA Application in different imaging system (X ray CT, Microwaves, PET, Ultrasound, Optical Diffusion Tomography (ODT), Acoustic source localization,...) Current Projects: Efficient implementation in 2D and 3D cases A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 68/77

Slide 69

Slide 69 text

Thanks to: Present PhD students: L. Gharsali (Microwave imaging for Cancer detection) M. Dumitru (Multivariate time series analysis for biological signals) S. AlAli (Diffraction imaging for geophysical applications) Freshly Graduated PhD students: C. Cai (2013: Multispectral X ray Tomography) N. Chu (2013: Acoustic sources localization) Th. Boulay (2013: Non Cooperative Radar Target Recognition) R. Prenon (2013: Proteomic and Masse Spectrometry) Sh. Zhu (2012: SAR Imaging) D. Fall (2012: Emission Positon Tomography, Non Parametric Bayesian) D. Pougaza (2011: Copula and Tomography) H. Ayasso (2010: Optical Tomography, Variational Bayes) Older Graduated PhD students: S. F´ ekih-Salem (2009: 3D X ray Tomography) N. Bali (2007: Hyperspectral imaging) O. F´ eron (2006: Microwave imaging) F. Humblot (2005: Super-resolution) M. Ichir (2005: Image separation in Wavelet domain) P. Brault (2005: Video segmentation using Wavelet domain) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 69/77

Slide 70

Slide 70 text

Thanks to: Older Graduated PhD students: H. Snoussi (2003: Sources separation) Ch. Soussen (2000: Geometrical Tomography) G. Mont´ emont (2000: Detectors, Filtering) H. Carfantan (1998: Microwave imaging) S. Gautier (1996: Gamma ray imaging for NDT) M. Nikolova (1994: Piecewise Gaussian models and GNC) D. Pr´ emel (1992: Eddy current imaging) Post-Docs: J. Lapuyade (2011: Dimentionality Reduction and multivariate analysis) S. Su (2006: Color image separation) A. Mohammadpour (2004-2005: HyperSpectral image segmentation) Colleagues: B. Duchˆ ene & A. Joisel (L2S)& G. Perruson (Inverse scattering and Microwave Imaging) N. Gac (L2S) (GPU Implementation) Th. Rodet (L2S) (Computed Tomography) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 70/77

Slide 71

Slide 71 text

Thanks to: National Collaborators A. Vabre & S. Legoupil (CEA-LIST), (3D X ray Tomography) E. Barat (CEA-LIST) (Positon Emission Tomography, Non Parametric Bayesian) C. Comtat (SHFJ, CEA) (PET, Spatio-Temporal Brain activity) J. Picheral (SSE, Sup´ elec) (Acoustic sources localization) D. Blacodon (ONERA) (Acoustic sources separation) J. Lagoutte (Thales Air Systems) (Non Cooperative Radar Target Recognition) P. Grangeat (LETI, CEA, Grenoble) (Proteomic and Masse Spectrometry) F. L´ evi (CNRS-INSERM, Hopital Paul Brousse) (Biological rythms and Chronotherapy of Cancer) International Collaborators K. Sauer (Notre Dame University, IN, USA) (Computed Tomography, Inverse problems) F. Marvasti (Sharif University), (Sparse signal processing) M. Aminghafari (Amir Kabir University) (Independent Components Analysis) A. Mohammadpour (AKU) (Statistical inference) Gh. Yari (Tehran Technological University) (Probability and Analysis) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 71/77

Slide 72

Slide 72 text

References 1 A. Mohammad-Djafari, “Bayesian approach with prior models which enforce sparsity in signal and image processing,” EURASIP Journal on Advances in Signal Processing, vol. Special issue on Sparse Signal Processing, (2012). A. Mohammad-Djafari (Ed.) Probl` emes inverses en imagerie et en vision (Vol. 1 et 2), Hermes-Lavoisier, Trait´ e Signal et Image, IC2, 2009, A. Mohammad-Djafari (Ed.) Inverse Problems in Vision and 3D Tomography, ISTE, Wiley and sons, ISBN: 9781848211728, December 2009, Hardback, 480 pp. A. Mohammad-Djafari, Gauss-Markov-Potts Priors for Images in Computer Tomography Resulting to Joint Optimal Reconstruction and segmentation, International Journal of Tomography & Statistics 11: W09. 76-92, 2008. A Mohammad-Djafari, Super-Resolution : A short review, a new method based on hidden Markov modeling of HR image and future challenges, The Computer Journal doi:10,1093/comjnl/bxn005:, 2008. H. Ayasso and Ali Mohammad-Djafari Joint NDT Image Restoration and Segmentation using Gauss-Markov-Potts Prior Models and Variational Bayesian Computation, IEEE Trans. on Image Processing, TIP-04815-2009.R2, 2010. H. Ayasso, B. Duchene and A. Mohammad-Djafari, Bayesian Inversion for Optical Diffraction Tomography Journal of Modern Optics, 2008. N. Bali and A. Mohammad-Djafari, “Bayesian Approach With Hidden Markov Modeling and Mean Field Approximation for Hyperspectral Data Analysis,” IEEE Trans. on Image Processing 17: 2. 217-225 Feb. (2008). H. Snoussi and J. Idier., “Bayesian blind separation of generalized hyperbolic processes in noisy and underdeterminate mixtures,” IEEE Trans. on Signal Processing, 2006. A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 72/77

Slide 73

Slide 73 text

References 2 O. F´ eron, B. Duch` ene and A. Mohammad-Djafari, Microwave imaging of inhomogeneous objects made of a finite number of dielectric and conductive materials from experimental data, Inverse Problems, 21(6):95-115, Dec 2005. M. Ichir and A. Mohammad-Djafari, Hidden Markov models for blind source separation, IEEE Trans. on Signal Processing, 15(7):1887-1899, Jul 2006. F. Humblot and A. Mohammad-Djafari, Super-Resolution using Hidden Markov Model and Bayesian Detection Estimation Framework, EURASIP Journal on Applied Signal Processing, Special number on Super-Resolution Imaging: Analysis, Algorithms, and Applications:ID 36971, 16 pages, 2006. O. F´ eron and A. Mohammad-Djafari, Image fusion and joint segmentation using an MCMC algorithm, Journal of Electronic Imaging, 14(2):paper no. 023014, Apr 2005. H. Snoussi and A. Mohammad-Djafari, Fast joint separation and segmentation of mixed images, Journal of Electronic Imaging, 13(2):349-361, April 2004. A. Mohammad-Djafari, J.F. Giovannelli, G. Demoment and J. Idier, Regularization, maximum entropy and probabilistic methods in mass spectrometry data processing problems, Int. Journal of Mass Spectrometry, 215(1-3):175-193, April 2002. H. Snoussi and A. Mohammad-Djafari, “Estimation of Structured Gaussian Mixtures: The Inverse EM Algorithm,” IEEE Trans. on Signal Processing 55: 7. 3185-3191 July (2007). N. Bali and A. Mohammad-Djafari, “A variational Bayesian Algorithm for BSS Problem with Hidden Gauss-Markov Models for the Sources,” in: Independent Component Analysis and Signal Separation (ICA 2007) Edited by:M.E. Davies, Ch.J. James, S.A. Abdallah, M.D. Plumbley. 137-144 Springer (LNCS 4666) (2007). A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 73/77

Slide 74

Slide 74 text

References 3 N. Bali and A. Mohammad-Djafari, “Hierarchical Markovian Models for Joint Classification, Segmentation and Data Reduction of Hyperspectral Images” ESANN 2006, September 4-8, Belgium. (2006) M. Ichir and A. Mohammad-Djafari, “Hidden Markov models for wavelet-based blind source separation,” IEEE Trans. on Image Processing 15: 7. 1887-1899 July (2005) S. Moussaoui, C. Carteret, D. Brie and A Mohammad-Djafari, “Bayesian analysis of spectral mixture data using Markov Chain Monte Carlo methods sampling,” Chemometrics and Intelligent Laboratory Systems 81: 2. 137-148 (2005). H. Snoussi and A. Mohammad-Djafari, “Fast joint separation and segmentation of mixed images” Journal of Electronic Imaging 13: 2. 349-361 April (2004) H. Snoussi and A. Mohammad-Djafari, “Bayesian unsupervised learning for source separation with mixture of Gaussians prior,” Journal of VLSI Signal Processing Systems 37: 2/3. 263-279 June/July (2004) F. Su and A. Mohammad-Djafari, “An Hierarchical Markov Random Field Model for Bayesian Blind Image Separation,” 27-30 May 2008, Sanya, Hainan, China: International Congress on Image and Signal Processing (CISP 2008). N. Chu, J. Picheral and A. Mohammad-Djafari, “A robust super-resolution approach with sparsity constraint for near-field wideband acoustic imaging,” IEEE International Symposium on Signal Processing and Information Technology pp 286–289, Bilbao, Spain, Dec14-17,2011 A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 74/77

Slide 75

Slide 75 text

Questions, Discussions, Open mathematical problems Sparsity representation, low rank matrix decomposition Sparsity and positivity or other constraints Group sparsity Algorithmic and implementation issues for great dimensional applications (Big Data) Joint estimation of Dictionary and coefficients Optimization of the KL divergence for Variational Bayesian Approximation Convergency of alternate optimization Other possible algorithms Properties of the obtained approximation Does the moments of q’s corresponds to the moments of p? How about any other statistics: entropy, ... Other divergency or Distance measures? Using Sparsity as a prior in Inverse Problems Applications in Biological data and signal analysis, Medical imaging, Non Destructive Testing (NDT) Industrial Imaging, Communication, Geophysical imaging, Radio Astronomy, ... A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 75/77

Slide 76

Slide 76 text

Blind deconvolution (1) θ2 c f c H g θ1 c θ3 c h © d d ‚ γ0 c θ2 c f c H g α0 c θ1 c γ0 c θ3 c h © d d ‚ g = h ∗ f + = Hf + = F h + Simple priors: p(f, h|g, θ) ∝ p(g|f, h, θ1 ) p(f|θ2 ) p(h|θ3 ) Objective: Infer on f, h JMAP: (f, z) = arg max(f,z) {p(f, z|g)} VBA: Approximate p(f, h|g) by q1 (f) q2 (h) Unsupervised: p(f, h, θ|g) ∝ p(g|f, h, θ1 ) p(f|θ2 ) p(h|θ3 ) p(θ) Objective: Infer on f, h, θ JMAP: (f, z, θ) = arg max(f,z,θ) {p(f, z, θ|g)} VBA: Approximate p(f, h, θ|g) by q1 (f) q2 (h) q3 (θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 76/77

Slide 77

Slide 77 text

Blind deconvolution (2) γf 0 c θ2 c zf c f c H g α0 c α0 c θ1 c γh0 c θ3 c zh c h © d d ‚ g = h ∗ f + = Hf + = F h + Simple priors: p(f, h|g, θ) ∝ p(g|f, h, θ1 ) p(f|θ2 ) p(h|θ3 ) Unsupervised: p(f, h, θ|g) ∝ p(g|f, h, θ1 ) p(f|θ2 ) p(h|θ3 ) p(θ) Sparsity enforcing prior for f: p(f, zf , h, θ|g) ∝ p(g|f, h, θ1 ) p(f|zf ) p(zf |θ2 ) p(h|θ3 ) p(θ) Sparsity enforcing prior for h: p(f, h, zh, θ|g) ∝ p(g|f, h, θ1 ) p(f|θ2 ) p(h|z) p(z|θ3 ) p(θ) Hierarchical models for both f and h: p(f, zf , h, zh, θ|g) ∝ p(g|f, h, θ1 ) p(f|zf ) p(zf |θ2 ) p(h|zh ) p(zh|θ3 ) p(θ) JMAP: (f, zf , h, zh, θ) = arg max (f,zf ,h,zh,θ) p(f, zf , h, zh, θ|g) VBA: Approximate p(f, zf , h, zh, θ|g) by q1 (f) q2 (zf ) q3 (h) q4 (zh ) q5 (θ) A. Mohammad-Djafari, Inverse problems and Bayesian inference, Scube seminar, L2S, CentraleSupelec, March 27, 2015 77/77