Low-rank matrix estimation A bootstrap framework Results Other noise Low-rank matrix estimation ⇒ Model: X ∈ Rn×p ∼ L(µ) with E [X] = µ of low-rank k ⇒ Gaussian noise model: X = µ + ε, with εij iid ∼ N 0, σ2 Image, recommendation system... 3 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Low-rank matrix estimation ⇒ Model: X ∈ Rn×p ∼ L(µ) with E [X] = µ of low-rank k ⇒ Gaussian noise model: X = µ + ε, with εij iid ∼ N 0, σ2 Image, recommendation system... ⇒ Classical solution: truncated SVD ˆ µk = k l=1 ul dl v l Hard thresholding: dl · 1(l ≤ k) = dl · 1(dl ≥ λ) argminµ X − µ 2 2 : rank (µ) ≤ k ⇒ Selecting k? Cross-validation (Josse & Husson, 2012) Not the best in MSE to recover µ! 3 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise SVD via the autoencoder Gaussian model: X = µ + ε, with εij iid ∼ N 0, σ2 ⇒ Classical formulation: • ˆ µk = argminµ X − µ 2 2 : rank (µ) ≤ k • TSVD: ˆ µk = k l=1 ul dl v l 8 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise SVD via the autoencoder Gaussian model: X = µ + ε, with εij iid ∼ N 0, σ2 ⇒ Classical formulation: • ˆ µk = argminµ X − µ 2 2 : rank (µ) ≤ k • TSVD: ˆ µk = k l=1 ul dl v l = k l=1 fl v l F = XV (V V )−1 V = (F F)−1F X ˆ µk = FV = XV (V V )−1V = XPV 8 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise SVD via the autoencoder Gaussian model: X = µ + ε, with εij iid ∼ N 0, σ2 ⇒ Classical formulation: • ˆ µk = argminµ X − µ 2 2 : rank (µ) ≤ k • TSVD: ˆ µk = k l=1 ul dl v l = k l=1 fl v l F = XV (V V )−1 V = (F F)−1F X ˆ µk = FV = XV (V V )−1V = XPV ⇒ Another formulation: autoencoder (Boulard & Kamp, 1988) • ˆ µk = XBk, where Bk = argminB X − XB 2 2 : rank (B) ≤ k 8 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Parametric Bootstrap ⇒ Autoencoder: compress X ˆ µk = XBk, where Bk = argminB X − XB 2 2 : rank (B) ≤ k ⇒ Aim: recover µ ˆ µ∗ k = XB∗ k , B∗ k = argminB EX∼L(µ) µ − XB 2 2 : rank (B) ≤ k ⇒ Parametric Bootstrap: plug µ by X ˆ µboot k = XBk, Bk = argminB EX∼L(X) X − XB 2 2 : rank (B) ≤ k Model X = µ + ε ⇒ X = X + ε 9 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Stable Autoencoder ⇒ Bootstrap = Feature noising ˆ µboot k = XBk, Bk = argminB EX∼L(X) X − XB 2 2 : rkB ≤ k Bk = argminB Eε X − (X + ε)B 2 2 : rkB ≤ k ⇒ Equivalent to a ridge autoencoder problem ˆ µ(k) λ = XBλ, Bλ = argminB X − XB 2 2 + λ B 2 2 : rkB ≤ k ⇒ Solution: singular-value shrinkage estimator ˆ µboot k = k l=1 ul dl 1 + λ/d2 l vl with λ = nσ2 ˆ µ(λ) k = XBλ, Bλ = X X + S −1 X X, S = diag(λ) 10 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Feature noising and regularization in regression Bishop (1995). Training with noise is equivalent to tikhonov regularization. Neural computation. ˆ β = argminβ E εij iid ∼ N(0, σ2) Y − (X + ε) β 2 2 Many noisy copies of the data and average out the auxiliary noise is equivalent to ridge regularization with λ = nσ2: ˆ β(R) λ = argminβ Y − Xβ + λ β 2 2 ⇒ Control overfitting by artificially corrupting the training data 11 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise A centered parametric bootstrap Model: X = µ + ε, with εij ∼ N 0, σ2 ⇒ Aim: recover µ ˆ µ∗ k = XB∗ k , B∗ k = argminB EX∼L(µ) µ − XB 2 2 : rank (B) ≤ k ⇒ X as a proxy for µ ˆ µboot k = XBk, Bk = argminB EX∼L(X) X − XB 2 2 : rank (B) ≤ k ⇒ X is bigger than µ! ˆ µ = XB as a proxy for µ: B = argminB E˜ ˆ µ∼L(ˆ µ) ˆ µ − ˜ ˆ µB 2 2 Model X = µ + ε ⇒ ˜ ˆ µ = ˆ µ + ε 12 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Simulation results ⇒ Different noise regime • low noise: truncated SVD • moderate noise: Asympt, SA, ISA (non-linear transformation) • high noise (SNR low, k large): Soft thresholding ⇒ Adaptive estimator is very flexible! (selection with SURE) ⇒ ISA performs well in MSE (even outside the Gaussian noise) and as a by-product estimates accurately the rank! Rq: not the usual behavior n > p: rows more perturbed than the columns; n < p: columns more perturbed; n = p 16 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Expression data PCA rPCA Bi-clustering from rPCA better find the 4 nutritional statuses! 18 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Partial conclusion ⇒ Parametric Bootstrap: flexible framework for transforming noise model into regularized matrix estimator ⇒ Gaussian noise: singular value shrinkage ⇒ Gaussian noise is not always appropriate. Procedure is most useful outside the Gaussian framework 19 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Count data ⇒ Model: X ∈ Rn×p ∼ L(µ) with E [X] = µ of low-rank k Binomial noise model: Xij ∼ 1 1−δ Binomial (µij , 1 − δ) Poisson noise: Xij ∼ Poisson (µij ) (Binomial with δ = 0.5) ⇒ ˆ µk = k l=1 ul dl v l 21 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Count data ⇒ Model: X ∈ Rn×p ∼ L(µ) with E [X] = µ of low-rank k Binomial noise model: Xij ∼ 1 1−δ Binomial (µij , 1 − δ) Poisson noise: Xij ∼ Poisson (µij ) (Binomial with δ = 0.5) ⇒ ˆ µk = k l=1 ul dl v l ⇒ Bootstap estimator = noising: Xij ∼ Poisson (Xij ) ˆ µboot = XB B = argminB EX∼L(X) X − XB 2 2 ⇒ Estimator robust to subsampling the obs used to build X 21 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Bootstrap estimators ⇒ Feature noising = regularization B = argminB X − XB 2 2 + S 1 2 B 2 2 Sjj = n i=1 Var X∼L(X) Xij ˆ µ = X(X X + δ 1−δ S)−1X X, S diagonal with row-sums of X ⇒ New estimator ˆ µ that does not reduce to singular value shrinkage: new singular vectors! ⇒ ISA estimator - iterative algorithm 1. ˆ µ = X ˆ B 2. ˆ B = (ˆ µ ˆ µ + S)−1 ˆ µ ˆ µ 22 / 28
Low-rank matrix estimation A bootstrap framework Results Other noise Feature Noising = Ridge = Shrinkage (Proof) Eε X − (X + ε) B 2 2 = X − XB 2 2 + Eε εB 2 2 = X − XB 2 2 + i,j,k B2 ij Var εjk = X − XB 2 2 + nσ2 B 2 2 Let X = UDV be the SVD of X, λ = nσ2 Bλ = V BλV T , where Bλ = argminB D − DB 2 2 + λ B 2 2 Bii = argminBii (1 − Bii )2 D2 ii + λB2 ii = D2 ii λ + D2 ii ˆ µ(k) λ = n i=1 Ui. Dii 1 + λ/D2 ii Vi. 29 / 28
josse
sensetime_japan
thc1006
naoto0804
kuehara
kikuzo
upura
miyuoba
yoshipon
kuanyi_n_n
koharite
misonuma
bakanaouji
chriscoyier
kastner
bkeepers
erikaheidi
jensimmons
smashingmag
jacobian
jmmastey
brianwarren
chrislema
andyhume