A Linear-Time Kernel Goodness-of-Fit Test
Wittawat Jitkrittum1 Wenkai Xu1 Zolt´
an Szab´
o2 Kenji Fukumizu3 Arthur Gretton1
1Gatsby Unit, University College London 2CMAP, ´
Ecole Polytechnique 3The Institute of Statistical Mathematics
A Linear-Time Kernel Goodness-of-Fit Test
Wittawat Jitkrittum1 Wenkai Xu1 Zolt´
an Szab´
o2 Kenji Fukumizu3 Arthur Gretton1
1Gatsby Unit, University College London 2CMAP, ´
Ecole Polytechnique 3The Institute of Statistical Mathematics
Summary
•Given: {x
i}n
i=1 ⇠ q (unknown), and a density p.
•Goal: Test H0 : p = q vs H1 : p 6= q quickly.
•New multivariate goodness-of-fit test (FSSD):
1.Nonparametric: arbitrary, unnormalized p. x 2 Rd.
2.Linear-time: O(n) runtime complexity. Fast.
3.Interpretable: tell where p does not fit the data.
Previous: Kernel Stein Discrepancy (KSD)
•Let x(x, v) := 1
p(x)rx[k(x, v)p(x)] 2 Rd.
Stein witness function: g(v) = E
x⇠q
[x(x, v)] where
g = (g1
, . . . , gd
) and each gi 2 F, an RKHS associated
with kernel k.
- 4 - 2 2 4
- 0.2
0.2
0.4
p(x)
q(x)
g(x)
Known: Under some conditions, kgkFd
= 0 () p = q.
[Chwialkowski et al., 2016, Liu et al., 2016]
Statistic: KSD2 = kgk2
Fd
=
double sums
z }| {
E
x⇠q
E
y⇠q hp
(x, y) ⇡
2
n(n 1)
P
ihp
(x
i
, x
j
). where
hp
(x, y) := [rx log p(x)] k(x, y) [ry log p(y)] + rxryk(x, y)
+ [ry log p(y)] rxk(x, y) + [rx log p(x)] ryk(x, y).
Characteristics of KSD:
3 Nonparametric. Applicable to a wide range of p.
3 Do not need the normalizer of p.
7 Runtime: O(n
2). Computationally expensive.
Linear-Time KSD (LKS) Test: [Liu et al., 2016]
kgk2
Fd
⇡
2
n
P
n/2
i=1 hp
(x2i 1
, x2i
).
3 Runtime: O(n). 7 High variance. Low test power.
The Finite Set Stein Discrepancy (FSSD)
Idea: Evaluate witness g at J locations {v1
, . . . , v
J}. Fast.
FSSD2 =
1
dJ
J
X
j=1
kg(v
j
)k
2
2
.
Proposition (FSSD is a discrepancy measure).
Main conditions:
1.(Nice kernel) Kernel k is C0
-universal, and real analytic
(Taylor series at any point converges) e.g., Gaussian kernel.
2.(Vanishing boundary) lim
kxk!1 p(x)g(x) = 0.
3.(Avoid “blind spots”
) Locations {v1
, . . . , v
J} are drawn
from a distribution h which has a density.
Then, for any J 1, h-a.s. FSSD2 = 0 () p = q.
Characteristics of FSSD:
3 Nonparametric. 3 Do not need the normalizer of p.
3 Runtime: O(n). 3 Higher test power than LKS.
Model Criticism with FSSD
Proposal: Optimize locations {v1
, . . . , v
J} and kernel
bandwidth by arg max score = FSSD2/s
H1
(runtime: O(n)).
Proposition: This procedure maximizes the true positive
rate = P(detect di↵erence | p 6= q).
score: 0.034 score: 0.44
Interpretable Features for Model Criticism
12K robbery
events in Chicago
in 2016
Model p =
10-component
Gaussian mixture
F = v⇤ = where model
does not fit well.
Maximization objective
FSSD2/s
H1
.
Optimized v⇤
is highly
interpretable.
Bahadur Slope and Bahadur E ciency
•Bahadur slope u rate of p-value ! 0 of statistic Tn
under H1
. High = good.
•Bahadur e ciency = ratio slope(1)
slope(2)
of slopes of two tests. > 1 means test(1) better.
•Results: Slopes of FSSD and LKS tests when p = N(0, 1) and q = N(µ
q
, 1).
0 50 100
n
0.0
0.5
1.0
p-value
T(1)
n
T(2)
n
Proposition. Let s2
k
, k2 be kernel bandwidths of
FSSD and LKS. Fix s2
k
= 1. Then, 8
µ
q 6= 0,
9v 2 R, 8
k2 > 0, the Bahadur e ciency
slope(FSSD)(µ
q
, v, s2
k
)
slope(LKS)(µ
q
, k2)
> 2. FSSD is statistically
more e cient than LKS.
Experiment: Restricted Boltzmann Machine
•40 binary hidden units. d = 50 visible units. Significance level a = 0.05.
· · ·
· · ·
Model p
Perturb one
weight to get q.
2000 4000
Sample size n
0.00
0.25
0.50
0.75
Rejection rate
1000 2000 3000 4000
Sample size n
0
100
200
300
Time (s)
2000 4000
Sample size n
0.0
0.5
1.0
Rejection rate
FSSD-opt
FSSD-rand
KSD
LKS
MMD-opt
ME-opt
Better
•FSSD-opt, (FSSD-rand) = Proposed tests. J = 5 optimized, (random) locations.
•MMD-opt [Gretton et al., 2012] = State-of-the-art two-sample test (quadratic-time).
•ME-opt [Jitkrittum et al., 2016] = Linear-time two-sample test with optimized locations.
•Key: FSSD (O(n)), KSD (O(n
2)) have comparable power. FSSD is much faster.
WJ, WX, and AG thank the Gatsby Charitable Foundation
for the financial support. ZSz was financially supported
by the Data Science Initiative. KF has been supported by
KAKENHI Innovative Areas 25120012.
Contact: [email protected]
Code: github.com/wittawatj/kernel-gof
ooshita
asahide
ofutonton
ainischool
classijp
kfujita
codeforeveryone
ashitanoterakoya
arumakan
oripsolob
cbtlibrary
utokyoissr2360
addyosmani
62gerente
kastner
shpigford
ammeep
cromwellryan
zakiwarfel
danielanewman
phodgson
dougneiner
stephaniewalter