Fred J. Hickernell
July 31, 2019
73

# LANL 2019 July

Presentation given to scientists at Los Alamos on July 31, 2019

July 31, 2019

## Transcript

1. The Challenges of Approximating Functions of Many Variables
Fred J. Hickernell
Department of Applied Mathematics
Center for Interdisciplinary Scientiﬁc Computation
Illinois Institute of Technology
[email protected] mypages.iit.edu/~hickernell
Joint work with Yuhan Ding, Peter Kritzer, and Simon Mak
This work partially supported by NSF-DMS-1522687 and NSF-DMS-1638521 (SAMSI)
Happy Birthday to my brother Bob, Chief of the Quantum Electromagnetics Division at NIST
Thank you for the the kind invitation
Los Alamos National Laboratory, July 31, 2019

2. Introduction Solvability Smoothness Tractability Cones Design Example
Highlights
Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R
f − ALG(f, ε)
G
ε ∀ε > 0, f ∈ H ⊆ F (Banach space)
Impossible for inﬁnite dimensional Banach space H = F
Smoothness assumed by F speeds up ALG
Smoothness alone cannot save from the curse of dimensionality, but a low eﬀective-dimension
structure can
Choosing H to be a cone , rather than a ball , paves the way for adaptive algorithms
Interesting design (where to sample) problems remain
2/18

3. Introduction Solvability Smoothness Tractability Cones Design Example
Problem
Input
Black box providing noiseless information about f : Ω ⊆ Rd → R
e.g., function values or series coeﬃcients, costing \$(f) each
Error tolerance ε
Output ALG(f, ε) (as a surrogate, for solving PDEs, for uncertainty quantiﬁcation) that is
Cheap to evaluate and manipulate
Accurate f − ALG(f, ε)
G
ε ∀ε > 0
Eﬃcient to construct
3/18

4. Introduction Solvability Smoothness Tractability Cones Design Example
Problem
Input
Black box providing noiseless information about f : Ω ⊆ Rd → R
e.g., function values or series coeﬃcients, costing \$(f) each
Error tolerance ε
Output ALG(f, ε) that is
Cheap to evaluate and manipulate
Accurate f − ALG(f, ε)
G
ε ∀ε > 0
Eﬃcient to construct
Approximation with ﬁxed computation budget: APP(f, n) =
n
i=1
Li
(f)gi,n
L1
(f), L2
(f), . . . is input function information, e.g., function values or series coeﬃcients
gn
= (g1,n
, . . . , gn,n
) ∈ Gn
COST(f, n) = O(n\$(f) + COST(gn
))
Algorithm ALG(f, ε) = APP(f, n∗(f, ε)) satisfying f − APP(f, n∗(f, ε))
G
ε ∀ε > 0
COST(f, ε) = COST(f, n∗(f, ε)) + cost to determine n∗(f, ε)
3/18

5. Introduction Solvability Smoothness Tractability Cones Design Example
Problem
Input
Black box providing noiseless information about f : Ω ⊆ Rd → R costing \$(f) each
f ∈ F, deﬁnition of · F
enshrines smoothness assumptions
Error tolerance ε
Output ALG(f, ε) that is
Cheap to evaluate and manipulate
Accurate f − ALG(f, ε)
G
ε ∀ε > 0, f ∈ H ⊂ F, provably
Eﬃcient to construct
Approximation with ﬁxed computation budget: APP(f, n) =
n
i=1
Li
(f)gi,n
L1
(f), L2
(f), . . . is input function information
gn
= (g1,n
, . . . , gn,n
) ∈ Gn
COST(f, n) = O(n\$(f) + COST(gn
))
Algorithm ALG(f, ε) = APP(f, n∗(f, ε)) satisfying f − APP(f, n∗(f, ε))
G
ε ∀ε > 0, f ∈ H ⊂ F
COST(f, ε) = COST(f, n∗(f, ε)) + cost to determine n∗(f, ε)
3/18

6. Introduction Solvability Smoothness Tractability Cones Design Example
Impossible for All f in Inﬁnite Dimensional F
f − ALG(f, ε)
G
ε ∀f ∈ H ⊂ F
Suppose H = F
Fix ε > 0
Let L1
, . . . , Ln
be the linear information used to construct ALG(0, ε)
Choose nonzero fooling function f ∈ F, such that L1
(f) = · · · = Ln
(f) = 0
ALG(±cf, ε) = ALG(0, ε) for all c > 0
For all c > 0
ε max cf − ALG(cf, ε)
G
, −cf − ALG(−cf, ε)
G
1
2
cf − ALG(cf, ε)
G
+ −cf − ALG(−cf, ε)
G
1
2
cf − ALG(0, ε)
G
+ cf + ALG(0, ε)
G
c f
G
=⇒⇐=
4/18

7. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Makes Algorithm Less Expensive
For d = 1, let {u0
, u1
, . . .} be an orthogonal (polynomial) basis for F and G
F := f =

k=0
f(k)uk
: f
F
:=
f(k)
λk

k=0 2
< ∞ , λ0
λ1 · · · > 0
G := g =

k=0
^
g(k)uk
: g
G
:= ^
g(k) ∞
k=0 2
< ∞ , APP(f, n) =
n−1
k=0
f(k)uk
5/18

8. Introduction Solvability Smoothness Tractability Cones Design Example
Bases for Function Approximation
Legendre
Chebyshev
Sine and Cosine
6/18

9. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Makes Algorithm Less Expensive
For d = 1, let {u0
, u1
, . . .} be an orthogonal (polynomial) basis for F and G
F := f =

k=0
f(k)uk
: f
F
:=
f(k)
λk

k=0 2
< ∞ , λ0
λ1 · · · > 0
G := g =

k=0
^
g(k)uk
: g
G
:= ^
g(k) ∞
k=0 2
< ∞ , APP(f, n) =
n−1
k=0
f(k)uk
f − APP(f, n)
G
= f(k) ∞
k=n 2
=
λk
f(k)
λk

k=n 2 tight
f
F
λn
?
ε, require λn ↓ 0
7/18

10. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Makes Algorithm Less Expensive
For d = 1, let {u0
, u1
, . . .} be an orthogonal (polynomial) basis for F and G
F := f =

k=0
f(k)uk
: f
F
:=
f(k)
λk

k=0 2
< ∞ , λ0
λ1 · · · > 0
G := g =

k=0
^
g(k)uk
: g
G
:= ^
g(k) ∞
k=0 2
< ∞ , APP(f, n) =
n−1
k=0
f(k)uk
f − APP(f, n)
G
= f(k) ∞
k=n 2
=
λk
f(k)
λk

k=n 2 tight
f
F
λn
?
ε, require λn ↓ 0
By choosing H = BR
:= {f ∈ F : f
F
R}, we can deﬁne our algorithm
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λn
= O(n−1/p) =⇒ COST(BR
, ε) = O(Rpε−p)
7/18

11. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Makes Algorithm Less Expensive
For d = 1, let {u0
, u1
, . . .} be an orthogonal (polynomial) basis for F and G
F := f =

k=0
f(k)uk
: f
F
:=
f(k)
λk

k=0 2
< ∞ , λ0
λ1 · · · > 0
G := g =

k=0
^
g(k)uk
: g
G
:= ^
g(k) ∞
k=0 2
< ∞ , APP(f, n) =
n−1
k=0
f(k)uk
f − APP(f, n)
G
= f(k) ∞
k=n 2
=
λk
f(k)
λk

k=n 2 tight
f
F
λn
?
ε, require λn ↓ 0
By choosing H = BR
:= {f ∈ F : f
F
R}, we can deﬁne our algorithm
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λn
= O(n−1/p) =⇒ COST(BR
, ε) = O(Rpε−p)
ALG has optimal cost among all successful algorithms using Fourier coeﬃcients
(look at the cost of approximating the zero function)
7/18

12. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Makes Algorithm Less Expensive
For d = 1, let {u0
, u1
, . . .} be an orthogonal (polynomial) basis for F and G
F := f =

k=0
f(k)uk
: f
F
:=
f(k)
λk

k=0 2
< ∞ , λ0
λ1 · · · > 0
G := g =

k=0
^
g(k)uk
: g
G
:= ^
g(k) ∞
k=0 2
< ∞ , APP(f, n) =
n−1
k=0
f(k)uk
f − APP(f, n)
G
= f(k) ∞
k=n 2
=
λk
f(k)
λk

k=n 2 tight
f
F
λn
?
ε, require λn ↓ 0
By choosing H = BR
:= {f ∈ F : f
F
R}, we can deﬁne our algorithm
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λn
= O(n−1/p) =⇒ COST(BR
, ε) = O(Rpε−p)
ALG has optimal cost among all successful algorithms using Fourier coeﬃcients
(look at the cost of approximating the zero function)
Similar results for algorithms based on function values, but need to choose the design carefully
7/18

13. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Cannot Save You from the Curse of Dimensionality1
For arbitrary d, let {u0
= 1, u1
} be used to construct a product basis F and G (multlinear functions)
F :=

f(x) =
k∈{0,1}d
f(k)uk : f
F
:=
f(k)
λk
k∈{0,1}d
2
< ∞

, uk(x) :=
d
=1
uk
(x )
G :=

g =
k∈{0,1}d
^
g(k)uk : g
G
:= ^
g(k)
k∈{0,1}d
2
< ∞

, λk :=
d
=1
k =0
s = s k 0
APP(f, n) =
n
i=1
f(ki
)uki
, λk1
= 1 s = λk2
· · · sd,
1NovWoz08a. 8/18

14. Introduction Solvability Smoothness Tractability Cones Design Example
Bases for Function Approximation
Legendre
Chebyshev
Sine and Cosine
9/18

15. Introduction Solvability Smoothness Tractability Cones Design Example
Smoothness Cannot Save You from the Curse of Dimensionality1
For arbitrary d, let {u0
= 1, u1
} be used to construct a product basis F and G (multlinear functions)
F :=

f(x) =
k∈{0,1}d
f(k)uk : f
F
:=
f(k)
λk
k∈{0,1}d
2
< ∞

, uk(x) :=
d
=1
uk
(x )
G :=

g =
k∈{0,1}d
^
g(k)uk : g
G
:= ^
g(k)
k∈{0,1}d
2
< ∞

, λk :=
d
=1
k =0
s = s k 0
APP(f, n) =
n
i=1
f(ki
)uki
, λk1
= 1 s = λk2
· · · sd,
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λkn
= O n−1/pespd/p =⇒ COST(BR
, ε) = O Rpε−pespd ∀p exponential growth in d
1NovWoz08a. 10/18

16. Introduction Solvability Smoothness Tractability Cones Design Example
Proof that λkn+1
= O n−1/pespd/p ∀p > 0
λp
kn+1
1
n
λp
k1
+ · · · + λp
kn
λki
are ordered
λkn+1
1
n1/p
λp
k1
+ · · · + λp
kn
1/p
pth root
1
n1/p
λp
k1
+ · · · + λp
k
2d
1/p
1
n1/p
1 + sp d/p
binomial theorem
espd/p
n1/p
1 + x ex for x 0
There is a similar proof that provides a lower bound on λkn+1
11/18

17. Introduction Solvability Smoothness Tractability Cones Design Example
Coordinate Weights Can Save You1
For arbitrary d, let {u0
= 1, u1
} be used to construct a product basis F and G (multlinear functions)
F :=

f(x) =
k∈{0,1}d
f(k)uk : f
F
:=
f(k)
λk
k∈{0,1}d
2
< ∞

, uk(x) :=
d
=1
uk
(x )
G :=

g =
k∈{0,1}d
^
g(k)uk : g
G
:= ^
g(k)
k∈{0,1}d
2
< ∞

, λk :=
d
=1
k =0
w s
APP(f, n) =
n
i=1
f(ki
)uki
, λk1
= 1 w1
s = λk2
· · · , 1 = w1
w2 · · ·
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λkn
= O n−1/p exp p−1sp d
=1
wp =⇒ COST(BR
, ε) = O Rpε−p exp sp d
=1
wp ∀p
cost is independent of d if coordinate weights decay quickly
1NovWoz08a. 12/18

18. Introduction Solvability Smoothness Tractability Cones Design Example
Coordinate Weights Can Save You, Even with Higher Order Polynomials1
For arbitrary d, let {u0
= 1, u1
, . . .} be used to construct a product basis F and G
F :=

f(x) =
k∈Nd
0
f(k)uk : f
F
:=
f(k)
λk
k∈Nd
0 2
< ∞

, uk(x) :=
d
=1
uk
(x )
G :=

g =
k∈Nd
0
^
g(k)uk : g
G
:= ^
g(k)
k∈Nd
0 2
< ∞

, λk :=
d
=1
k =0
w sk
APP(f, n) =
n
i=1
f(ki
)uki
, λk1
= 1 λk2
· · · , 1 = w1
w2 · · ·
ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1
ε/R} =⇒ f − ALG(f, ε)
G
ε ∀f ∈ BR
λkn
= O n−1/p exp p−1 ∞
k=1
sp
k
d
=1
wp =⇒ COST(BR
, ε) = O Rpε−p exp ∞
k=1
sp
k
d
=1
wp ∀p
cost is independent of d if coordinate and smoothness weights decay quickly
1NovWoz08a. 12/18

19. Introduction Solvability Smoothness Tractability Cones Design Example
Look to Cones for Adaptive Algorithms
Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R
f − ALG(f, ε)
G
ε ∀ε > 0, f ∈ H ⊆ F (Banach space)
So far, H = BR
Hard to know a priori how large R should be for your problem
Computational cost depends on R and ε, but not on f data
Choosing H = makes adaptive algorithms possible2
2HicEtal17a, KunEtal19a, DinHic20a, RatHic19a. 13/18

20. Introduction Solvability Smoothness Tractability Cones Design Example
Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3
F := f =

i=1
f(ki
)uki
: f
F
:=
f(ki
)
λki

i=1 2
λk1
λk2
· · · > 0
λ aﬀects convergence rate &
tractability
G := g =

i=1
^
g(ki
)uki
: g
G
:= ^
g
2
, APP(f, n) =
n
i=1
f(ki
)uki
3DinEtal20a. 14/18

21. Introduction Solvability Smoothness Tractability Cones Design Example
Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3
F := f =

i=1
f(ki
)uki
: f
F
:=
f(ki
)
λki

i=1 2
λk1
λk2
· · · > 0
λ aﬀects convergence rate &
tractability
G := g =

i=1
^
g(ki
)uki
: g
G
:= ^
g
2
, APP(f, n) =
n
i=1
f(ki
)uki
Cd,λ,n1,A
:= f ∈ F : f
F
A
f(ki
)
λki
n1
i=1 2
pilot sample bounds the norm of the input
A is inﬂation factor, n1
is initial sample size
f − APP(f, n)
G

A2
f(ki
)
λki
n1
i=1
2
2

f(ki
)
λki
n
i=1
2
2

1/2
upper bound on f− n
i=1
f(ki)uki F
λkn+1
=: ERR f(ki
) n
i=1
, n
data-driven
3DinEtal20a. 14/18

22. Introduction Solvability Smoothness Tractability Cones Design Example
Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3
F := f =

i=1
f(ki
)uki
: f
F
:=
f(ki
)
λki

i=1 2
λk1
λk2
· · · > 0
λ aﬀects convergence rate &
tractability
G := g =

i=1
^
g(ki
)uki
: g
G
:= ^
g
2
, APP(f, n) =
n
i=1
f(ki
)uki
Cd,λ,n1,A
:= f ∈ F : f
F
A
f(ki
)
λki
n1
i=1 2
pilot sample bounds the norm of the input
A is inﬂation factor, n1
is initial sample size
f − APP(f, n)
G

A2
f(ki
)
λki
n1
i=1
2
2

f(ki
)
λki
n
i=1
2
2

1/2
upper bound on f− n
i=1
f(ki)uki F
λkn+1
=: ERR f(ki
) n
i=1
, n
data-driven
ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki
) n
i=1
, n ε}
3DinEtal20a. 14/18

23. Introduction Solvability Smoothness Tractability Cones Design Example
Adaptive Algorithm for Cone of Inputs Based on Pilot Sample
F := f =

i=1
f(ki
)uki
: f
F
:=
f(ki
)
λki

i=1 2
λk1
λk2
· · · > 0
λ aﬀects convergence rate &
tractability
G := g =

i=1
^
g(ki
)uki
: g
G
:= ^
g
2
, APP(f, n) =
n
i=1
f(ki
)uki
Cd,λ,n1,A
:= f ∈ F : f
F
A
f(ki
)
λki
n1
i=1 2
pilot sample bounds the norm of the input
A is inﬂation factor, n1
is initial sample size
f − APP(f, n)
G

A2
f(ki
)
λki
n1
i=1
2
2

f(ki
)
λki
n
i=1
2
2

1/2
upper bound on f− n
i=1
f(ki)uki F
λkn+1
=: ERR f(ki
) n
i=1
, n
data-driven
ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki
) n
i=1
, n ε}
COST(ALG, Cd,λ,n1,A
, ε, R) = max n∗(f, ε) : f ∈ Cλ,n1,A ∩ BR
= ∩
= min n n1
: λkn+1
ε/[(A2 − 1)1/2R]
ALG is essentially optimal; computational cost is d independent if λk
decay quickly 14/18

24. Introduction Solvability Smoothness Tractability Cones Design Example
Challenges When Using Function Values as Information
Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R
f − ALG(f, ε)
G
ε ∀ε > 0, f ∈ H ⊆ F (Banach space)
So far, the function information is series coeﬃcients
COST(f, ε) = O n∗(f, ε) \$(f) , the best one can hope for
Cost of constructing the approximation and determining the stopping sample size is essentially the same
as getting the data
But using series coeﬃcients is not so realistic
Developing theory for multivariate function approximation using function values is challenging
One must bound the aliasing eﬀects of using interpolation or other means to approximate the coeﬃcients
Interpolation, reproducing kernel Hilbert space methods, and kriging typically require O(n3) operations
to compute approximation, perhaps more if one is tuning the parameters of the kernels; but there are
eﬀorts to speed this up3
Space ﬁlling designs such as integration lattices4, digital nets5, and sparse grids6 are promising
3SchEtal19.
4DicEtal14a.
5DicPil10a.
6BunGrie04a. 15/18

25. Introduction Solvability Smoothness Tractability Cones Design Example
Cheng and Sandu Function7
Chebyshev polynomials, Coordinate weights w inferred, Smoothness weights sk
inferred
function values used
7DinEtal20a, VirLib17a. 16/18

26. Introduction Solvability Smoothness Tractability Cones Design Example
Highlights
Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R
f − ALG(f, ε)
G
ε ∀ε > 0, f ∈ H ⊆ F (Banach space)
Impossible for inﬁnite dimensional Banach space H = F
Smoothness assumed by F speeds up ALG
Smoothness alone cannot save from the curse of dimensionality, but a low eﬀective-dimension
structure can
Choosing H to be a cone , rather than a ball , paves the way for adaptive algorithms
Interesting design (where to sample) problems remain
17/18

27. Thank you
These slides are available at
speakerdeck.com/fjhickernell/lanl-2019-july