The Challenges of Approximating Functions of Many Variables Fred J. Hickernell Department of Applied Mathematics Center for Interdisciplinary Scientific Computation Illinois Institute of Technology [email protected] mypages.iit.edu/~hickernell speakerdeck.com/fjhickernell Joint work with Yuhan Ding, Peter Kritzer, and Simon Mak This work partially supported by NSF-DMS-1522687 and NSF-DMS-1638521 (SAMSI) Computational Mathematics and Statistics Seminar, August 30, 2019

Introduction Solvability Smoothness Tractability Cones Design Example References Highlights Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R f − ALG(f, ε) G ε ∀ε > 0, f ∈ H ⊆ F (Banach space) Impossible for infinite dimensional Banach space H = F, so what is H? Smoothness assumed by F speeds up ALG, not surprising Smoothness alone cannot save from the curse of dimensionality, but a low effective-dimension structure can Choosing H to be a cone , rather than a ball , paves the way for adaptive algorithms Interesting design (where to sample) problems remain 2/18

Introduction Solvability Smoothness Tractability Cones Design Example References Problem Input Black box providing noiseless information about f : Ω ⊆ Rd → R e.g., function values or series coefficients, costing $(f) each Error tolerance ε Output ALG(f, ε) (as a surrogate, for solving PDEs, for uncertainty quantification) that is Cheap to evaluate and manipulate Accurate f − ALG(f, ε) G ε ∀ε > 0 Efficient to construct 3/18

Introduction Solvability Smoothness Tractability Cones Design Example References Problem Input Black box providing noiseless information about f : Ω ⊆ Rd → R e.g., function values or series coefficients, costing $(f) each Error tolerance ε Output ALG(f, ε) that is Cheap to evaluate and manipulate Accurate f − ALG(f, ε) G ε ∀ε > 0 Efficient to construct Approximation with fixed computation budget: APP(f, n) = n i=1 Li (f)gi,n L1 (f), L2 (f), . . . is input function information, e.g., function values or series coefficients gn = (g1,n , . . . , gn,n ) ∈ Gn cardinal functions COST(f, n) = O(n$(f) + COST(gn )) Algorithm ALG(f, ε) = APP(f, n∗(f, ε)) satisfying f − APP(f, n∗(f, ε)) G ε ∀ε > 0 COST(f, ε) = COST(f, n∗(f, ε)) + cost to determine n∗(f, ε) 3/18

Introduction Solvability Smoothness Tractability Cones Design Example References Problem Input Black box providing noiseless information about f : Ω ⊆ Rd → R costing $(f) each f ∈ F, definition of · F enshrines smoothness assumptions Error tolerance ε Output ALG(f, ε) that is Cheap to evaluate and manipulate Accurate f − ALG(f, ε) G ε ∀ε > 0, f ∈ H ⊂ F, provably Efficient to construct Approximation with fixed computation budget: APP(f, n) = n i=1 Li (f)gi,n L1 (f), L2 (f), . . . is input function information gn = (g1,n , . . . , gn,n ) ∈ Gn cardinal functions COST(f, n) = O(n$(f) + COST(gn )) Algorithm ALG(f, ε) = APP(f, n∗(f, ε)) satisfying f − APP(f, n∗(f, ε)) G ε ∀ε > 0, f ∈ H ⊂ F COST(f, ε) = COST(f, n∗(f, ε)) + cost to determine n∗(f, ε) 3/18

Introduction Solvability Smoothness Tractability Cones Design Example References Impossible for All f in Infinite Dimensional F f − ALG(f, ε) G ε ∀f ∈ H ⊂ F Proof by contradiction Suppose H = F Fix ε > 0 Let L1 , . . . , Ln be the linear information used to construct ALG(0, ε) Choose nonzero fooling function f ∈ F, such that L1 (f) = · · · = Ln (f) = 0 ALG(±cf, ε) = ALG(0, ε) for all c > 0 For all c > 0 ε max cf − ALG(cf, ε) G , −cf − ALG(−cf, ε) G 1 2 cf − ALG(cf, ε) G + −cf − ALG(−cf, ε) G 1 2 cf − ALG(0, ε) G + cf + ALG(0, ε) G c f G =⇒⇐= 4/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Makes Algorithm Less Expensive For d = 1, let {u0 , u1 , . . .} be an orthogonal (polynomial) basis for F and G F := f = ∞ k=0 f(k)uk : f F := f(k) λk ∞ k=0 2 < ∞ , λ0 λ1 · · · > 0 G := g = ∞ k=0 ^ g(k)uk : g G := ^ g(k) ∞ k=0 2 < ∞ , APP(f, n) = n−1 k=0 f(k)uk 5/18

Introduction Solvability Smoothness Tractability Cones Design Example References Bases for Function Approximation Legendre Chebyshev Sine and Cosine 6/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Makes Algorithm Less Expensive For d = 1, let {u0 , u1 , . . .} be an orthogonal (polynomial) basis for F and G F := f = ∞ k=0 f(k)uk : f F := f(k) λk ∞ k=0 2 < ∞ , λ0 λ1 · · · > 0 G := g = ∞ k=0 ^ g(k)uk : g G := ^ g(k) ∞ k=0 2 < ∞ , APP(f, n) = n−1 k=0 f(k)uk f − APP(f, n) G = f(k) ∞ k=n 2 = λk f(k) λk ∞ k=n 2 tight f F λn ? ε require λn ↓ 0 smoothness of F > smoothness of G 7/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Makes Algorithm Less Expensive For d = 1, let {u0 , u1 , . . .} be an orthogonal (polynomial) basis for F and G F := f = ∞ k=0 f(k)uk : f F := f(k) λk ∞ k=0 2 < ∞ , λ0 λ1 · · · > 0 G := g = ∞ k=0 ^ g(k)uk : g G := ^ g(k) ∞ k=0 2 < ∞ , APP(f, n) = n−1 k=0 f(k)uk f − APP(f, n) G = f(k) ∞ k=n 2 = λk f(k) λk ∞ k=n 2 tight f F λn ? ε require λn ↓ 0 smoothness of F > smoothness of G By choosing H = BR := {f ∈ F : f F R} , we can define our algorithm ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λn = O(n−1/p) =⇒ COST(BR , ε) = O(Rpε−p) 7/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Makes Algorithm Less Expensive For d = 1, let {u0 , u1 , . . .} be an orthogonal (polynomial) basis for F and G F := f = ∞ k=0 f(k)uk : f F := f(k) λk ∞ k=0 2 < ∞ , λ0 λ1 · · · > 0 G := g = ∞ k=0 ^ g(k)uk : g G := ^ g(k) ∞ k=0 2 < ∞ , APP(f, n) = n−1 k=0 f(k)uk f − APP(f, n) G = f(k) ∞ k=n 2 = λk f(k) λk ∞ k=n 2 tight f F λn ? ε require λn ↓ 0 smoothness of F > smoothness of G By choosing H = BR := {f ∈ F : f F R} , we can define our algorithm ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λn = O(n−1/p) =⇒ COST(BR , ε) = O(Rpε−p) ALG has optimal cost among all successful algorithms using Fourier coefficients (look at the cost of approximating the zero function) 7/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Makes Algorithm Less Expensive For d = 1, let {u0 , u1 , . . .} be an orthogonal (polynomial) basis for F and G F := f = ∞ k=0 f(k)uk : f F := f(k) λk ∞ k=0 2 < ∞ , λ0 λ1 · · · > 0 G := g = ∞ k=0 ^ g(k)uk : g G := ^ g(k) ∞ k=0 2 < ∞ , APP(f, n) = n−1 k=0 f(k)uk f − APP(f, n) G = f(k) ∞ k=n 2 = λk f(k) λk ∞ k=n 2 tight f F λn ? ε require λn ↓ 0 smoothness of F > smoothness of G By choosing H = BR := {f ∈ F : f F R} , we can define our algorithm ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λn ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λn = O(n−1/p) =⇒ COST(BR , ε) = O(Rpε−p) ALG has optimal cost among all successful algorithms using Fourier coefficients (look at the cost of approximating the zero function) Similar results for algorithms based on function values, but need to choose the design carefully 7/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Cannot Save You from the Curse of Dimensionality1 For arbitrary d, let {u0 = 1, u1 } be used to construct a product basis F and G (multlinear functions) F :=    f(x) = k∈{0,1}d f(k)uk : f F := f(k) λk k∈{0,1}d 2 < ∞    , uk(x) := d =1 uk (x ) G :=    g = k∈{0,1}d ^ g(k)uk : g G := ^ g(k) k∈{0,1}d 2 < ∞    , λk := d =1 k =0 s = s k 0 APP(f, n) = n i=1 f(ki )uki , λk1 = 1 s = λk2 · · · sd 1Novak, E. & Woźniakowski, H. Tractability of Multivariate Problems Volume I: Linear Information. EMS Tracts in Mathematics 6 (European Mathematical Society, Zürich, 2008). 8/18

Introduction Solvability Smoothness Tractability Cones Design Example References Bases for Function Approximation Legendre Chebyshev Sine and Cosine 9/18

Introduction Solvability Smoothness Tractability Cones Design Example References Smoothness Cannot Save You from the Curse of Dimensionality1 For arbitrary d, let {u0 = 1, u1 } be used to construct a product basis F and G (multlinear functions) F :=    f(x) = k∈{0,1}d f(k)uk : f F := f(k) λk k∈{0,1}d 2 < ∞    , uk(x) := d =1 uk (x ) G :=    g = k∈{0,1}d ^ g(k)uk : g G := ^ g(k) k∈{0,1}d 2 < ∞    , λk := d =1 k =0 s = s k 0 APP(f, n) = n i=1 f(ki )uki , λk1 = 1 s = λk2 · · · sd ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1 ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λkn = O n−1/pespd/p =⇒ COST(BR , ε) = O Rpε−pespd ∀p exponential growth in d 1Novak, E. & Woźniakowski, H. Tractability of Multivariate Problems Volume I: Linear Information. EMS Tracts in Mathematics 6 (European Mathematical Society, Zürich, 2008). 10/18

Introduction Solvability Smoothness Tractability Cones Design Example References Proof that λkn+1 = O n−1/pespd/p ∀p > 0 λp kn+1 1 n λp k1 + · · · + λp kn λki are ordered λkn+1 1 n1/p λp k1 + · · · + λp kn 1/p pth root 1 n1/p λp k1 + · · · + λp k 2d 1/p add the rest in 1 n1/p 1 + sp d/p binomial theorem espd/p n1/p 1 + x ex for x 0 There is a similar proof that provides a lower bound on λkn+1 11/18

Introduction Solvability Smoothness Tractability Cones Design Example References Coordinate Weights Can Save You1 For arbitrary d, let {u0 = 1, u1 } be used to construct a product basis F and G (multlinear functions) F :=    f(x) = k∈{0,1}d f(k)uk : f F := f(k) λk k∈{0,1}d 2 < ∞    , uk(x) := d =1 uk (x ) G :=    g = k∈{0,1}d ^ g(k)uk : g G := ^ g(k) k∈{0,1}d 2 < ∞    , λk := d =1 k =0 w s APP(f, n) = n i=1 f(ki )uki , λk1 = 1 w1 s = λk2 · · · , 1 = w1 w2 · · · ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1 ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λkn = O n−1/p exp p−1sp d =1 wp =⇒ COST(BR , ε) = O Rpε−p exp sp d =1 wp ∀p cost is independent of d if coordinate weights decay quickly 1Novak, E. & Woźniakowski, H. Tractability of Multivariate Problems Volume I: Linear Information. EMS Tracts in Mathematics 6 (European Mathematical Society, Zürich, 2008). 12/18

Introduction Solvability Smoothness Tractability Cones Design Example References Coordinate Weights Can Save You, Even with Higher Order Polynomials1 For arbitrary d, let {u0 = 1, u1 , u2 , . . .} be used to construct a product basis F and G F :=    f(x) = k∈Nd 0 f(k)uk : f F := f(k) λk k∈Nd 0 2 < ∞    , uk(x) := d =1 uk (x ) G :=    g = k∈Nd 0 ^ g(k)uk : g G := ^ g(k) k∈Nd 0 2 < ∞    , λk := d =1 k =0 w sk APP(f, n) = n i=1 f(ki )uki , λk1 = 1 λk2 · · · , 1 = w1 w2 · · · ALG(f, ε) = APP(f, n∗) & n∗ = min{n : λkn+1 ε/R} =⇒ f − ALG(f, ε) G ε ∀f ∈ BR λkn = O n−1/p exp p−1 ∞ k=1 sp k d =1 wp =⇒ COST(BR , ε) = O Rpε−p exp ∞ k=1 sp k d =1 wp ∀p cost is independent of d if coordinate and smoothness weights decay quickly 1Novak, E. & Woźniakowski, H. Tractability of Multivariate Problems Volume I: Linear Information. EMS Tracts in Mathematics 6 (European Mathematical Society, Zürich, 2008). 12/18

Introduction Solvability Smoothness Tractability Cones Design Example References Look to Cones for Adaptive Algorithms Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R f − ALG(f, ε) G ε ∀ε > 0, f ∈ H ⊆ F (Banach space) So far, H = BR Hard to know a priori how large R should be for your problem Computational cost depends on R and ε, but not on f data Choosing H = makes adaptive algorithms possible2 2H., F. J., Jiménez Rugama, Ll. A. & Li, D. in Contemporary Computational Mathematics — a celebration of the 80th birthday of Ian Sloan (eds Dick, J., Kuo, F. Y. & Woźniakowski, H.) 597–619 (Springer-Verlag, 2018). doi:10.1007/978-3-319-72456-0, Kunsch, R. J., Novak, E. & Rudolf, D. Solvable Integration Problems and Optimal Sample Size Selection. Journal of Complexity. To appear (2018), Ding, Y., H., F. J. & Jiménez Rugama, Ll. A. An Adaptive Algorithm Employing Continuous Linear Functionals. in Monte Carlo and Quasi-Monte Carlo Methods: MCQMC, Lille, France, July 2018 (eds L’Ecuyer, P. & Tuffin, B.) Under revision (Springer-Verlag, Berlin, 2019+), Jagadeeswaran, R. & H., F. J. Fast Automatic Bayesian Cubature Using Lattice Sampling. Stat. Comput. Under revision (2019+). 13/18

Introduction Solvability Smoothness Tractability Cones Design Example References Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3 F := f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 2 λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )uki : g G := ^ g 2 , APP(f, n) = n i=1 f(ki )uki 3Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 14/18

Introduction Solvability Smoothness Tractability Cones Design Example References Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3 F := f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 2 λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )uki : g G := ^ g 2 , APP(f, n) = n i=1 f(ki )uki Cd,λ,n1,A := f ∈ F : f F A f(ki ) λki n1 i=1 2 pilot sample bounds the norm of the input A is inflation factor, n1 is initial sample size f − APP(f, n) G  A2 f(ki ) λki n1 i=1 2 2 − f(ki ) λki n i=1 2 2   1/2 upper bound on f− n i=1 f(ki)uki F λkn+1 =: ERR f(ki ) n i=1 , n data-driven 3Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 14/18

Introduction Solvability Smoothness Tractability Cones Design Example References Adaptive Algorithm for Cone of Inputs Based on Pilot Sample3 F := f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 2 λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )uki : g G := ^ g 2 , APP(f, n) = n i=1 f(ki )uki Cd,λ,n1,A := f ∈ F : f F A f(ki ) λki n1 i=1 2 pilot sample bounds the norm of the input A is inflation factor, n1 is initial sample size f − APP(f, n) G  A2 f(ki ) λki n1 i=1 2 2 − f(ki ) λki n i=1 2 2   1/2 upper bound on f− n i=1 f(ki)uki F λkn+1 =: ERR f(ki ) n i=1 , n data-driven ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki ) n i=1 , n ε} 3Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 14/18

Introduction Solvability Smoothness Tractability Cones Design Example References Adaptive Algorithm for Cone of Inputs Based on Pilot Sample F := f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 2 λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )uki : g G := ^ g 2 , APP(f, n) = n i=1 f(ki )uki Cd,λ,n1,A := f ∈ F : f F A f(ki ) λki n1 i=1 2 pilot sample bounds the norm of the input A is inflation factor, n1 is initial sample size f − APP(f, n) G  A2 f(ki ) λki n1 i=1 2 2 − f(ki ) λki n i=1 2 2   1/2 upper bound on f− n i=1 f(ki)uki F λkn+1 =: ERR f(ki ) n i=1 , n data-driven ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki ) n i=1 , n ε} COST(ALG, Cd,λ,n1,A , ε, R) = max n∗(f, ε) : f ∈ Cλ,n1,A ∩ BR = ∩ = min n n1 : λkn+1 ε/[(A2 − 1)1/2R] ALG is essentially optimal; computational cost is d independent if λki decay quickly 14/18

Introduction Solvability Smoothness Tractability Cones Design Example References Challenges When Using Function Values as Information Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R f − ALG(f, ε) G ε ∀ε > 0, f ∈ H ⊆ F (Banach space) So far, the function information is series coefficients COST(f, ε) = O n∗(f, ε) $(f) , the best one can hope for Cost of constructing the approximation and determining the stopping sample size is essentially the same as getting the data But using series coefficients is not so realistic Developing theory for multivariate function approximation using function values is challenging One must bound the aliasing effects of using interpolation or other means to approximate the coefficients Interpolation, reproducing kernel Hilbert space methods, and kriging typically require O(n3) operations to compute approximation, perhaps more if one is tuning the parameters of the kernels; but there are efforts to speed this up3 Space filling designs such as integration lattices4, digital nets5, and sparse grids6 are promising 3Schäfer, F., Sullivan, T. J. & Owhadi, H. Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity. arXiv:1706.02205. 2019+. 4Dick, J., Kuo, F. & Sloan, I. H. High dimensional integration — the Quasi-Monte Carlo way. Acta Numer. 22, 133–288 (2013). 5Dick, J. & Pillichshammer, F. Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration. (Cambridge University Press, Cambridge, 2010). 6Bungartz, H.-J. & Griebel, M. Sparse grids. Acta Numer. 1–123 (2004). 15/18

Introduction Solvability Smoothness Tractability Cones Design Example References Cheng and Sandu Function7 Chebyshev polynomials, Coordinate weights w inferred, Smoothness weights sk inferred function values used 7Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+), Bingham, D. The Virtual Library of Simulation Experiments: Test Functions and Data Sets. 2017. https://www.sfu.ca/~ssurjano/index.html (2017). 16/18

Introduction Solvability Smoothness Tractability Cones Design Example References Highlights Goal: Construct ALG such that given a black box providing information about f : Ω ⊂ Rd → R f − ALG(f, ε) G ε ∀ε > 0, f ∈ H ⊆ F (Banach space) Impossible for infinite dimensional Banach space H = F, so what is H? Smoothness assumed by F speeds up ALG, not surprising Smoothness alone cannot save from the curse of dimensionality, but a low effective-dimension structure can Choosing H to be a cone , rather than a ball , paves the way for adaptive algorithms Interesting design (where to sample) problems remain 17/18

Thank you These slides are available at speakerdeck.com/fjhickernell/compmath-stat-2019-aug

Introduction Solvability Smoothness Tractability Cones Design Example References Novak, E. & Woźniakowski, H. Tractability of Multivariate Problems Volume I: Linear Information. EMS Tracts in Mathematics 6 (European Mathematical Society, Zürich, 2008). H., F. J., Jiménez Rugama, Ll. A. & Li, D. in Contemporary Computational Mathematics — a celebration of the 80th birthday of Ian Sloan (eds Dick, J., Kuo, F. Y. & Woźniakowski, H.) 597–619 (Springer-Verlag, 2018). doi:10.1007/978-3-319-72456-0. Kunsch, R. J., Novak, E. & Rudolf, D. Solvable Integration Problems and Optimal Sample Size Selection. Journal of Complexity. To appear (2018). Ding, Y., H., F. J. & Jiménez Rugama, Ll. A. An Adaptive Algorithm Employing Continuous Linear Functionals. in Monte Carlo and Quasi-Monte Carlo Methods: MCQMC, Lille, France, July 2018 (eds L’Ecuyer, P. & Tuffin, B.) Under revision (Springer-Verlag, Berlin, 2019+). Jagadeeswaran, R. & H., F. J. Fast Automatic Bayesian Cubature Using Lattice Sampling. Stat. Comput. Under revision (2019+). Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 18/18

Introduction Solvability Smoothness Tractability Cones Design Example References Schäfer, F., Sullivan, T. J. & Owhadi, H. Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity. arXiv:1706.02205. 2019+. Dick, J., Kuo, F. & Sloan, I. H. High dimensional integration — the Quasi-Monte Carlo way. Acta Numer. 22, 133–288 (2013). Dick, J. & Pillichshammer, F. Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration. (Cambridge University Press, Cambridge, 2010). Bungartz, H.-J. & Griebel, M. Sparse grids. Acta Numer. 1–123 (2004). Bingham, D. The Virtual Library of Simulation Experiments: Test Functions and Data Sets. 2017. https://www.sfu.ca/~ssurjano/index.html (2017). 18/18