An Optimal Adaptive Algorithm Based on a Pilot Sample Fred J. Hickernell Department of Applied Mathematics Center for Interdisciplinary Scientific Computation Illinois Institute of Technology [email protected] mypages.iit.edu/~hickernell with Yuhan Ding, Peter Kritzer, and Simon Mak partially supported by NSF-DMS-1522687 and NSF-DMS-1638521 (SAMSI) Twelfth International Conference on Monte Carlo Methods and Applications, July 8, 2019

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Context Linear solution operator SOL : F → G, input space F contains functions defined on Ω ⊆ Rd e.g., SOL(f) = f, −∇2 SOL(f) = f, SOL(f) = 0 on boundary Approximation APP(f, n) = n i=1 Li (f)gi , can sample linear functionals SOL(f) − APP(f, n) G SOL − APP(·, n) F→G f F Algorithm ALG(f, ε) = APP f, n∗(f, ε) satisfying SOL(f) − ALG(f, ε) G ε for all f ∈ C ⊂ F 2/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Context Linear solution operator SOL : F → G, input space F contains functions defined on Ω ⊆ Rd e.g., SOL(f) = f, −∇2 SOL(f) = f, SOL(f) = 0 on boundary Approximation APP(f, n) = n i=1 Li (f)gi , can sample linear functionals SOL(f) − APP(f, n) G SOL − APP(·, n) F→G f F Algorithm ALG(f, ε) = APP f, n∗(f, ε) satisfying SOL(f) − ALG(f, ε) G ε for all f ∈ C ⊂ F Solvability: what C? how to determine n∗(f, ε) ∈ N? n∗(f, ε) = min{n : SOL − APP(·, n) F→G ε/R} if f ∈ of radius R Alternatively, assume f ∈ and bound f F or equivalent What you do not see is not much worse than what you see? H., F. J., Jiménez Rugama, & Li, D. in Contemporary Computational Mathematics — a celebration of the 80th birthday of Ian Sloan (eds Dick, J., Kuo, F. Y. & Woźniakowski, H.) 597–619 (Springer-Verlag, 2018), Kunsch, R. J., Novak, E. & Rudolf, D. Solvable Integration Problems and Optimal Sample Size Selection. Journal of Complexity. To appear (2018), Ding, Y., H., F. J. & Jiménez Rugama, An Adaptive Algorithm Employing Continuous Linear Functionals. in Monte Carlo and Quasi-Monte Carlo Methods: MCQMC, Rennes, France, July 2018 (eds L’Ecuyer, P. & Tuffin, B.) Under revision (Springer-Verlag, Berlin, 2019+). 2/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Context Linear solution operator SOL : F → G, input space F contains functions defined on Ω ⊆ Rd e.g., SOL(f) = f, −∇2 SOL(f) = f, SOL(f) = 0 on boundary Approximation APP(f, n) = n i=1 Li (f)gi , can sample linear functionals SOL(f) − APP(f, n) G SOL − APP(·, n) F→G f F Algorithm ALG(f, ε) = APP f, n∗(f, ε) satisfying SOL(f) − ALG(f, ε) G ε for all f ∈ C ⊂ F Solvability: what C? how to determine n∗(f, ε) ∈ N? ×or Optimality: is n∗(f, ε) essentially as small as possible? Tractability: does n∗(f, ε) depend nicely on d? 2/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References General Linear Problems Defined on Series Spaces F :=    f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 ρ    λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )vki : g G := ^ g τ , SOL(f) = ∞ i=1 f(ki )vki , τ ρ Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 3/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Legendre and Chebyshev Bases for Function Approximation Legendre Chebyshev 4/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Adaptive Algorithm for General Linear Problems on Cone of Inputs F :=    f = ∞ i=1 f(ki )uki : f F := f(ki ) λki ∞ i=1 ρ    λk1 λk2 · · · > 0 λ affects convergence rate & tractability G := g = ∞ i=1 ^ g(ki )vki : g G := ^ g τ , SOL(f) = ∞ i=1 f(ki )vki , τ ρ Cλ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input APP(f, n) = n i=1 f(ki )vki optimal for fixed n SOL(f) − APP(f, n) G ERR f(ki ) n i=1 , n ERR f(ki ) n i=1 , n :=  Aρ f(ki ) λki n1 i=1 ρ ρ − f(ki ) λki n i=1 ρ ρ   1/ρ upper bound on f− n i=1 f(ki)uki F λki ∞ i=n+1 ρ SOL − APP(·,n) F→G 1 ρ + 1 ρ = 1 τ data-driven Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 5/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Adaptive Algorithm for General Linear Problems on Cone of Inputs Cλ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input APP(f, n) = n i=1 f(ki )vki optimal for fixed n SOL(f) − APP(f, n) G ERR f(ki ) n i=1 , n ERR f(ki ) n i=1 , n :=  Aρ f(ki ) λki n1 i=1 ρ ρ − f(ki ) λki n i=1 ρ ρ   1/ρ upper bound on f− n i=1 f(ki)uki F λki ∞ i=n+1 ρ SOL − APP(·,n) F→G 1 ρ + 1 ρ = 1 τ data-driven ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki ) n i=1 , n ε} Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 5/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Adaptive Algorithm for General Linear Problems on Cone of Inputs Cλ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input APP(f, n) = n i=1 f(ki )vki optimal for fixed n SOL(f) − APP(f, n) G ERR f(ki ) n i=1 , n ERR f(ki ) n i=1 , n :=  Aρ f(ki ) λki n1 i=1 ρ ρ − f(ki ) λki n i=1 ρ ρ   1/ρ upper bound on f− n i=1 f(ki)uki F λki ∞ i=n+1 ρ SOL − APP(·,n) F→G 1 ρ + 1 ρ = 1 τ data-driven ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki ) n i=1 , n ε} COST(ALG, Cλ,n1 ,A , ε, R) = max n∗(f, ε) : f ∈ Cλ,n1 ,A ∩ BR = ∩ = min n n1 : λki ∞ i=n+1 ρ ε/[(Aρ − 1)1/ρR] Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 5/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Optimality of Algorithm Cλ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input ALG(f, ε) = APP(f, n∗(f, ε)) for n∗(f, ε) = min{n ∈ N : ERR f(ki ) n i=1 , n ε} COST(ALG, Cλ,n1 ,A , ε, R) = max n∗(f, ε) : f ∈ Cλ,n1 ,A ∩ BR = ∩ = min n n1 : λki ∞ i=n+1 ρ ε/[(Aρ − 1)1/ρR] COMP(A(Cλ,n1 ,A , ε, R)) = min{COST(ALG , Cλ,n1 ,A , ε, R) : ALG ∈ A(Cλ,n1 ,A )} COST(ALG, Cλ,n1 ,A , ωε, R), ω = 2A(Aρ − 1)1/ρ A − 1 > 1 via fooling functions ALG is essentially optimal Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 6/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Tractability of Solving General Linear Problems on Cone of Inputs Cd,λd ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input For τ = ρ (ρ = ∞), the problem SOLd : Cd,λd ,n1 ,A → Gd is Strongly polynomial tractable iff there exist i0 ∈ N and η > 0 such that sup d∈N ∞ i=i0 λη d,ki < ∞ Polynomial tractable iff there exist η1 , η2 0 and η3 , K > 0 such that sup d∈N d−η1 ∞ i= Kdη2 λη3 d,ki < ∞ Weakly tractable iff sup d∈N exp(−cd) ∞ i=1 exp −c 1 λd,ki < ∞ for all c > 0 Analogous results exist for τ < ρ (ρ < ∞) Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 7/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References An Alternate Cone, Similar to Our Cone f F := f(ki ) λki ∞ i=1 ρ , λki ↓ 0, f F := f(ki )ζki λki ∞ i=1 ρ , ζk1 = 1, ζki ↓ Cλ,n1 ,A := f ∈ F : f F A f(ki ) λki n1 i=1 ρ pilot sample bounds the norm of the input Cλ,ζ,A := f ∈ F : f F A f F stronger norm is bounded above by weaker norm Cλ,ζ,A ⊆ Cλ,n1 ,A for A A 1 + ζρ kn1+1 1 − (ζkn1+1 A )ρ 1/ρ , ζkn1+1 A < 1 Cλ,n1 ,A ⊆ Cλ,ζ,A for A A ζkn1 8/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References An Alternative Cone, Similar to Our Cone For any f ∈ Cλ,ζ,A it follows that f ρ F − f(ki )ζki λki n1 i=1 ρ ρ = f(ki )ζki λki ∞ i=n1 +1 ρ ρ ζρ kn1+1 f(ki ) λki ∞ i=n1 +1 ρ ρ = ζρ kn1+1   f ρ F − f(ki ) λki n1 i=1 ρ ρ   ζρ kn1+1  A ρ f ρ F − f(ki )ζki λki n1 i=1 ρ ρ   9/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References An Alternative Cone, Similar to Our Cone For any f ∈ Cλ,ζ,A it follows that f ρ F − f(ki )ζki λki n1 i=1 ρ ρ = f(ki )ζki λki ∞ i=n1 +1 ρ ρ ζρ kn1+1 f(ki ) λki ∞ i=n1 +1 ρ ρ = ζρ kn1+1   f ρ F − f(ki ) λki n1 i=1 ρ ρ   ζρ kn1+1  A ρ f ρ F − f(ki )ζki λki n1 i=1 ρ ρ   f ρ F 1 + ζρ kn1+1 1 − (ζkn1+1 A )ρ f(ki )ζki λki n1 i=1 ρ ρ f F A f F A 1 + ζρ kn1+1 1 − (ζkn1+1 A )ρ 1/ρ f(ki ) λki n1 i=1 ρ 9/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References An Alternate Cone, Similar to Our Cone For any f ∈ Cλ,n1 ,A it follows that f F A f(ki ) λki n1 i=1 ρ A ζkn1 f(ki )ζki λki n1 i=1 ρ A ζkn1 f F 10/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Conclusion Summary Adaptive algorithms can be constructed for non-convex, symmetric cones of inputs One possible cone assumes that a pilot sample tells you enough about the norm of the input Information cost and complexity depend on the norm of the input, but the adaptive algorithm does not know this norm There are tractability results for problems defined on cones Our algorithm can be extended to infer the λk in terms of coordinate weights Further Work Need to develop and analyze algorithms based on function values, not series coefficients Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). 11/14

Introduction Algorithm Optimality Tractability Alternate Cone Summary References Cheng and Sandu Function Function values for data Chebyshev polynomial basis, λk inferred , Bingham, D. The Virtual Library of Simulation Experiments: Test Functions and Data Sets. 2017. https://www.sfu.ca/~ssurjano/index.html (2017). 12/14

Thank you These slides are available at speakerdeck.com/fjhickernell/mcm-2019-pilot-sample

Introduction Algorithm Optimality Tractability Alternate Cone Summary References References I H., F. J., Jiménez Rugama, & Li, D. in Contemporary Computational Mathematics — a celebration of the 80th birthday of Ian Sloan (eds Dick, J., Kuo, F. Y. & Woźniakowski, H.) 597–619 (Springer-Verlag, 2018). Kunsch, R. J., Novak, E. & Rudolf, D. Solvable Integration Problems and Optimal Sample Size Selection. Journal of Complexity. To appear (2018). Ding, Y., H., F. J. & Jiménez Rugama, An Adaptive Algorithm Employing Continuous Linear Functionals. in Monte Carlo and Quasi-Monte Carlo Methods: MCQMC, Rennes, France, July 2018 (eds L’Ecuyer, P. & Tuffin, B.) Under revision (Springer-Verlag, Berlin, 2019+). Ding, Y., H., F. J., Kritzer, P. & Mak, S. in RICAM Fall Semester Proceedings (eds H., F. J. & Kritzer, P.) Submitted for publication (DeGruyter, 2019+). Bingham, D. The Virtual Library of Simulation Experiments: Test Functions and Data Sets. 2017. https://www.sfu.ca/~ssurjano/index.html (2017). 14/14