Riesz Representation Theorem

A talk about my favorite theorem

Transcript

1. My Favorite Theorem:
   The Reisz Representation Theorem
   Fred J. Hickernell
   Department of Applied Mathematics Center for Interdisciplinary Scientific Computation
   Office of Research
   Illinois Institute of Technology hickernell@iit.edu mypages.iit.edu/~hickernell
   Thanks to the Illinois Tech SIAM Student Chapter for the invitation
   Thanks to many students and collaborators for teaching me
   Slides available at speakerdeck.com/riesz-representation-theorem
   Please interrupt and ask questions
   October 19, 2021
   

   View Slide

2. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Main Idea
   What seems obvious for two-dimensional vectors becomes a power tool for numerical analysis
   2/17
   

   View Slide

3. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Main Idea
   What seems obvious for two-dimensional vectors becomes a power tool for numerical analysis
   Obvious If LINEAR : R2 → R is any linear, real-valued function, meaning,
   LINEAR(cf + h) = c LINEAR(f) + LINEAR(h) ∀f, h ∈ R2, c ∈ R,
   then LINEAR(f) can be represented as an inner product: ∃ coefficient g ∈ R2 such that
   LINEAR(f) = g1
   f1
   + g2
   f2
   = gTf =: ⟨g, f⟩ ≡ g • f ∀f ∈ R2.
   2/17
   

   View Slide

4. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Main Idea
   What seems obvious for two-dimensional vectors becomes a power tool for numerical analysis
   Obvious If LINEAR : R2 → R is any linear, real-valued function, meaning,
   LINEAR(cf + h) = c LINEAR(f) + LINEAR(h) ∀f, h ∈ R2, c ∈ R,
   then LINEAR(f) can be represented as an inner product: ∃ coefficient g ∈ R2 such that
   LINEAR(f) = g1
   f1
   + g2
   f2
   = gTf =: ⟨g, f⟩ ≡ g • f ∀f ∈ R2.
   Power Tool Generalization—Riesz Representation Theorem—gives error bounds for numerical
   algorithms, e.g.,
   LINEAR(f)
   [0,1]d
   f(t) dt −
   1
   n
   n
   i=1
   f(xi
   ) ⩽ BAD(x1
   , . . . , xn
   ) BAD(f)
   2/17
   

   View Slide

5. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Main Idea
   What seems obvious for two-dimensional vectors becomes a power tool for numerical analysis
   Obvious If LINEAR : R2 → R is any linear, real-valued function, meaning,
   LINEAR(cf + h) = c LINEAR(f) + LINEAR(h) ∀f, h ∈ R2, c ∈ R,
   then LINEAR(f) can be represented as an inner product: ∃ coefficient g ∈ R2 such that
   LINEAR(f) = g1
   f1
   + g2
   f2
   = gTf =: ⟨g, f⟩ ≡ g • f ∀f ∈ R2.
   Power Tool Generalization—Riesz Representation Theorem—gives error bounds for numerical
   algorithms, e.g.,
   LINEAR(f)
   [0,1]d
   f(t) dt
   average of f
   e.g., option price
   −
   1
   n
   n
   i=1
   f(xi
   )
   average of f values
   e.g., payoffs under various scenarios
   ⩽ BAD(x1
   , . . . , xn
   ) BAD(f)
   2/17
   

   View Slide

6. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Main Idea
   What seems obvious for two-dimensional vectors becomes a power tool for numerical analysis
   Obvious If LINEAR : R2 → R is any linear, real-valued function,
   then LINEAR(f) can be represented as an inner product
   Power Tool Generalization—Riesz Representation Theorem—gives error bounds for numerical
   algorithms, e.g.,
   LINEAR(f)
   [0,1]d
   f(t) dt
   average of f
   e.g., option price
   −
   1
   n
   n
   i=1
   f(xi
   )
   average of f values
   e.g., payoffs under various scenarios
   ⩽ BAD(x1
   , . . . , xn
   )
   Concentrate on
   choosing the xi
   well
   BAD(f)
   2/17
   

   View Slide

7. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Why Is this My Favorite Theorem?
   [0,1]d
   f(t) dt −
   1
   n
   n
   i=1
   f(xi
   ) ⩽ BAD(x1
   , . . . , xn
   ) BAD(f),
   My most cited paper [1] according to Google Scholar is a simple application of the Riesz Representation
   Theorem
   3/17
   

   View Slide

8. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Riesz Representation Theorem for R2
   Theorem
   If LINEAR : R2 → R is any linear, real-valued function, and ⟨h, f⟩ := h1
   f1
   + h2
   f2
   = hTf, then there exists a
   unique representer g ∈ R2, dependent on LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ R2.
   Proof.
   Existence. Let e1
   = (1, 0)T and e2
   = (0, 1)T. Then for all f = (f1
   , f2
   )T,
   LINEAR(f) = LINEAR e1
   f1
   + e2
   f2
   = LINEAR(e1
   )f1
   + LINEAR(e2
   )f2
   by linearity
   = ⟨g, f⟩ , where g = LINEAR(e1
   ), LINEAR(e2
   ) T
   4/17
   

   View Slide

9. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
   Riesz Representation Theorem for R2
   Theorem
   If LINEAR : R2 → R is any linear, real-valued function, and ⟨h, f⟩ := h1
   f1
   + h2
   f2
   = hTf, then there exists a
   unique representer g ∈ R2, dependent on LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ R2.
   Proof.
   Existence. Let e1
   = (1, 0)T and e2
   = (0, 1)T. Then for all f = (f1
   , f2
   )T,
   LINEAR(f) = LINEAR e1
   f1
   + e2
   f2
   = LINEAR(e1
   )f1
   + LINEAR(e2
   )f2
   by linearity
   = ⟨g, f⟩ , where g = LINEAR(e1
   ), LINEAR(e2
   ) T
   Example. If LINEAR(e1
   ) = −3 and LINEAR(e2
   ) = 2, then
   LINEAR(f) = −3f1
   , +2f2
   = (−3, 2)T, f .
   4/17
   

   View Slide

10. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for R2
    Theorem
    If LINEAR : R2 → R is any linear, real-valued function, and ⟨h, f⟩ := h1
    f1
    + h2
    f2
    = hTf, then there exists a
    unique representer g ∈ R2, dependent on LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ R2.
    Proof.
    Existence. Done.
    Uniqueness. If g and ˜
    g are both representers, i.e., LINEAR(f) = ⟨g, f⟩ = ⟨ ˜
    g, f⟩ for all f ∈ Rd, then
    0 = LINEAR(g − ˜
    g) − LINEAR(g − ˜
    g) = ⟨g, g − ˜
    g⟩ − ⟨ ˜
    g, g − ˜
    g⟩ = ⟨g − ˜
    g, g − ˜
    g⟩ = ∥g − ˜
    g∥2
    so g − ˜
    g = 0 and g = ˜
    g.
    4/17
    

    View Slide

11. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    The Dot/Inner Product
    My experience with f • h ≡ ⟨f, h⟩
    Geometry (Euclidean, crow flying)
    distance/size/length/norm:
    ∥f∥ := f2
    1
    + f2
    2
    Pythagorean Theorem
    Trigonometry Law of cosines:
    ∥f − h∥2 = ∥f∥2 + ∥h∥2 − 2 ∥f∥ ∥h∥ cos(∡(f, h))
    Physics f • h := ∥f∥ ∥h∥ cos(∡(f, h))
    f • h = f1
    h1
    + f2
    h2
    5/17
    

    View Slide

12. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    The Dot/Inner Product
    My experience with f • h ≡ ⟨f, h⟩
    Geometry (Euclidean, crow flying)
    distance/size/length/norm:
    ∥f∥ := f2
    1
    + f2
    2
    Pythagorean Theorem
    Trigonometry Law of cosines:
    ∥f − h∥2 = ∥f∥2 + ∥h∥2 − 2 ∥f∥ ∥h∥ cos(∡(f, h))
    Physics f • h := ∥f∥ ∥h∥ cos(∡(f, h))
    f • h = f1
    h1
    + f2
    h2
    Right way to think about ⟨f, h⟩ for arbitrary vectors
    spaces V, e.g., spaces of functions.
    Inner product first: ⟨·, ·⟩ : V × V → R satisfying
    ⟨0, 0⟩ = 0, ⟨f, f⟩ > 0 ∀f ̸= 0, ⟨f, h⟩ = ⟨h, f⟩
    ⟨cf + h, k⟩ = c ⟨f, k⟩ + c ⟨h, k⟩
    Then distance and cosine
    ∥f∥ := ⟨f, f⟩
    cos(∡(f, h)) :=
    ⟨f, h⟩
    ∥f∥ ∥h∥
    ∈ [−1, 1] by Cauchy-Schwartz Proof
    Law of cosines follows
    5/17
    

    View Slide

13. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Hilbert space = a vector space (vectors can be added and multiplied by scalars)
    that is complete under ∥·∥ (sequences that should converge do)
    e.g., R is complete, Q is not
    may be infinite dimensional, e.g., all f : [0, 1] → R with a first derivative
    bounded means sup
    f∈V
    |LINEAR(f)|
    ∥f∥
    < ∞ (automatic for finite dimensional V, but look here )
    linear + bounded =⇒ continuous, linear + continuous =⇒ bounded
    Can we prove this theorem without referring to a basis for V?
    6/17
    

    View Slide

14. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Proof.
    Existence. Define ker(LINEAR) = {v ∈ V : LINEAR(v) = 0} as the subspace of V that LINEAR maps into
    0. If ker(LINEAR) = V, then all vectors in V are mapped to 0 and g = 0.
    6/17
    

    View Slide

15. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Proof.
    Existence. Define ker(LINEAR) = {v ∈ V : LINEAR(v) = 0} as the subspace of V that LINEAR maps into
    0. If ker(LINEAR) = V, then all vectors in V are mapped to 0 and g = 0. Otherwise, pick any nonzero
    g⊥
    ∈ {u ∈ V : ⟨u, v⟩ = 0 ∀v ∈ ker(LINEAR)} How? , i.e., g⊥
    is orthogonal to all vectors in ker(LINEAR).
    6/17
    

    View Slide

16. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Proof.
    Existence. Define ker(LINEAR) = {v ∈ V : LINEAR(v) = 0} as the subspace of V that LINEAR maps into
    0. If ker(LINEAR) = V, then all vectors in V are mapped to 0 and g = 0. Otherwise, pick any nonzero
    g⊥
    ∈ {u ∈ V : ⟨u, v⟩ = 0 ∀v ∈ ker(LINEAR)} How? , i.e., g⊥
    is orthogonal to all vectors in ker(LINEAR).
    For any f ∈ V, let h = LINEAR(f)g⊥
    − LINEAR(g⊥
    )f, and note that
    LINEAR(h) = LINEAR LINEAR(f)g⊥
    − LINEAR(g⊥
    )f
    = LINEAR(f) LINEAR(g⊥
    ) − LINEAR(g⊥
    ) LINEAR(f) = 0,
    so h ∈ ker(LINEAR).
    6/17
    

    View Slide

17. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Proof.
    Existence. Define ker(LINEAR) = {v ∈ V : LINEAR(v) = 0} as the subspace of V that LINEAR maps into
    0. If ker(LINEAR) = V, then all vectors in V are mapped to 0 and g = 0. Otherwise, pick any nonzero
    g⊥
    ∈ {u ∈ V : ⟨u, v⟩ = 0 ∀v ∈ ker(LINEAR)} How? , i.e., g⊥
    is orthogonal to all vectors in ker(LINEAR).
    For any f ∈ V, let h = LINEAR(f)g⊥
    − LINEAR(g⊥
    )f, and note that
    LINEAR(h) = LINEAR LINEAR(f)g⊥
    − LINEAR(g⊥
    )f
    = LINEAR(f) LINEAR(g⊥
    ) − LINEAR(g⊥
    ) LINEAR(f) = 0,
    so h ∈ ker(LINEAR). The choice of g⊥
    implies that
    0 = ⟨g⊥
    , h⟩ = LINEAR(f) ⟨g⊥
    , g⊥
    ⟩ − LINEAR(g⊥
    ) ⟨g⊥
    , f⟩ ,
    LINEAR(f) =
    LINEAR(g⊥
    ) ⟨g⊥
    , f⟩
    ⟨g⊥
    , g⊥
    ⟩
    = ⟨g, f⟩ for g :=
    LINEAR(g⊥
    )g⊥
    ∥g∥2
    .
    6/17
    

    View Slide

18. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Riesz Representation Theorem for a Hilbert Space (V, ⟨·, ·⟩)
    Theorem (Riesz Representation Theorem for Hilbert Spaces)
    If LINEAR : V → R is any bounded linear real-valued function on the Hilbert space (V, ⟨·, ·⟩), then there
    exists a unique g ∈ V, called the representer of LINEAR, for which LINEAR(f) = ⟨g, f⟩ for all f ∈ V.
    Proof.
    Existence. Done.
    Uniqueness. Same proof as before.
    6/17
    

    View Slide

19. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    What Can We Do with the Riesz Representation Theorem?
    Theorem (Error Bound for Numerical Integration)
    Let (V, ⟨·, ·⟩) be a Hilbert space of functions on [0, 1]d. Suppose that integration and function evaluation
    are both bounded, linear real-valued functions on V. Then the error of approximating the integral of a
    function in V by the sample mean is
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ = cos ∡(η, f) ∥η∥ ∥f∥ ⩽ ∥η∥ ∥f∥ ∀f ∈ V,
    for some representer η ∈ V that depends on x1
    , . . . , xn
    , but not on f.
    Significance
    Error bound separates the f dependent part from the algorithm dependent part (η)
    Algorithm developers can concentrate on making ∥η∥ small
    Providential if η is nearly orthogonal to f [2], but don’t count on it
    7/17
    

    View Slide

20. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    What Can We Do with the Riesz Representation Theorem?
    Theorem (Error Bound for Numerical Integration)
    Let (V, ⟨·, ·⟩) be a Hilbert space of functions on [0, 1]d. Suppose that integration and function evaluation
    are both bounded, linear real-valued functions on V. Then the error of approximating the integral of a
    function in V by the sample mean is
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ = cos ∡(η, f) ∥η∥ ∥f∥ ⩽ ∥η∥ ∥f∥ ∀f ∈ V,
    for some representer η ∈ V that depends on x1
    , . . . , xn
    , but not on f.
    Proof.
    Note that INT : f →
    [0,1]d
    f(t) dt and AVG : f →
    1
    n
    n
    i=1
    f(xi
    ) are bounded, linear real-valued functions.
    7/17
    

    View Slide

21. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    What Can We Do with the Riesz Representation Theorem?
    Theorem (Error Bound for Numerical Integration)
    Let (V, ⟨·, ·⟩) be a Hilbert space of functions on [0, 1]d. Suppose that integration and function evaluation
    are both bounded, linear real-valued functions on V. Then the error of approximating the integral of a
    function in V by the sample mean is
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ = cos ∡(η, f) ∥η∥ ∥f∥ ⩽ ∥η∥ ∥f∥ ∀f ∈ V,
    for some representer η ∈ V that depends on x1
    , . . . , xn
    , but not on f.
    Proof.
    Note that INT : f →
    [0,1]d
    f(t) dt and AVG : f →
    1
    n
    n
    i=1
    f(xi
    ) are bounded, linear real-valued functions.
    Thus, so is ERR = INT − AVG. By the Reisz Representation Theorem, there exists a representer η ∈ V
    such that
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ERR(f) = ⟨η, f⟩.
    7/17
    

    View Slide

22. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    What Can We Do with the Riesz Representation Theorem?
    Theorem (Preliminary Error Bound for Numerical Integration)
    Let (V, ⟨·, ·⟩) be a Hilbert space of functions on [0, 1]d. Suppose that integration and function evaluation
    are both bounded, linear real-valued functions on V. Then the error of approximating the integral of a
    function in V by the sample mean is
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ = cos ∡(η, f) ∥η∥ ∥f∥ ⩽ ∥η∥ ∥f∥ ∀f ∈ V,
    for some representer η ∈ V that depends on x1
    , . . . , xn
    , but not on f.
    Significance
    Error bound separates the f dependent part from the algorithm dependent part (η)
    Algorithm developers can concentrate on making ∥η∥ small
    Providential if η is nearly orthogonal to f [2], but don’t count on it
    How to find η?
    7/17
    

    View Slide

23. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Reproducing Kernels [3]
    Suppose that (V, ⟨·, ·⟩) is Hilbert space of functions on Ω for which function evaluation is a bounded,
    linear functional. Then there exists, K : Ω × Ω → R called a reproducing kernel for which
    K(t, x) = K(x, t)
    symmetry
    , K(·, x) ∈ V
    belonging
    , f(x) = ⟨K(·, x), f⟩
    reproduction
    ∀t, x ∈ Ω, f ∈ V
    What do reproducing kernels look like for V = Rd
    Look here
    8/17
    

    View Slide

24. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Reproducing Kernels [3]
    Suppose that (V, ⟨·, ·⟩) is Hilbert space of functions on Ω for which function evaluation is a bounded,
    linear functional. Then there exists, K : Ω × Ω → R called a reproducing kernel for which
    K(t, x) = K(x, t)
    symmetry
    , K(·, x) ∈ V
    belonging
    , f(x) = ⟨K(·, x), f⟩
    reproduction
    ∀t, x ∈ Ω, f ∈ V
    Combining with the Riesz Representation Theorem
    ERR(f) :=
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ , representer η =?
    η(x) =
    reproduction
    ⟨K(·, x), η⟩ =
    symmetry
    ⟨η, K(·, x)⟩ =
    representer
    ERR K(·, x)
    8/17
    

    View Slide

25. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Reproducing Kernels [3]
    Suppose that (V, ⟨·, ·⟩) is Hilbert space of functions on Ω for which function evaluation is a bounded,
    linear functional. Then there exists, K : Ω × Ω → R called a reproducing kernel for which
    K(t, x) = K(x, t)
    symmetry
    , K(·, x) ∈ V
    belonging
    , f(x) = ⟨K(·, x), f⟩
    reproduction
    ∀t, x ∈ Ω, f ∈ V
    Combining with the Riesz Representation Theorem
    ERR(f) :=
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ , representer η =?
    η(x) =
    reproduction
    ⟨K(·, x), η⟩ =
    symmetry
    ⟨η, K(·, x)⟩ =
    representer
    ERR K(·, x) =
    [0,1]d
    K(t, x) dt −
    1
    n
    n
    i=1
    K(xi
    , x)
    8/17
    

    View Slide

26. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Reproducing Kernels [3]
    Suppose that (V, ⟨·, ·⟩) is Hilbert space of functions on Ω for which function evaluation is a bounded,
    linear functional. Then there exists, K : Ω × Ω → R called a reproducing kernel for which
    K(t, x) = K(x, t)
    symmetry
    , K(·, x) ∈ V
    belonging
    , f(x) = ⟨K(·, x), f⟩
    reproduction
    ∀t, x ∈ Ω, f ∈ V
    Combining with the Riesz Representation Theorem
    ERR(f) :=
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ , representer η =?
    η(x) =
    reproduction
    ⟨K(·, x), η⟩ =
    symmetry
    ⟨η, K(·, x)⟩ =
    representer
    ERR K(·, x) =
    [0,1]d
    K(t, x) dt −
    1
    n
    n
    i=1
    K(xi
    , x)
    ∥η∥2 = ⟨η, η⟩ =
    representer
    ERR(η)
    8/17
    

    View Slide

27. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Reproducing Kernels [3]
    Suppose that (V, ⟨·, ·⟩) is Hilbert space of functions on Ω for which function evaluation is a bounded,
    linear functional. Then there exists, K : Ω × Ω → R called a reproducing kernel for which
    K(t, x) = K(x, t)
    symmetry
    , K(·, x) ∈ V
    belonging
    , f(x) = ⟨K(·, x), f⟩
    reproduction
    ∀t, x ∈ Ω, f ∈ V
    Combining with the Riesz Representation Theorem
    ERR(f) :=
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) = ⟨η, f⟩ , representer η =?
    η(x) =
    reproduction
    ⟨K(·, x), η⟩ =
    symmetry
    ⟨η, K(·, x)⟩ =
    representer
    ERR K(·, x) =
    [0,1]d
    K(t, x) dt −
    1
    n
    n
    i=1
    K(xi
    , x)
    ∥η∥2 = ⟨η, η⟩ =
    representer
    ERR(η) =
    [0,1]2d
    K(t, x) dt dx −
    2
    n
    n
    i=1 [0,1]d
    K(xi
    , x) dx +
    1
    n2
    n
    i,j=1
    K(xi
    , xj
    )
    8/17
    

    View Slide

28. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Putting It Together
    Theorem (Error Bound for Numerical Integration)
    Let (V, ⟨·, ·⟩) be a Hilbert space of functions on [0, 1]d with reproducing kernel, K. Suppose that
    integration and function evaluation are both bounded, linear real-valued functions on V. Then the error of
    approximating the integral of a function in V by the sample mean is
    [0,1]d
    f(t) dx −
    1
    n
    n
    i=1
    f(xi
    ) ⩽ BAD(x1
    , . . . , xn
    ) BAD(f),
    where
    BAD2(x1
    , . . . , xn
    ) =
    [0,1]2d
    K(t, x) dt dx −
    2
    n
    n
    i=1 [0,1]d
    K(xi
    , x) dx +
    1
    n2
    n
    i,j=1
    K(xi
    , xj
    ),
    BAD(f) = ∥f∥ .
    For an explicit example of a K and BAD(x1
    , . . . , xn
    ) Look here
    9/17
    

    View Slide

29. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Why Do Others Cite This Paper?
    [0,1]d
    f(t) dt −
    1
    n
    n
    i=1
    f(xi
    ) ⩽ BAD(x1
    , . . . , xn
    ) BAD(f),
    You can pick reproducing kernel K well and analyze
    Convergence How fast BAD(x1
    , . . . , xn
    ) → 0 with n for clever x1
    , x2
    , . . .
    Tractability How this convergence depends on d
    10/17
    

    View Slide

30. Thank you
    These slides are available at
    speakerdeck.com/riesz-representation-theorem
    

    View Slide

31. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    References
    1. H., F. J. A Generalized Discrepancy and Quadrature Error Bound. Math. Comp. 67, 299–322 (1998).
    2. H., F. J. The Trio Identity for Quasi-Monte Carlo Error Analysis. in Monte Carlo and Quasi-Monte
    Carlo Methods: MCQMC, Stanford, USA, August 2016 (eds Glynn, P. & Owen, A.) (Springer-Verlag,
    Berlin, 2018), 3–27.
    3. N. Aronszajn. Theory of Reproducing Kernels. Trans. Amer. Math. Soc. 68, 337–404 (1950).
    12/17
    

    View Slide

32. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    How Can a Linear Function on a Hilbert Space be Unbounded? Back
    Short answer: V must be infinite dimensional
    13/17
    

    View Slide

33. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    How Can a Linear Function on a Hilbert Space be Unbounded? Back
    Short answer: V must be infinite dimensional
    Consider a vector space of real-valued sequences with the typical inner product:
    V := {f = (f1
    , f2
    , . . .)T : fi ∈ R, ∥f∥ < ∞}, ⟨f, h⟩ := f1
    h1
    + f2
    h2
    + f3
    h3
    + · · ·
    Define the linear real-valued function
    LINEAR(f) = f1
    + 2f2
    + 3f3
    + 4f4
    + · · ·
    Let ei
    := (0, . . . , 0, 1
    ith position
    , 0, . . .)T. For any ε, δ > 0, choose i > ε/δ. Then ∥δei∥ = δ, but
    sup
    f̸=0
    |LINEAR(f)|
    ∥f∥
    ⩾ sup
    i=1,2,...
    |LINEAR(δei
    )|
    ∥δei∥
    = sup
    i=1,2,...
    iδ
    δ
    = sup
    i=1,2,...
    i = ∞
    Cannot guarantee that |LINEAR(f)| is small enough, no matter how small you make ∥f∥. This LINEAR is
    unbounded.
    13/17
    

    View Slide

34. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Proof of the Cauchy-Schwarz Inequality Back
    Theorem (Cauchy-Schwarz)
    Let (V, ⟨·, ·⟩) be a real-valued inner product space. Then
    ⟨f, h⟩ ⩽ ∥f∥ ∥h∥ ∀f, h ∈ V, (C-S)
    with equality iff c1
    f + c2
    h = 0 for some nonzero (c1
    , c2
    ).
    Proof of Inequality.
    If f or h are zero, the inequality becomes an equality by direct calculation. For any nonzero f, h ∈ V,
    define the quadratic polynomial p as follows:
    p(t) := ∥tf + h∥2 = ⟨tf + h, tf + h⟩ = t2 ⟨f, f⟩ + 2t ⟨f, h⟩ + ⟨h, h⟩ = t2 ∥f∥2 + 2t ⟨f, h⟩ + ∥h∥2 .
    Since p(t) ⩾ 0 by definition, p cannot have two roots, which means that ⟨f, h⟩ 2
    − ∥f∥2 ∥h∥2 must be
    non-positive. This implies the inequality for ⟨f, h⟩ .
    14/17
    

    View Slide

35. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    Proof of the Cauchy-Schwarz Inequality Back
    Theorem (Cauchy-Schwarz)
    Let (V, ⟨·, ·⟩) be a real-valued inner product space. Then
    ⟨f, h⟩ ⩽ ∥f∥ ∥h∥ ∀f, h ∈ V, (C-S)
    with equality iff c1
    f + c2
    h = 0 for some nonzero (c1
    , c2
    ).
    Proof of Equality.
    Recall that
    p(t) := ∥tf + h∥2 = t2 ∥f∥2 + 2t ⟨f, h⟩ + ∥h∥2 .
    Equality in (C-S) happens iff ⟨f, h⟩ = ∥f∥ ∥h∥, which implies that p has a single root, t0
    . Then
    0 = p(t0
    ) = ∥t0
    f + h∥2 ,
    which is true iff t0
    f + h = 0 by the definition of a norm.
    14/17
    

    View Slide

36. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    There Exists a Nonzero g⊥
    Orthogonal to All of ker(V) Back
    Lemma
    If LINEAR : V → R is any bounded, linear functional on the Hilbert space (V, ⟨·, ·⟩), and
    ker(LINEAR) := LINEAR−1({0}) = {f ∈ V : LINEAR(f) = 0} ̸= V, then there exists a nonzero
    g⊥
    ∈ {h ∈ V : ⟨h, f⟩ = 0 ∀f ∈ ker(LINEAR)}.
    Proof.
    Define dist h, ker(LINEAR) := inf{∥h − f∥ : f ∈ ker(LINEAR)}, i.e., the closest ker(LINEAR) comes to h.
    For any h /
    ∈ ker(LINEAR), choose kn ∈ ker(LINEAR) such that ∥h − kn∥2
    ⩽ dist2 h, ker(LINEAR) + 1/n
    for n = 1, 2, . . .. By the parallelogram law,
    ∥km
    − kn∥2 = ∥(km
    − h) − (kn
    − h)∥2 = 2 ∥km
    − h∥2 + 2 ∥kn
    − h∥2 − ∥(km
    − h) + (kn
    − h)∥2
    = 2 ∥km
    − h∥2 + 2 ∥kn
    − h∥2 − 4 ∥(km
    + kn
    )/2 − h)∥2 (km
    + kn
    )/2 ∈ ker(LINEAR))
    ⩽ 2 dist2 h, ker(LINEAR) + 1/m] + 2 dist2 h, ker(LINEAR) + 1/n]
    − 4 dist2 h, ker(LINEAR) = 2(1/m + 1/n)
    So, kn → k ∈ V due to completeness of V; k ∈ ker(LINEAR) due to continuity of LINEAR.
    ∥h − k∥ = lim ∥h − kn∥ = lim dist2 h, ker(LINEAR) + 1/n = dist h, ker(LINEAR) 15/17
    

    View Slide

37. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    There Exists a Nonzero g⊥
    Orthogonal to All of ker(V) Back
    Lemma
    If LINEAR : V → R is any bounded, linear functional on the Hilbert space (V, ⟨·, ·⟩), and
    ker(LINEAR) := LINEAR−1({0}) = {f ∈ V : LINEAR(f) = 0} ̸= V, then there exists a nonzero
    g⊥
    ∈ {h ∈ V : ⟨h, f⟩ = 0 ∀f ∈ ker(LINEAR)}.
    Proof.
    Let g⊥
    = h − k. Since h /
    ∈ ker(LINEAR) and k ∈ ker(LINEAR), then h ̸= k and
    ∥g⊥
    ∥ = ∥h − k∥ = dist h, ker(LINEAR) ̸= 0.
    Pick any f ∈ ker(LINEAR), and note that k + tf ∈ ker(LINEAR) for all t ∈ R. Thus,
    ∥g⊥
    ∥2
    ⩽ ∥h − (k + tf)∥2 = ∥g⊥
    − tf∥2 = ∥g⊥
    ∥2 − 2t ⟨g⊥
    , f⟩ + t2 ∥f∥2
    0 ⩽ t −2 ⟨g⊥
    , f⟩ + t ∥f∥2 ∀t ∈ R
    The only way to ensure this inequality for all t is for ⟨g⊥
    , f⟩ = 0.
    15/17
    

    View Slide

38. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    What Does a Reproducing Kernel Look Like for V = Rd? Back
    The functions have domain Ω := {1, . . . , d}, and are represented as vectors of matrices. Pick a
    symmetric, positive definite (all eigenvalues are positive) matrix A ∈ Rd×d to define the inner product
    ⟨f, h⟩ := fTAh, where f = f(i) d
    i=1
    .
    Then the reproducing kernel, K, is defined by
    K(i, j) d
    i,j=1
    = K := A−1.
    Note that
    K(i, j) = K(j, i) because A is symmetric and thus so is K
    K(·, j) = jth column of K =: Kj ∈ Rd = V
    ⟨K(·, j), f⟩ = KT
    j
    Af = ej
    f = f(j) since K := A−1
    where ej
    := (0, . . . , 0, 1
    jth position
    , 0, . . .)T
    16/17
    

    View Slide

39. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    An Example of the Cubature Error Bound [1] Back
    Consider the reproducing kernel
    K(t, x) :=
    d
    k=1
    1 +
    1
    2
    |tk
    − 1/2| + |xk
    − 1/2| − |tk
    − xk
    | t, x ∈ [0, 1]d
    Then
    BAD2(x1
    , . . . , xn
    ) =
    13
    12
    d
    −
    2
    n
    n
    i=1
    d
    k=1
    1 +
    1
    2
    |xik
    − 1/2| − |xik
    − 1/2|2
    +
    1
    n2
    n
    i,j=1
    d
    k=1
    1 +
    1
    2
    |xik
    − 1/2| + xjk
    − 1/2 − xik
    − xjk
    Requires O(dn2) operations to compute
    17/17
    

    View Slide

40. Background Riesz Rep Thm in R2 Inner Product Riesz Rep Thm for Hilbert Spaces Reproducing Kernels References Deleted Scenes
    An Example of the Cubature Error Bound [1] Back
    Consider the reproducing kernel
    K(t, x) :=
    d
    k=1
    1 +
    1
    2
    |tk
    − 1/2| + |xk
    − 1/2| − |tk
    − xk
    | t, x ∈ [0, 1]d
    which corresponds to the Hilbert space for functions defined on [0, 1]d with the following norm:
    ∥f∥2 := |f(1/2, . . . , 1/2)|2
    +
    [0,1]
    ∂f
    ∂x1
    (x1
    , 1/2, . . . , 1/2)
    2
    dx1
    +
    [0,1]
    ∂f
    ∂x1
    (1/2, x2
    , 1/2, . . . , 1/2)
    2
    dx2
    + · · ·
    +
    [0,1]2
    ∂2f
    ∂x1
    ∂x2
    (x1
    , x2
    , 1/2, . . . , 1/2)
    2
    dx1
    dx2
    + · · · +
    [0,1]d
    ∂df
    ∂x1 · · · ∂xd
    (x)
    2
    dx
    BAD(f) = ∥f − f(1/2, . . . , 1/2)∥
    17/17
    

    View Slide