1/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Isospectralization, or how to hear shape, style, and correspondence Sricharan Chiruvolu Zorah Lähner June 24, 2020 Sricharan Chiruvolu Isospectralization

2/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Introduction Published in CVPR 2019. Explores the practical possibility of using the spectrum of an object for shape reconstruction. Proposes an inverse mapping between geometric domain and its Laplacian via simple regularizers. Isospectralization: A numerical optimization procedure to deform a mesh in order to align its (finite) Laplacian spectrum with a given one. Sricharan Chiruvolu Isospectralization

3/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Isospectralization Figure 1: Isospectralization Sricharan Chiruvolu Isospectralization

4/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Spectral Geometry Relationship between geometric structures of manifolds and spectra of canonically defined differential operators. Eigenvalue problems involving the Laplace operator on manifolds have proven to be a consistently fertile area of geometric analysis with deep connections to number theory, physics, and applied mathematics. Sricharan Chiruvolu Isospectralization

5/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Manifolds (As discussed in previous talks) In computer vision, the notion of a manifold is used quite frequently. in rotation averaging (SO3), structure and motion (“Essential” manifold) to capture the shape of an object (“Shape” manifolds) to model a set of images (“Grassman” manifolds) to simply represent a sphere (Sn). ... Sricharan Chiruvolu Isospectralization

6/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Riemannian Manifold and Geodesics (As discussed in previous talks) A Riemannian manifold is a manifold with an inner product defined in the tangent space at each point. Geodesics are locally shortest curves. They preserve a direction on a surface and have many interesting properties. Sricharan Chiruvolu Isospectralization

7/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Riemannian Manifold and Geodesics Figure 2: Geodesic and tangent space in a Riemannian manifold. 1 1Medinria: DT-MRI processing and visualization software; Xavier Pennec. Sricharan Chiruvolu Isospectralization

8/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Isometry and Isomorphism Figure 3: Isometry and Isomorphism.2 2Smooth Shells: Multi-Scale Shape Registration with Functional Maps; Eisenberger, Lahner and Cremers. Sricharan Chiruvolu Isospectralization

9/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Laplace-Beltrami Operator Figure 4: Laplace-Beltrami operator 3 Laplacian ( f ) = div(gradient(f )). The Laplacian is generalized to the Riemannian manifold (M; g) by the Laplace-Beltrami operator ( g ). 3Khan Academy; Grant Sanderson. Sricharan Chiruvolu Isospectralization

10/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Isospectrals Given a compact Riemannian manifold, we can associate to it the Laplace-Beltrami operator. Two closed Riemannian manifolds are said to be isospectral if the eigenvalues of their Laplace–Beltrami operator (Laplacians) coincide. Thus, spectral geometry is the connection between the spectrum Spec(M; g), i.e. the eigenvalues, and the geometry of the manifold (M; g). This fundamentally deals with two kinds of problems: Direct problems Inverse problems Sricharan Chiruvolu Isospectralization

11/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Riemannian Manifold and Geodesics Isometry and Isomorphism Laplace-Beltrami Operator Direct and Inverse Problems Direct and Inverse Problems Direct Problems Direct problems attempt to infer the behavior of the eigenvalues of a Riemannian manifold from knowledge of the geometry. Compute (exactly or not) the spectrum Spec(M, g)? And (or) find properties on the spectrum Spec(M, g)? Inverse Problems Inverse problem - does the data of the spectrum Spec(M, g) determine the "shape" of the manifold (M, g) ? Which geometric information of the manifold can we determine from the spectrum? Sricharan Chiruvolu Isospectralization

12/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Hearing the Shape of a Drum Figure 5: Membranes of these two different shapes (but otherwise identical) would "sound" the same. "Can one hear the shape of a drum?" Mark Kac’s 1966 question, "Can one hear the shape of a drum?" set off research into spectral theory, with the idea of understanding the extent to which the spectrum allows one to read back the geometry. Sricharan Chiruvolu Isospectralization

13/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Hearing the Shape of a Drum Theoretically Negative. Why? - There exists isospectral manifolds that are nonisometric. With additional priors, practically feasible. We can recover the structure of an object from its spectrum. Sricharan Chiruvolu Isospectralization

14/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Notations Figure 6: Notations Discretization. Sricharan Chiruvolu Isospectralization

15/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Isospectralization Find the optimal embedding that aligns the spectrum to another: min V∈Rn×d λ (∆X (V)) − µ ω + ρX (V) V is the (unknown) embedding of the mesh vertices in R. ∆X (V) is the associated discrete Laplacian. Sricharan Chiruvolu Isospectralization

16/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Flat Shape Regularizers Encourage Smaller Boundary Edge Lengths ρX (V) = eij ∈Eb lij (V) Penalize Triangle Flips ρX (V ) = (min (0, rX (V )))2 rX (V ) = ijk∈F Rπ/22 vj − vi T vk − vi Sricharan Chiruvolu Isospectralization

17/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Flat Shape Regularizers Encourage Small Interior Edge Lengths ρX (V) = eij ∈Ei l2 ij (V) Note: Optimization is effected by the mesh resolution and spectral bandwidth. Sricharan Chiruvolu Isospectralization

18/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Shape reconstruction on Flat Shapes Figure 7: Shape recovery optimization for flat shapes. Sricharan Chiruvolu Isospectralization

19/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Surface Regularizers Encourage Vertices to Lie Close to Barycenter of neighbors, L is the initial graph Laplacian. ρX (V) = Lg 0 V 2 2 Encourage small total displacement from initial embedding - volume to grow/shrink to disambiguate isometric shapes. ρX (V) = n i=1 ai vi − vi 0 2 Sricharan Chiruvolu Isospectralization

20/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Hearing the Shape of a Drum Isospectralization Regularizers for flat shapes Regularizers for surfaces shapes Shape reconstruction on Surfaces Figure 8: Two isometric shapes ("Surfaces") with different volume; their (identical) spectra. This is the reason we need the second Regularizer. Sricharan Chiruvolu Isospectralization

21/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Non-isometric shape matching Style Transfer Non-isometric shape matching Figure 9: Isospectralization as pre-processing for dense correspondence matching. Sricharan Chiruvolu Isospectralization

22/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Non-isometric shape matching Style Transfer Style Transfer Figure 10: Isospectralization for Style Transfer Sricharan Chiruvolu Isospectralization

23/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary Summary Improvement for modern shape matching problems via addressing a decades-old problem. Applications in various domains of geometry processing and computer vision. No mathematical backing on optimization yet. No experimental proof on shape interpolation yet. Sricharan Chiruvolu Isospectralization

24/24 Introduction Background: Spectral Geometry Approach: Isospectralization Applications/Results Summary End Quick Demo. Questions. Thanks for listening! Sricharan Chiruvolu Isospectralization