Ouafae Karmouda

(SIGMA team at CRIStAL laboratory, Lille, France)

https://s3-seminar.github.io/seminars/ouafae-karmouda/

Title — Speeding up of kernel-based learning for high-order tensor
Abstract — Supervised learning is a major task to classify datasets. In our context, we are interested into classification from high-order tensors datasets. The “curse of dimensionality” states that the complexities in terms of storage and computation grow exponentially with the order. As a consequence, the method from the state-of-art based on the Higher-Order SVD (HOSVD) works well but suffers from severe limitation in terms of complexities. In this work, we propose a fast Grassmannian kernel-based method for high-order tensor learning based on the equivalence between the Tucker and the tensor-train decompositions. Our solution is linked to the tensor network, where the aim is to break the initial high-order tensor into a collection of low-order tensors (at most 3-order). We show on several real datasets that the proposed method reaches a similar accuracy classification rate as the Grassmannian kernel-based method based on the HOSVD but for a much lower complexity.

Biography — Ouafae KARMOUDA receives a Master’s degree in Applied Mathematics from the Faculty of Science and Technologies of Fes, Morocco, in 2018. She receives a Master's degree in Data Science from the Aix-Marseille University in 2019. Currently, she is a second year PhD student under the supervision of Rémy Boyer and Jérémie Boulanger in SIGMA team at CRIStAL laboratory, Lille. Her research interests focus on developping/improving Maching learning Algorithms for multidimensional data (tensors). She is particulary interested in kernel methods and Deep Learning techniques for high dimensional data. The challenge when dealing with tensors lies in the « curse of dimensionality ». In other words, the computational complexity grows exponentially with the order of tensors. To mitigate this issue, she is interested in Tensor network models and particularly the Tensor Train decomposition.

Transcript

1. SPEEDING UP OF KERNEL-BASED LEARNING FOR
   HIGH-ORDER TENSOR
   12/03/2020
   Presented by : Ouafae Karmouda
   ouafae.karmouda@univ-lille.fr
   Supervised by : R´
   emy Boyer and J´
   er´
   emie Boulanger
   Ouafae Karmouda 12/03/2020 1 / 24
   

   View Slide

2. 1 Introduction
   2 Background in Tensor Algebra
   3 Method of the sate of the art: A Kernel-based Framework to tensorial
   Data Analysis
   4 Fast Kernel Subspace Estimation based on Tensor Train Decomposition
   5 Numerical Experiments
   6 Conclusion
   Ouafae Karmouda 12/03/2020 2 / 24
   

   View Slide

3. Introduction
   Applications of Support Vector Machines (SVMs)
   Ouafae Karmouda 12/03/2020 3 / 24
   

   View Slide

4. Introduction
   Principle of SVMs
   Figure: SVMs looks for the optimal hyperplane to separate the classes.
   SVMs assume data is linearly separable.
   Parameters of the hyperplane can be computed by soving a quadratic
   optimization problem where the inner product between data samples
   is needed.
   Ouafae Karmouda 12/03/2020 4 / 24
   

   View Slide

5. Introduction
   SVMs and Kernel functions
   Figure: Projection of data in a feature space.
   Kernel trick : k(., .) =< φ(.), φ(.) >
   Examples of kernel functions for vectors:
   Radius-Basis function kernel (RBF): k(x, y) = exp(−γ||x − y||2).
   Polynomial kernel : k(x, y) = (xT y + c)d .
   Ouafae Karmouda 12/03/2020 5 / 24
   

   View Slide

6. Introduction
   How to define kernel functions for tensors ?
   Ouafae Karmouda 12/03/2020 6 / 24
   

   View Slide

7. Background in Tensor Algebra
   What is a tensor ?
   Algebraic view :A tensor is a multidimensional array.
   The order of a tensor is the number of its dimensions, also known as
   modes or ways.
   Figure: Different orders of a Tensor
   Ouafae Karmouda 12/03/2020 7 / 24
   

   View Slide

8. Background in Tensor Algebra
   Fibers and Slices of a 3rd-order tensor
   Figure: Top: Fibers of a 3rd-order tensor. Down: Slices of a 3rd-order tensor [G.
   Kolda and W. Bader, SIAM 2009].
   Ouafae Karmouda 12/03/2020 8 / 24
   

   View Slide

9. Background in Tensor Algebra
   Matrix representation of a higher-order tensor
   Figure: Mode-1, mode-2 and mode-3 matricization (unfolding) of a 3rd ord
   tensor. [Y.Chen, R.Xu ,MIPPR 2009]
   Ouafae Karmouda 12/03/2020 9 / 24
   

   View Slide

10. Background in Tensor Algebra
    Tensor n-Mode Multiplication
    Let X ∈ RI1×I2×...×IQ , U ∈ RJ×In , the n-mode product is denoted by
    Y = X ×n U. and its elements are given by:
    Y is of size I1 × · · · In−1 × J × In+1 × · · · × IN,
    Yi1,...,in−1,jn,in+1,...,iN
    =
    In
    in=1
    xi1...iN
    ujin
    .
    In terms of unfolded tensors:
    Y = X ×n
    U ⇒ Y(n)
    = UX(n)
    .
    Example : X ∈ RI1×I2×I3 , B ∈ RM×I2 .
    Y = X ×2
    B ⇒ Y(2)
    = BX(2)
    Ouafae Karmouda 12/03/2020 10 / 24
    

    View Slide

11. Background in Tensor Algebra
    Higher-Order SVD (HOSVD)
    Theorem
    Every complex (I1 × · · · × IQ)-tensor can be approximated by [L.De
    Lathauwer, B.De Moor et al, SIAM 2000]:
    X ≈ G ×1 U1 ×2 ... ×Q UQ, (1)
    Uq is an Iq × Tq orthonormal matrix.
    G is a (T1 × · · · × TQ) all-orthogonal tensor.
    Tq are multilinear ranks of X.
    The q-mode singular matrix Uq is the left singular matrix of the
    q-mode matrix unfolding.
    The complexity of HOSVD is O(QTIQ).
    G ≈ X ×1 UT
    1
    ×2 ... ×Q UT
    Q
    . (2)
    Ouafae Karmouda 12/03/2020 11 / 24
    

    View Slide

12. Background in Tensor Algebra
    Visual illustration of the HOSVD of 3rd order tensor
    Figure: Visualisation of HOSVD of a multilinear rank-(R1,R2,R3) tensor and the
    different spaces. [Multilinear singular value decomposition and low multilinear
    rank approximation, Tensorlab]
    Ouafae Karmouda 12/03/2020 12 / 24
    

    View Slide

13. Background in Tensor Algebra
    Question: How to define a similarity measure based on the
    mulidimensional structure of input tensors?
    Possible answer:
    *Regarding the input tensor as the collection of linear subspaces
    coming from each matricization.
    *Define a kernel between subspaces in a Grassmann manifold.
    Ouafae Karmouda 12/03/2020 13 / 24
    

    View Slide

14. Method of the sate of the art: A Kernel-based Framework to
    tensorial Data Analysis
    Grassmann Manifold
    For integers n ≥ k > 0, the Grassmann Manifold is defined by:
    G(n, k) = {span(M) : M ∈ Rn×kMT M = Ik}.
    Figure: Example of a Grassmann Manifold. X, Y , Z: Points on the Grassmann
    manifold: subspaces. [Grassmannian Learning, 2018].
    Ouafae Karmouda 12/03/2020 14 / 24
    

    View Slide

15. Method of the sate of the art: A Kernel-based Framework to
    tensorial Data Analysis
    Kernel on a Grassmann manifold
    Consider the HOSVD of X, Y ∈ RI1×···×IQ ,
    X = G ×1 U1 ×2 ... ×Q UQ (3)
    Y = H ×1 V1 ×2 ... ×Q VQ (4)
    The kernel-based part of the proposed method in [M.Signoretto, L.De
    Lathauwer, J. Suykens, 2011] is :
    k(X, Y) =
    Q
    q=1
    ˜
    k (span(Uq), span(Vq)) , (5)
    where span(Uq), span(Vq) ∈ G(Iq, Tq) and,
    ˜
    k (span(Uq), span(Vq)) = exp −2γ UqUT
    q
    − VqV T
    q
    2
    F
    .
    Ouafae Karmouda 12/03/2020 15 / 24
    

    View Slide

16. Method of the sate of the art: A Kernel-based Framework to
    tensorial Data Analysis
    Limitation of the method of the state if the art
    The complexity of HOSVD is O(QRIQ).
    The limitation becomes severe for higher-order tensors.
    Objectif: Reduce the complexity of the HOSVD.
    Mean : Use an algebraic equivalence between HOSVD and the
    structured Tensor Train Decomposition (TTD).
    What is TTD ?
    Ouafae Karmouda 12/03/2020 16 / 24
    

    View Slide

17. Fast Kernel Subspace Estimation based on Tensor Train
    Decomposition
    Tensor-Train Decomposition (TTD)
    X(i1, . . . , iQ) =
    R1,··· ,RQ
    r1,...,rQ−1
    G1(i1, r1)G2(r1, i2, r2) . . .
    GQ−1(rQ−2, iQ−1, rQ−1)GQ(rQ−1, iQ),
    GQ ∈ RRQ−1×IQ .
    Gq ∈ RRq−1×Iq×Rq , q ∈ {2, . . . , Q − 1},
    G1 ∈ RI1×R1 .
    Figure: TT decomposition of a D-order Tensor.[I.V.Oseledets, SIAM 2011]
    Ouafae Karmouda 12/03/2020 17 / 24
    

    View Slide

18. Fast Kernel Subspace Estimation based on Tensor Train
    Decomposition
    Key property
    HOSVD : X = G ×1 U1 ×2 ... ×Q UQ,
    TTD : X = G1 ×1
    2
    G2 · · · ×1
    Q−1
    GQ−1 ×1
    Q
    GQ
    Interesting property:
    span(U1) = span(G1), span(UQ) = span(GT
    Q
    ), span(Uq) = span (Fq) ,
    whereFq = Matrix of the left singular vectors of SVD of (Gq)(2)
    .
    Figure: In the case of Tq
    = 2 with Uq
    = [U1
    q
    , U2
    q
    ]
    Ouafae Karmouda 12/03/2020 18 / 24
    

    View Slide

19. Fast Kernel Subspace Estimation based on Tensor Train
    Decomposition
    Equivalence TTD and HOSVD
    Assume that tensor X follows a Q-order HOSVD of multilinear
    rank-(T1, · · · , TQ). A TTD of X is given by [Zniyed, Boyer et al, LAA]:
    G1 = U1
    Gq = Tq ×2 Uq(1 < q < ¯
    q)
    with Tq = reshape(IRq
    ; T1 . . . Tq−1, Tq, T1 · · · Tq)
    G¯
    q = G¯
    q ×2 U¯
    q(1 < q < ¯
    q)
    with G¯
    q = reshape(G; R ¯
    Q−1
    , T¯
    q, R¯
    q)
    Gq = Tq ×2 Uq(¯
    q < q < Q)
    with ¯
    Tq = reshape(IRq−1
    ; Tq . . . TQ, Tq, Tq+1 · · · TQ)
    GQ = UT
    Q
    ¯
    q is the smallest q that verifies Q
    i=1
    Ti ≥ Q
    i=q+1
    Ti
    Ouafae Karmouda 12/03/2020 19 / 24
    

    View Slide

20. Fast Kernel Subspace Estimation based on Tensor Train
    Decomposition
    FAKSETT: Fast Kernel Subspace Estimation based on
    Tensor Train decomposition
    Consider the TTD of X, Y ∈ RI1×···×IQ ,
    X = G1 ×1
    2
    G2 · · · ×1
    Q−1
    GQ−1 ×1
    Q
    GQ (6)
    Y = G1
    ×1
    2
    G 2 · · · ×1
    Q−1
    G Q−1 ×1
    Q
    GQ
    (7)
    The kernel-based part of the proposed method is [Karmouda, Boulanger,
    Boyer, ICASSP 2021]:
    k(X, Y) =
    Q
    q=1
    ˜
    k span(Fq), span(Fq
    ) , (8)
    where Fq = Matrix of the left singular vectors of (Gq)(2)
    ,
    Fq
    = Matrix of the left singular vectors of (G q)(2)
    and
    ˜
    k span(Fq), span(Fq
    ) =
    exp −2γ (Fq)(FT
    q
    ) − (Fq
    )(Fq
    )T
    2
    F
    .
    Ouafae Karmouda 12/03/2020 20 / 24
    

    View Slide

21. Numerical Experiments
    Classification performance for the UCF11 dataset
    UCF11 : Composed of videos that contain human actions of size
    240
    frames
    × 240 × ×320 × 3
    dim.frames
    .
    Figure: Two human actions considered.
    s% m-ranks FAKSETT native method
    %50 [2,2,2,2] 0.72(10−2) 0.73(10−2)
    %60 [3,3,3,3] 0.7(10−2) 0.7(10−2)
    %80 [3,3,3,3] 0.76(10−2) 0.77(10−2)
    Table: Mean accuracy (standard deviation) on test data for UCF11 database
    Ouafae Karmouda 12/03/2020 21 / 24
    

    View Slide

22. Numerical Experiments
    Classification Performance for Extended Yale dataset
    Extended Yale dataset : This dataset contains images of size
    9
    nb.poses
    × 480 × 640
    dim.images
    × 16
    nb.illum
    of 28 human subjects.
    Figure: 3 classes of Extended Yale dataset.
    s% m-ranks FAKSETT native method
    %50 [1,3,2,1] 0.98(10−2) 0.99(10−2)
    %60 [1,2,2,1] 0.99(10−2) 0.99(10−2)
    Table: Mean accuracy (standard deviation) on test data for Extended Yale
    database
    Ouafae Karmouda 12/03/2020 22 / 24
    

    View Slide

23. Numerical Experiments
    Computational time
    Database m-ranks FAKSETT native method
    UCF11
    [2,2,2,2] 14(0.42) 69(3)
    [3,3,3,3] 15(0.63) 104(5)
    Extended Yale [1,2,2,1] 2.56(0.09) 9.47(0.1)
    Table: Mean time (standard deviation) on seconds consumed to compute HOSVD
    for different databases w.r.t to different values of multi-linear ranks.
    Ouafae Karmouda 12/03/2020 23 / 24
    

    View Slide

24. Conclusion
    Conclusion
    Despite of a good classification, the method of the state of the art
    suffers from a high complexity cost.
    Exploit some algebraic link beween TTD and HOSVD to speed up the
    native method.
    We have proposed the FAKSETT method.
    FAKSETT reaches similar scores and considerably reduces
    computational time.
    Ouafae Karmouda 12/03/2020 24 / 24
    

    View Slide