[TRICAP 2025] Energy-based non-negative tensor factorization via multi-body modeling

We can extract features or patterns from tensor-formatted data by approximating a given tensor with a low-rank tensor, represented as a linear combination of a small number of bases components. However, traditional low-rank approximation methods have well-known challenges, such as initial value dependency, ill-posedness, and the non-trivial rank tuning. To address these issues, this presentation introduces an alternative approach, called many-body approximation, which reduces higher-order interactions among tensor modes. Specifically, we describe the interactions among tensor modes using an energy function, and approximate the tensor by restricting the model to dominant low-order interactions. Our information geometrical analysis guarantees that our approach always finds the globally optimal solution that minimizes the KL divergence from the given tensor, regardless of initial values. The proposed method also facilitates intuitive model design by representing interactions among modes using a factor graph. This talk further shows how to convert the factor graph representation into a traditional tensor network, and illustrates the mathematical relationship between conventional low-rank approximations and the proposed approach. We demonstrate the effectiveness of the proposed method through numerical experiments on representation learning and missing value imputation.

TRICAP 2025: Three-way methods in Chemistry and Psychology, Norway, Ålesund, 2025.6.22-27

Transcript

1. @KazuGhalamkari Energy-based non-negative tensor factorization via multi-body modeling Kazu Ghalamkari @ Technical University of Denmark https://arxiv.org/abs/2209.15338 (accepted Neurips2023) https://arxiv.org/abs/2405.18220 (under review) TRICAP2025: Three-way methods in Chemistry and Psychology, Norway, Ålesund, 2025.6.22-27
2. @KazuGhalamkari Energy-based non-negative tensor factorization via multi-body modeling Outline 1. Many-body approximation for tensors (accepted Neurips2023) ▪ Tensor factorization with global optimization ▪ Energy-based tensor fractionation Ghalamkari, K. DTU Compute Sugiyama, M. Kawahara, Y. NII RIKEN Kazu Ghalamkari 2. Double bounded α-divergence optimization for tensors Generalized KL-divergence Technical University of Denmark (under review, preprint is in@ https://arxiv.org/abs/2405.18220 ) Mørup, M Ghalamkari, K. Hinrich, J. L. https://arxiv.org/abs/2209.15338 (accepted Neurips2023) DTU Compute DTU Compute DTU Compute ▪ em-projections basedhttps://arxiv.org/abs/2405.18220 low-rank approximation. (under review) ▪ α-divergence optimization offers robust learning against noise and outliers. TRICAP2025: Three-way methods in Chemistry and Psychology, Norway, Ålesund, 2025.6.22-27
3. At the intersection of Informatics, Physics, and Geometry. Tensor Modeling Optimization Flatness Tensor decomp. Pattern extraction Interaction, Energy Mean-field approximation Information geometry Pattern extraction and information reduction by tensor factorization Modeling with physics, e.g., interaction, energy, and mean-field Optimization via information geometry — the geometry of distributions 4
4. Extracting features from non-negative tensors. ▪ Various data are stored in computers as non-negative tensors. ▪ Non-negative tensor decomposition extracts features from data. 𝒫 Shop mode-２ Who Applications for data analysis, density estimation, compression, preprocessing, data mining, pattern recognition, and denoising.. ≃ +⋯+ 𝒫ത = mode-１ minmize 𝒫 − 𝒫ത 𝐹 We want to extract pattern or feature in tensor formatted data 5
5. Difficulties in tensor factorization ▪ We need to decide both of structure and a rank. Tucker decomp. CP decomp. 𝒫ത 𝒫 ≃ minmize 𝒫 − 𝒫ത Tensor Train decomposion 𝒫 = +⋯+ Decomp. with tensor networks Tensor ring decomposion ≃ 𝐹 ▪ Optimization is difficult. 𝒫 − 𝒫ത 𝐹 Non-convex Tensor decomposition is often ill-posed or NP-hard. ・ The best rank-1 CP-decomp. for minimizing Frobenius norm is NP-hard. The objective function is typically non-convex. ・Initial values dependency Solution often might be indeterminate. No guarantee to be the best solution. A convex, stable and intuitive designable tensor decomposition is desired.6
6. Many-body approximation for non-negative tensors Energy function Natural parameter of exponential distribution family. 7
7. Many-body approximation for non-negative tensors Energy function Natural parameter of exponential distribution family. 8
8. Many-body approximation for non-negative tensors Energy function Natural parameter of exponential distribution family. Control relation between mode-k and mode-l. Control relation among mode-j, -k and -l. 9
9. Many-body approximation for non-negative tensors One-body approx. Rank-1 approximation （mean-field approximation） [NeurIPS 2021 Ghalamkari, K., Sugiyama, M. ] 10
10. Many-body approximation for non-negative tensors Control relation between mode-k and mode-l. One-body approx. Two-body approx. Two-body Interaction Rank-1 approximation （mean-field approximation） [NeurIPS 2021 Ghalamkari, K., Sugiyama, M. ] Larger Capability 11
11. Many-body approximation for non-negative tensors Control relation between mode-k and mode-l. One-body approx. Control relation among mode-j, -k and -l. Three-body approx. Two-body approx. Two-body Interaction Rank-1 approximation （mean-field approximation） Three-body Interaction [NeurIPS 2021 Ghalamkari, K., Sugiyama, M. ] Intuitive modeling focusing on interactions between modes The global optimal solution minimizing KL divergence from Larger Capability can be obtained by a convex optimization. 12
12. Theoretical idea behind proposal Index is discrete random variable We regard a normalized tensor as a discrete joint probability distribution whose sample space is an index set We use information geometry to formulate factorization as convex problem 𝜽- Representation Geometry of 𝜽-space Coordinates transformation Low-rank space （Not flat） Difficult to optimize Representation with natural parameters of exponential family We construct flat subspace by focusing interaction among tensor modes Flat subspace Describing tensor factorization in 𝜽-coordinate system makes it convex problem 13
13. Mapping non-negative tensors to mass function Sample space for a 3×3×3 tensor 14
14. Mapping non-negative tensors to mass function Partially ordered sample space for a 3×3×3 tensor Log-linear model on Relation between distribution and tensor Random variables ： 𝑖, 𝑗, 𝑘, indices of the tensor Sample space ： index set Probability values ： tensor values 𝒫𝑖𝑗𝑘 15
15. Mapping non-negative tensors to mass function Partially ordered sample space for a 3×3×3 tensor Log-linear model on Relation between distribution and tensor Random variables ： 𝑖, 𝑗, 𝑘, indices of the tensor Sample space ： index set Probability values ： tensor values 𝒫𝑖𝑗𝑘 16
16. Describe a tensor with θ-parameter Partially ordered sample space for a 3×3×3 tensor Log-linear model on 1, 2, 2 1, 2, 2 Relation between distribution and tensor Random variables ： 𝑖, 𝑗, 𝑘, indices of the tensor Sample space ： index set Probability values ： tensor values 𝒫𝑖𝑗𝑘 Non-negative tensor is described by θ-parameter It has some useful properties for optimization. Natural parameter of exponential distribution family 17
17. Each point in θ-space corresponds to a tensor 18
18. e-flatness of model space in θ-space （empirical dist.） e-flat subspace Any subspace bounded by linear conditions on θ is e-flat 19
19. Projection onto flat space （empirical dist.） e-flat subspace Any subspace bounded by linear conditions on θ is e-flat MLE is a convex problem Projection onto e-flat subspace uniquely minimizes the KL divergence. m-projection We design the model space by linear conditions on θ such that can be written in a product form. tensor after the projection 20 8.00
20. Many-body approximation for four-order tensor Energy function Some theta should be zero. θ-parameters with many ‘1’ in its index control low-order interactions. The origin of stability of many-body approximation Many-body approximation reduces these θ-parameters by m-projection. Convex optimization. Destination is always unique. But… how to choose interactions to be reduced? 21
21. Example tensor reconstruction by proposal Reconstruction for 40×40×3×10 tensor. (width, height, colors, # images) Three-body Approx. Color is uniform within each image. Color depends on image index Shape of each image Color depends on pixel Larger capability Intuitive model designable that captures the relationship between modes 24
22. Color image is decomposed into shape × color 40×40×3×10 Approximation
23. Color image is decomposed into shape × color 40×40×10 Shape of each image ≃ 40×40×3×10 × 3×10 Color of each image =
24. ▪ Many-body approx. for non-negative tensors - Global optimization - Intuitive model design ▪ Relation between many-body approx. and tensor networks ▪ e2m: alpha-divergence optimization for non-negative tensors.
25. Interaction-based or rank-based factorization K.Ghalamkari, et.al., NeurIPS(2023) Many-body approx. Low-rank approx. Interaction among tensor modes Linear combination of base Connection? Interaction representation Nodes : Indices Edges via ▪ : Interactions [factor tensors] Convex optimization Tensor Network Nodes : Factor tensors Edges : Indices Non-convex optimization 29
26. Assuming a marginalized “hidden” mode 30
27. Assuming a marginalized “hidden” mode Hidden mode 31
28. Two-body interaction between hidden and visible modes Hidden mode 32
29. Two-body interaction between hidden and visible modes Hidden mode 33
30. MBA with hidden modes is low-rank approx. Hidden mode Marginalization Indices are random variables. Hidden variable (Latent variable) Rank-R CP decomp. Degree of freedom of hidden variable Equivalent ≃ 34
31. MBA with hidden modes is tensor-network. Tucker decomposition Tensor train decomp. Example: Add hidden variables r, s, and t. 6th dim tensor. ≃ Rank-(R,S,T) Tucker decomposition Rank-(R,S,T) TT decomposition 35
32. MBA with hidden modes is tensor-network. Tucker decomposition Tensor train decomp. Example: Add hidden variables r, s, and t. 6th dim tensor. ≃ Rank-(R,S,T) Tucker decomposition Rank-(R,S,T) TT decomposition 36
33. MBA with hidden modes is tensor-network. Tucker decomposition Tensor train decomp. Example: Add hidden variables r, s, and t. 6th dim tensor. ≃ Rank-(R,S,T) Tucker decomposition Rank-(R,S,T) TT decomposition 37
34. MBA with hidden modes is tensor-network. Tucker decomposition Tensor train decomp. Example: Add hidden variables r, s, and t. ≃ Rank-(R,S,T) Tucker decomposition Rank-(R,S,T) TT decomposition 38
35. MBA with hidden modes is tensor-network. Tensor ring decomp. Tensor train decomp. Rank-(R,S,T,U) Ring decomposition Rank-(R,S,T) TT decomposition 39
36. Low-rank approximation is MBA with hidden variables Many-body approx. with hidden variables is traditional low-rank approximation. Many-body approximation decomposes tensors with only visible variables → Convexity and intuitive modeling
37. Low-rank approximation is MBA with hidden variables Many-body approx. with hidden variables is traditional low-rank approximation. Algorithm to find the nearest point between two flat manifolds e-projection m-projection algorithm approximately finds the solution of MLE with hidden variables. 47 We can develop an e-projection m-projection-based low-rank approximation.
38. CP decompositions through the lens of information geometry CP-Decomposition for 3rd order tensor (3+1)-dim tensor space … A series of 3rd order tensor is a 4th order tensor m-flat 3-dim tensor space Convex m-projection Convex e-projection Non-convex e-flat Non-flat manifold Low-rank tensors Low-body tensors Linear condition of θ-parameter. Marginalization recovers low-rank tensors 48
39. Information geometric EM-CPD proposed by geometry CP decompositions throughview the of lens of information Huang, K. and Sidiropoulos, N. D. (2017) and Yeredor, A. and Haardt, M. (2019). Convexity of rank-1 KL-CPD em-based non-negative low-rank approx. Application to missing values estimation of the PMT (3+1)-dim tensor space e-step: e-projection onto m-flat manifold The optimal update is given as m-flat Expectation maximization m-step: m-projection onto e-flat manifold Convex m-projection Convex e-projection Many-body approximation Closed-form solution e-flat Low-body tensors Parameter update where 49
40. Low-rank approx. through the lens of information geometry em-based non-negative low-rank approx. e-step: e-projection onto m-flat manifold (3+V)-dim tensor space # of hidden variables in the model m-flat The optimal update is given as m-step: m-projection onto e-flat manifold Convex m-projection Convex e-projection Many-body approximation e-flat CPD Closed-form Tucker Closed-form Low-body tensors TT Closed-form where 50
41. CP decompositions through the lens of information geometry em-based non-negative low-rank approx. e-step: e-projection onto m-flat manifold m-step updates s.t. e-step The optimal update is given as e-step updates m-step: m-projection onto e-flat manifold Given tensor Low-rank tensor Jensen’s inequality shows: Many-body approximation For any 3-dim tensors (3+V)-dim tensors e-step makes it tight CPD Closed-form α-divergence, which will be Tucker Closed-form TT introduced in the next slide Closed-form where 51
42. e2m algorithm to optimize alpha-divergence em-based non-negative low-rank approx. e-step: e-projection onto m-flat manifold m-step updates + const. e-step The optimal update is given as e-step updates m-step: m-projection onto e-flat manifold Jensen’s inequality shows: Many-body approximation For any e-step makes it tight CPD Closed-form Tucker Closed-form TT Closed-form where where and tensor 52 + const.
43. Globally optimal 2 e m algorithm to optimize alpha-divergence by closed-form e1-step: e-projection onto m-flat manifold m-step updates . The optimal update is given as Globally optimal by closed-form e2-step: e-projection onto m-flat manifold The optimal update is given as . + const. e2-step e-step updates Globally optimal by closed-form ore1-step convex optimization E1-step updates m-step: m-projection onto e-flat manifold Jensen’s inequality shows: For any Many-body approximation e1-step makes it tight where e2-step makes it tight and tensor The overall Tuckeralgorithm is a TTnon-convex optimization and has an initial value dependency CPD Closed-form Closed-form Closed-form where m-flat manifold 53 + const.
44. Why do we optimize the alpha-divergence? A generalization of the KL-divergence since it holds that Rényi-alpha-divergence Given tensor Low-rank tensor Reconstructed rank 30 CP tensor P with varying α. Observed T True distribution class A T with 50 noise class B 90 × 90 × 2 tensor Noise robust Noise sensitive We can adjust the sensitivity to outliers and noise by the hyperparameter54alpha.
45. Comparison of the convergence performance Input T Reconstruction P Proposed method Gradient-based method requires learning-rate tuning. No need for learning rate tuning. 55 Monotonic decreasing of the objective function and convergence guarantee.
46. Comparison of the convergence performance No need for learning rate tuning. 56 Monotonic decreasing of the objective function and convergence guarantee.
47. Python code is available with demo code https://github.com/gkazunii/eemix/ 57
48. Summary □ Many-body approximation for non-negative tensors [NeurIPS2023, K.Ghalamkari, et.al] https://arxiv.org/abs/2209.15338 Two-body Approx. One-body Approx. Three-body Approx. Many-body approximation with hidden variables is low-rank approximation Parameterization via a log-linear model on poset Projection onto a flat manifold guarantees global optimization □ e2m algorithm to optimize α-divergence for discrete density estimation Visit high-dimensional space to seek flatness. https://arxiv.org/abs/2405.18220 https://github.com/gkazunii/eemix/ m-flat + const. Non-convex e2-step m-proj. e-proj. e1-step Non-flat manifold e-flat Double-bounded EM algorithm. α-div. optimization with closed-form updates. Noise robust Noise sensitive Adjust the sensitivity to outliers and noise by the hyperparameter α. K., Ghalamkari, et.al, Non-negative Tensor Low-rank Decompositions Through the Lens of Information Geometry (AAAI WS 2025) 58