[UAI2026, Tutorial Talk] Tensor Networks for Density Estimation: The merits of low-rank structures, tractable inference, and uncertainty quantification

How can we estimate the true distribution underlying the given data? This is one of the fundamental questions in machine learning. With the current advances in GPU accelerators, the community relies heavily on deep learning-based density estimation, which offers great success in expressivity and scalability; however, theoretical intractability, the need for costly hyper-parameter tuning, weaker optimization guarantees, and the non-trivial extension to the discrete settings remain important challenges. Recently, alternative approaches for density estimation, centered on the use of tensor networks, are garnering attention as they can overcome these difficulties. In addition, their connections have also recently been established to other fields such as probabilistic circuits, information geometry, logic programming, and relational learning, forming a rich community in which tensors play a role of shared language, as seen in recent tensor-related workshops, Connecting Low-Rank Representations in AI at AAAI’25 and ICML’26, and a tutorial, Foundations of Tensor/Low-Rank Computations for AI at Neurips 2025. Given the current situation, we presently provide a tutorial on tensor-based density estimation where its exact marginalization, natural Bayesian extension, and convergence guarantees directly match the interests of the UAI community. Aiming to welcome newcomers as well as bridging various fields, this tutorial covers the following topics: i) How tensors are useful for density estimation, ii) how tensor-based density estimation connects diverse fields, and iii) what are important future directions of tensor networks for the UAI community.

The first part of this tutorial can be found in
https://drive.google.com/file/d/1kmZvOst4s7CkI3lNpewZ0tMnpmChU57u/view?usp=drive_link

Transcript

1. @KazuGhalamkari Tensor Networks for Density Estimation: The merits of low-rank structures, tractable inference, and uncertainty quantification Kazu Ghalamkari Morten Mørup @ Technical University of Denmark Uncertainty in Artificial Intelligence 2026 (UAI 2026), Amsterdam, 17th Aug. 2026
2. Non-negative tensor factorization for density estimation minimize Discrete samples +⋯+ Low-rank model Empirical dist. (non-negative) (non-negative) person shop = good [Kargas et al., 2018; Glasser et al., 2019; Vora et al., 2021; Kargas & Sidiropoulos, 2017; Ibrahim & Fu, 2021] 2
3. Non-negative tensor factorization for density estimation minimize Discrete samples +⋯+ = Low-rank model Empirical dist. (non-negative) (non-negative) Tensor train density estimation [Novikov et al., 2021] 3
4. Non-negative tensor factorization for density estimation minimize Discrete samples +⋯+ = Low-rank model Empirical dist. (non-negative) (non-negative) Non-negative mixture tensor learning [Ghalamkari et al., 2026] 4
5. Non-negative tensor factorization for density estimation minimize Discrete samples +⋯+ = Low-rank model Empirical dist. (non-negative) (non-negative) As a side note…. RESCAL … = [Nickel et al., 2011] 5
6. Non-negative tensor factorization for density estimation minimize Discrete samples +⋯+ = Low-rank model Empirical dist. (non-negative) (non-negative) KL-based tensor factorization estimates the true distribution underlying data M-step updates E-step updates r is a hidden variable (marginalized R.V.) [Huang & Sidiropoulos, 2017; Yeredor & Haardt, 2019; Chege et al., 2022] E-step Non-Convex Function
7. Contents ▪ Deformed tensor density estimation with 𝜒-divergence optimization Geometry Algebra Observation ▪ Double-bounded E2M-algorithm for q-divergence minimization Traditional EM-algorithm E2-step E-step E1-step 7
8. The KL-divergence is not a perfect measure KL-divergence q-divergence Model Data Hyper-parameter (q>0) The KL-divergence has a large penalty if Fitting with q=0.1 and , which induces overfitting to noise. Fitting with q=0.5 Fitting with q=1.0 Large penalty for KL noise Good fitting ignores noise noise Overfit to noise due to the nature of the KL-divergence. [Basu et al. 1998] Alternative divergences, such as q-divergence, reduce this weakness. ⇒ Generated via information geometry. 8
9. Deformed many-body approximation for non-negative tensors Energy function Natural parameter of deformed exponential family. Free energy χ-exponential and logarithm function For any increasing function Examples: ❖ Standard exponential function ⇒ ❖ Tsallis deformation (appeared in statistical mechanics) Without deformation ⇒ Temperature parameter [Tsallis, 1999] ❖ Kaniadakis deformation (appeared in theory of relativity) ⇒ [Kaniadakis et al., 2005] We can adjust model properties by changing function. 9
10. Deformed many-body approximation for non-negative tensors Natural parameter of deformed exponential family. Control relation between mode-k and mode-l. Control relation among mode-j, -k and -l. 10
11. Deformed many-body approximation for non-negative tensors One-body approx. Deformed product [Matsuzoe and Wada, 2015] Examples: ❖ Tsallis product Deformed rank-1 approximation ⇒ （deformed mean-field approximation） ⇒ ⇒ 11
12. Deformed many-body approximation for non-negative tensors One-body approx. Deformed product [Matsuzoe and Wada, 2015] Examples: ❖ Tsallis product rank-1 approximation （mean-field approximation） [Ghalamkari & Sugiyama, 2021] ⇒ ⇒ ⇒ 12
13. Deformed many-body approximation for non-negative tensors Deformed product One-body approx. [Matsuzoe and Wada, 2015] Intermediate between standard sum and standard product Deformed rank-1 approximation Examples: ❖ Tsallis product ⇒ （deformed mean-field approximation） ⇒ ⇒ 13
14. Deformed many-body approximation for non-negative tensors χ-Exponential family One-body approx. Deformed rank-1 approximation （deformed mean-field approximation） One-body tensors One-body tensors Non-convex Convex 14
15. Deformed many-body approximation for non-negative tensors How should we measure data? e.g., Emphasize/discard rare events χ-divergence One-body approx. where the escort is defined as Deformed rank-1 approximation （deformed mean-field approximation） escort One-body tensors Re-weighting [Naudts,2004] Convex 15
16. Deformed many-body approximation for non-negative tensors χ-divergence One-body approx. How should we measure data? e.g., Emphasize/discard rare events where the escort is defined as Examples KL-div. Deformed rank-1 approximation （deformed mean-field approximation） q-div. One-body tensors Convex (coordinate system) (product in model) Matching of geometry and algebra relaxes the problem into convex optimization
17. Deformed many-body approximation for non-negative tensors Control relation between mode-k and mode-l. One-body approx. Two-body approx. Deformed rank-1 approximation （deformed mean-field approximation） Larger Capability 17
18. Deformed many-body approximation for non-negative tensors [Ghalamkari et. al, 2026] Control relation between mode-k and mode-l. One-body approx. Control relation among mode-j, -k and -l. Three-body approx. Two-body approx. Two-body Interaction Three-body Interaction Deformed rank-1 approximation （deformed mean-field approximation） Intuitive modeling focusing on interactions between modes The global optimal solution minimizing from Larger Capability can be obtained by a convex optimization. 18
19. Tsallis deformed three-body approx. in noisy settings; χ(t)=tq True Image Noisy Image Temperature q controls the sensitivity against the noise. Worse Vanilla MBA (q=1.0) Tsallis MBA (q=0.8) Tsallis MBA (q=0.6) Better Tsallis MBA (q=0.4) Less noisy Tsallis MBA (q=0.2) More noisy 19
20. Kaniadakis deformed three-body approx. in noisy settings True Image Noisy Image Parameter κ enhances the sensitivity against the noise. Worse Vanilla MBA (κ=0.0) Kani’s MBA (κ=0.8) Kani’s MBA (κ=0.6) Better Kani’s MBA (κ=0.4) Less noisy Kani’s MBA (κ=0.2) More noisy 20
21. Mixture of deformed one-body approximation deformed one-body approx. = Deformed One-body tensor Convex One-body tensors Flat manifold 21
22. Mixture of deformed one-body approximation mixture deformed one-body approx. +⋯+ = Deformed rank-R tensor Convex One-body tensors Flat manifold 22
23. Mixture destroys the flatness of the model space mixture deformed one-body approx. +⋯+ = Deformed rank-R tensor Destroy flatness of the model space Non-convex Convex One-body tensors Non-flat manifold Deformed rank-R tensors Flat manifold 23
24. Mixture destroys the flatness of the model space mixture deformed one-body approx. +⋯+ = Deformed rank-R tensor Destroy flatness of the model space Non-convex Convex One-body tensors Non-flat manifold Deformed rank-R tensors Flat manifold 24
25. Mixture destroys the flatness of the model space 𝑞-deformed rank-R approx. as mixture deformed one-body approx. +⋯+ = 𝑞-deformed rank Deformed rank-R tensor Non-convex Convex One-body tensors Non-flat manifold Deformed rank-R tensors Flat manifold 25
26. Mixture destroys the flatness of the model space Non-convex optimization 𝑞-deformed rank-R approx. as mixture deformed one-body approx. +⋯+ = 𝑞-deformed rank Deformed rank-R tensor Non-convex optimization Traditional rank-R approximation +⋯+ = Tensor rank rank-R tensor What is the benefit of q-deformed rank-R approximation?
27. Regularization induced by the deformation Large R increases # of parameters ⇒ Large capacity 𝑞-deformed rank-R approx. = Deformed rank-R tensor Traditional rank-R approximation +⋯+ = stable learning Large R increases # of parameters ⇒ Large capacity Test error +⋯+ Test error Small q reduces model capacity (regularization) Overfitting 27
28. Regularization induced by the deformation Large R increases # of parameters ⇒ Large capacity 𝑞-deformed rank-R approx. +⋯+ = Deformed rank-R tensor Test error Small q reduces model capacity (regularization) stable learning Intermediate between standard sum and standard product 𝑞-deformed rank ❖ Tsallis product ⇒ ⇒ q restricts the model capacity ⇒ 28
29. Regularization in discrete density estimation Discrete density estimation ① Low-rank approx. Training samples ② q-deformed low-rank approx. Deformed rank ② Proposed method w. deformation worse More over-fitting Less over-fitting Baseline ① w/o. deformation better Tsallis-deformation induces regularization and prevents overfitting 29
30. Regularization in discrete density estimation Discrete density estimation ① Low-rank approx. Training samples ② q-deformed low-rank approx. worse More over-fitting Deformed rank ② Proposed method w. deformation Less over-fitting better worse Baseline ① w/o. deformation better Tsallis-deformation induces regularization and prevents overfitting 30
31. Noisy data reconst. by deformed rank-R approximation Noisy image reconstruction True Noisy data Low-rank model ① Traditional low-rank approx. Deformed rank ② q-deformed low-rank approx. ② q-deformed low-rank model ① Traditional low-rank model Training error Test error Training error Test error Small q Small q KL-div. (q=1.0) KL-div. (q=1.0) Small q KL-div. (q=1.0) Overfitting Smaller q leads to robustness against the noise. (Known effect of the q-divergence) Smaller q leads to regularization 31
32. Regularization induced by Tsallis deformation ① Traditional low-rank approx. Training error ② q-deformed low-rank approx. Test error Training error Test error Small q Small q KL-div. (q=1.0) KL-div. (q=1.0) Reconstructions with q=0.5 Small q Overfitting Overfit to noise if the rank is large. The model’s capability increases as the rank increases. KL-div. (q=1.0) Reconstructions with q=0.5 No noise even with larger ranks Theorem. For small 𝑞, the model capacity remains limited despite a large deformed rank. tensor order Traditional rank 32
33. em-algorithm for deformed low-rank approx. 𝑞-deformed rank-R approx. +⋯+ = Destroy flattens of the model space 4-dim tensor space 3-dim tensor space Non-convex Convex Flat manifold Convex Non-flat manifold Deformed rank-R tensors Deformed Low-body tensors (Flat manifold ) 33
34. em-algorithm for deformed low-rank approx. 𝑞-deformed rank-R approx. Generalization of the EM (ExpectationMaximization) algorithm em-algorithm [Amari, 1995] +⋯+ = 4-dim tensor space e-projection m-projection Convex Flat manifold Convex Each iteration is convex but needs gradient-based methods Deformed Low-body tensors (Flat manifold ) 34
35. Double-bound strategy for q-divergence optimization Simple low-rank model q-divergence KL-divergence Huang & Sidiropoulos, 2017; Yeredor & Haardt, 2019; E2-step E1-step M-step minimizes with closed-form Still no closed-form due to summation in the logarithm. More details can be found here M-step minimizes with closed-form E-step 35
36. Double-bound strategy for q-divergence optimization Simple low-rank model q-divergence E2-step E2M algorithm E1-step More details can be found here No need for learning-rate tuning. Monotonic decrease of the objective function. 36 Convergence guarantee.
37. Noise-robustness of the alpha-divergence q-divergence [Ghalamkari et al, 2026] Hyper-parameter (q>0) Reconstructed rank 30 CP tensor P with varying q. Observed T q:0.1 q:0.3 q:0.5 q:0.7 q:1.0 True distribution class A T with 50 noise class B 90 × 90 × 2 tensor 37
38. Noise-robustness of the alpha-divergence q-divergence [Ghalamkari et al, 2026] Hyper-parameter (q>0) Reconstructed rank 30 CP tensor P with varying q. q:0.1 Observed T q:0.1 q:0.3 q:1.0 q:0.5 q:0.7 q:1.0 True distribution class A Outlier T with 50 noise Mislabeled data class B 90 × 90 × 2 tensor Outlier 38
39. Noise-robustness of the alpha-divergence q-divergence [Ghalamkari et al, 2026] Hyper-parameter (q>0) Reconstructed rank 30 CP tensor P with varying q. Observed T q:0.1 q:0.3 q:0.5 q:0.7 q:1.0 True distribution class A T with 50 noise class B 90 × 90 × 2 tensor Noise robust Noise sensitive 39 We can adjust the sensitivity to outliers and noise by the hyperparameter q.
40. Convenient libraries for tensors PyTorch, Numpy backends ロゴ 自動的に生成さ れた説明 Probabilistic Tensor Decomposition Toolbox 箱ひげ図が含まれている画像 自動的に生成さ れた説明 JesperLH/prob-tensor-toolbox Tullio.jl pyttb: Python Tensor Toolbox sandialabs/pyttb ITensor ▪ Deformed decomposition gkazunii/pymba ▪ EEM-algorithm for tensors gkazunii/eemix TensorKit.jl 40
41. Conclusion of Part II +⋯+ Geometry = Coupling Statistical gains Observation Algebra Noise robustness Test error E2-step Regularization Decay tunability E1-step 41
42. Future directions • Bridging advanced tensor network structures with probabilistic graphical models exploring dependency structures. I.e., MERA, PEPS/Tensor Trees etc. • Combining tensor network based densities with deep generative modeling. I.e, efficient and expressive factorizations of latent distributions used in the generative process. Tensor network layers as part of deep learning processing architecture. • Exploring uncertainty quantification in the context of network structures. I.e., typically tensor networks are based on point estimates whereas uncertainty aware modeling provides added robustness. • Efficient tensor network structure search. I.e., whereas there are many computational merits there are also substantial architectural choices to consider. • Exploring efficient learning approaches beyond standard maximum likelihood I.e., DMRG, self-supervised and constrastive learning. Most tensor network structure learning problems NP-hard, best practices for initializing and setting up suitable learning algorithms. • Further explore what are tensor network model structures producing unique and explainabe represenations I.e., CPD, PARAFAC2, Block-term decompositions include uniqueness results. 42 DTU Compute Tensor Networks for Density Estimation: The merits of low-rank structures, tractable inference, and uncertainty quantification 17-8-2026