Modeling Latent Non-Linear Dynamical System over Time Series

The slide for AAAI 2025 oral presentation

Transcript

1. Modeling Latent Non-Linear Dynamical System over Time Series Ren Fujiwara, Yasuko Matsubara, Yasushi Sakurai {r-fujiwr88, yasuko, yasushi}@sanken.osaka-u.ac.jp SANKEN, Osaka University 1
2. Non-Linear Dynamics are Everywhere Non-linear dynamics shape the world around us! 2 Nature Technology Biology
3. How to Analyze Non-Linear Dynamics? Mathematical Models • Equation-based Models (e.g., ODEs, PDEs) • Non equation-based Models (e.g., DMD, Koopman operator) Visualization Techniques • Phase portrait • Delay-embedding • Recurrence plot 3
4. Challenge 1. Noise vs. Non-linearity • It is difficult to distinguish between noise and non-linearity 4 Noise?
5. Challenge 2. Number of Variables • Real world data has often many variables 5 (i) Original Sequence (ii) Latent Dynamics A (left), F (right) (iv) Query Component Web-search counts data (6 outdoor-related keywords)
6. Challenge 2. Number of Variables • Real world data has often many variables 6 Equation: Too complicated (i) Original Sequence (ii) Latent Dynamics A (left), F (right) (iv) Query Component Web-search counts data (6 outdoor-related keywords)
7. Challenge 2. Number of Variables • Real world data has often many variables 7 Equation: Too complicated But Related works only focus on this setting (i) Original Sequence (ii) Latent Dynamics A (left), F (right) (iv) Query Component Web-search counts data (6 outdoor-related keywords)
8. Latent State 8 • Latent state (generated by removed noise) Non-linearity!
9. Latent State • Real world data has often latent groups 9 • Seasonality: Winter or Summer? • Trend: Increasing or Decreasing? • Interaction: Competitive or Mutualistic? (i) Original Sequence (ii) Latent Dynamics A (left), F (right) (iv) Query Component Web-search counts data (6 outdoor-related keywords)
10. Latent State • Real world data has often latent groups 10 Compress Equation: Easy to understand (i) Original Sequence (ii) Latent Dynamics A (left), F (right) (iv) Query Component Web-search counts data (6 outdoor-related keywords)
11. Our Problem (Informal) • Given: • Time series • Goal: • Estimate the latent states • Estimate dynamics with mathematical expression 11
12. Roadmap 12 Introduction Our Method Experiments Case Study Conclusions
13. About Latent Non-Linear equation modeling (LaNoLem) LaNoLem consists of two components • Latent non-linear dynamical system • Criterion for model complexity Our Algorithm consists of two sub-algorithms • Inference: Estimate latent states • Learning: Estimate parameter set 13
14. Sparsity in Non-Linear Dynamics • Non-linear dynamics can be represented by only a few terms [1] e.g., Lorenz attractor 14 Lorenz Rossler Halvorsen Arneodo [1] S.L. Brunton, J.L. Proctor, & J.N. Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. U.S.A. 113 (15) 3932-3937, https://doi.org/10.1073/pnas.1517384113 (2016). Equation:
15. Latent Non-Linear Dynamical System 15 Details • Latent non-linear dynamical system • Criteria for model complexity Data encoding cost Model description cost Proposed Model - LaNoLem Latent State Representation Observed values X are transformed into low-dimensional latent states S. Dynamics of Latent States • Linear transition: A • Non-linear transition: F
16. Criteria for Model Complexity We balance model accuracy and complexity by introducing the Minimum Description Length (MDL) principle. 16 Details • Latent non-linear dynamical system • Criteria for model complexity Data encoding cost Model description cost Proposed Model - LaNoLem • Latent non-linear dynamical system • Criteria for model complexity Data encoding cost Model description cost Proposed Model - LaNoLem
17. Optimization Algorithm 17 Problem definition: Our objective function: Details
18. Inference Objective: • Estimate Latent state Key Steps: • Forward passing: Estimate posterior distribution of • Backward passing: Estimate moments of (e.g., ) 18 Details
19. Learning Objective: • Estimate parameter set Key Steps: • Estimate A, F （Sparse parameters） • Estimate C, b, u （Non-sparse parameters) 19 Details
20. Learning • A, F （Sparse parameters） → Proximal gradient method • C, b, u （Non-sparse parameters） → Optimization via first-order condition (i.e., !" !# = 0) Theoretical analysis 20 Details
21. Roadmap 21 Introduction Our Method Experiments Case Study Conclusions
22. Experimental Setting • Dataset: dysts database[2], which is synthetic data generated by 71 chaotic ODEs • Baseline Methods: 3 SINDy-based methods: STLSQ, SSR, MIOSR 22 Lorenz Rossler Halvorsen Arneodo [2] Gilpin, W. 2021. Chaos as an interpretable benchmark for forecasting and data-driven modeling. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
23. (a-i) Coefficient error (lower is better) (b-i) Prediction error (lower is better) (b-ii) Critical difference diagram of prediction error Critical difference diagram of coefficient error Experimental Results 23 Our experimental result when the noise ratio is 5%, 25%, and 50% (Top): Coefficient error (Bottom): Prediction error LaNoLem is than the existing baseline (a-i) Coefficient error (lower is better) (a-ii) Critical difference diagram of coefficient error Avg. rank of LaNoLem : 1.54
24. Roadmap 24 Introduction Our Method Experiments Case Study Conclusions
25. Case Study（Web data mining） • LaNoLem provides various insights into the Google Trends. e.g., (ii) latent limit cycles, (iii) equations, and (iv) query grouping. 25 (i) Original Sequence (ii) Latent Dynamics (iii) Estimated System A (left), F (right) (iv) Query Component (Observation matrix C)
26. Case Study （Imputation in Time Series） 26 • LaNoLem not only estimates (iii) the equation but also enables (i) the imputation of these missing values. (i) Imputation Results (Noise ratio is 50%) (ii) Ground Truth A (left), F (right) (iii) Estimated System A (left), F (right)
27. Roadmap 27 Introduction Our Method Experiments Case Study Conclusions
28. Conclusions LaNoLem : Latent Non-Linear equation modeling Code : https://github.com/renfujiwara/LaNoLem Paper : https://arxiv.org/abs/2412.08114 28 Applicable Accurate and Robust Effective
29. Modeling Latent Non-Linear Dynamical System over Time Series 29 Ren Fujiwara Yasuko Matsubara Yasushi Sakurai