Efficient Long-Sequence Learning with Deep State Space Models

In this Biomed Seminar, Pedro Devogelaere (KU Leuven) presents "Efficient long-sequence learning with deep State Space Models." The talk covers the theoretical foundations of High-order Polynomial Projection Operations (HiPPO) and Structured State Spaces (S4), and how these methods enable efficient modeling of long sequences. A biomedical application in automated sleep staging is then explored, comparing the S4-based S4Sleep model against the RNN-based L-SeqSleepNet on standard polysomnography benchmarks.

Transcript

1. Efficient long-sequence learning with deep State Space Models Biomed Meetings Pedro Devogelaere – 12/06/2026 1
2. • Methods • High-order Polynomial Projection Operations (HiPPO) • Efficiently Modeling Long Sequences with Structured State Spaces (S4) • Results • Biomedical application • Sleep staging • L-SeqSleepNet • S4Sleep Content 2
3. Methods 3
4. • Recurrent Neural Networks (RNNs) • Transformers • Deep State Space Models (SSMs) • HiPPO [1] • S4 [2] • Mamba [3] State-of-the-Art Sequence modeling 4 [5] [4]
5. Learning memory representations function approximation ||𝑓≤𝑡 − 𝑔 𝑡 || 𝐿2 𝜇 𝑡 • Measure 𝜇 𝑡 e.g. importance of every sample • Basis (polynomial) 𝒢 e.g. allowable approximations How to solve this optimization problem? High-order Polynomial Projection Operations (HiPPO) [1] 5
6. High-order Polynomial Projection Operations (HiPPO) [1] 6
7. High-order Polynomial Projection Operations (HiPPO) [1] 7 1. proj: Projection that minimizes ||𝑓≤𝑡 − 𝑔 𝑡 || 𝐿2 𝜇 𝑡 2. coef: Maps polynomial 𝑔 𝑡 to the coefficient 𝑐 𝑡 ∈ ℝ𝑁 of basis of orthogonal polynomials w.r.t. 𝜇 𝑡 3. Results in ODE (see appendices [1]): 𝑑 𝑑t 𝑐 𝑡 = 𝐴 𝑡 𝑐 𝑡 + 𝐵 𝑡 𝑓 𝑡 4. Discretize to get sequence 𝑐𝑘 ∈ ℝ𝑁 hippo Memory unit
8. High-order Polynomial Projection Operations (HiPPO) [1] 8 For each measure 𝜇 𝑡 you can design a different transition matrix 𝐴 (and 𝐵) to get an orthogonal polynomial projection w.r.t. 𝜇 𝑡
9. Scaled Legendre measure is Δ𝑡-invariant (under bilinear transform) High-order Polynomial Projection Operations (HiPPO) [1] 9 Discretization: 𝑑 𝑑t 𝑐 𝑡 = 𝐴 𝑡 𝑐 𝑡 + 𝐵 𝑡 𝑓 𝑡 ↓ step size Δ𝑡 𝑐 𝑡 + Δ𝑡 − 𝑐 𝑡 = න 𝑡 𝑡+Δ𝑡 𝑓 𝑠, 𝑐 𝑠 𝑑𝑠 ↓ approximate RHS integral (e.g. bilinear (S4), ZOH (Mamba)) 𝑐𝑘+1 = 𝐴𝑐𝑘 + 𝐵𝑓𝑘 ⟹ analytical HiPPO state parameters
10. State Space Model (SSM): 𝑥𝑘+1 = 𝐴𝑥𝑘 + 𝐵𝑢𝑘 𝑦𝑘 = 𝐶𝑥𝑘 + 𝐷𝑢𝑘 𝑢𝑘 : input 𝐵: introduces input to memory 𝑥𝑘 : memory 𝐴: updates old memory 𝑦𝑘 : output 𝐶: filters memory to get output 𝐷: skip connection Efficiently Modeling Long Sequences with Structured State Spaces (S4) [2] 10
11. Efficiently Modeling Long Sequences with Structured State Spaces (S4) [2] 11 Scaled Legendre measure HiPPO parameters to initialize SSM parameters
12. Recurrent representation (sequential) 𝑥𝑘+1 = 𝐴𝑥𝑘 + 𝐵𝑢𝑘 𝑦𝑘 = 𝐶𝑥𝑘 + 𝐷𝑢𝑘 Train efficiently? Convolutional representation (parallel) 𝑦0 = 𝐶𝐵𝑢0 , 𝑦1 = 𝐶𝐴𝐵𝑢0 + 𝐶𝐵𝑢1 , … , 𝑦𝑘 = 𝐶𝐴 𝑘 𝐵𝑢0 + 𝐶𝐴 𝑘−1 𝐵𝑢1 + 𝐶𝐴 𝑘−2 𝐵𝑢2 + … + 𝐶𝐵𝑢𝑘 Convolution with a kernel: y = 𝐊 ∗ 𝑢 with 𝐊 = [𝐶𝐵, … , 𝐶𝐴 𝑘−2 𝐵, 𝐶𝐴 𝑘−1 𝐵, 𝐶𝐴 𝑘 𝐵], Direct convolution OR FFT (𝒪 𝐿𝑙𝑜𝑔𝐿 ) Efficiently Modeling Long Sequences with Structured State Spaces (S4) [2] 12 Main focus paper (𝒪 𝐿𝑁2 → 𝒪 𝐿𝑁 )
13. Efficiently Modeling Long Sequences with Structured State Spaces (S4) [2] 13 [6]
14. Where does it break down? - Associative memory - Short sequences (< 1000-2000 tokens) Efficiently Modeling Long Sequences with Structured State Spaces (S4) [2] 14
15. Results 15 Impressive results on long range tasks! [2]
16. Biomedical application 16
17. EEG, EOG, ECG, EMG, … recorded during sleep • Epoch: standard 30s • Sequence: consists of many epochs • 1 sleep cycle: 90-110 min → 180-220 epochs Standard sequence length automated sleep staging: 10-20 epochs Hypothesis: “An appropriate method that equips a sleep staging system with long sequential modelling capability would benefit its performance.” [7] Sleep staging with PolySomnoGraphy (PSG) 17 Classification
18. L-SeqSleepNet [7] 18 Spectogram Long-sequence modeling 𝑥 → 𝑜 Epoch encoding 𝑆 → 𝑥 Sleep stage classification (N1, N2, N3, REM, Awake) Input Output 1 sleep cycle – 100 min (L = 200) 600k params
19. S4Sleep [8] 19 5M params 7.5 min (L = 15) Sleep stage classification (N1, N2, N3, REM, Awake) Sequence modeling Epoch encoding Raw time series (ts) 7.5 min (L = 15) Spectogram (spec)
20. Results [7,8] 20 Model Macro-F1 Cohen’s Kappa Accuracy Macro AUC L-SeqSleepNet [7] 0.814 0.838 0.884 - S4Sleep(spect) [8] 0.819 - 0.883 0.977 S4Sleep(ts) [8] 0.817 - 0.882 0.977 SHHS Model Macro-F1 Cohen’s Kappa Accuracy Macro AUC L-SeqSleepNet [7] 0.793 0.813 0.863 - S4Sleep(spect) [8] 0.787 - 0.849 0.970 S4Sleep(ts) [8] 0.804 - 0.876 0.971 SEDF
21. • Lightweight S4 model • S4 for long-sequence modelling • Use Cohens Kappa in evaluation • Mamba • … Future work? 21
22. 22 Questions?
23. [1] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, “HiPPO: recurrent memory with optimal polynomial projections,” in NeurIPS, 2020. [2] A. Gu, K. Goel, and C. Re, “Efficiently Modeling Long Sequences with Structured State Spaces,” presented at ICLR, 2021. [3] A. Gu and T. Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” presented at ICLR, 2024. [4] https://www.geeksforgeeks.org/machine-learning/seq2seq-model-in-machine-learning/ [5] Nvidia’s Next GPU Shows That Transformers Are Transforming AI [6] https://www.youtube.com/watch?v=WC9tqkCpq4s [7] H. Phan et al., “L-SeqSleepNet: Whole-cycle Long Sequence Modeling for Automatic Sleep Staging,” IEEE Journal of Biomedical and Health Informatics, 2023. [8] T. Wang and N. Strodthoff, “S4Sleep: Elucidating the design space of deep-learning-based sleep stage classification models,” Computers in Biology and Medicine, 2025. [9] The annotated S4 References 23
24. Blogs • The annotated S4 • Simplifying S4 • On the Tradeoffs of SSMs and Transformers Talks • Efficiently Modeling Long Sequences with Structured State Spaces [MedAI – Stanford] • Structured State Space Models for Deep Sequence Modeling [LxMLS – Lisbon ML School] Interesting resources 24
25. Additional material 25
26. Efficiently Modeling Long Sequences with Structured State Spaces (S4) 26 Intuitive behaviour of hidden state 𝑥: weighting of polynomial bases in signal 𝑢 [8]
27. Efficiently Modeling Long Sequences with Structured State Spaces (S4) 27 1. Instead of computing 𝐊, we compute its spectrum (e.g. z-transform followed by evaluations at root of unity (transform to DFT)) by evaluating its truncated generating function. E.g. matrix inverse instead of power. If we have its spectrum we perform the iFFT to obtain 𝐊. 2. The diagonal matrix case on 𝐀 is equivalent to computing a Cauchy kernel, instead of computing a naive inverse. 3. DPLR: Relax the diagonal assumption and realize that the low-rank term can be corrected by applying the Woodbury identity. However, at this point HiPPO isn't a DPLR matrix, it does have special structure though, since it is Normal Plus Low-Rank (NPLR). After diagonalizing this is in principle the same as DPLR (it is unitarily diagonalizable since it is NPLR).