Physiological Modeling With Multispectral Imaging for Heart Rate Estimation

1. Physiological Modeling With Multispectral Imaging for Heart Rate Estimation Kosuke Kurihara 1 Yoshihiro Maeda 2 Daisuke Sugimura 3 Takayuki Hamamoto 1 1 Tokyo University of Science 2 Tokyo Metropolitan University 3 Shibaura Institute of Technology ICIP 2024 TP1.L5.5
2. ⚫ Non-contact video-based heart rate (HR) estimation • Attracts research attention because of its convenience ⚫ Principle: color change analysis on the facial videos • Blood volume pulse (BVP) associated with the cardiac pulse causes skin color changes → BVP signal and HR can be estimated Non-contact HR estimation using cameras 1 HR: 68 bpm RGB video Temporal variations G B R BVP signal Blood vessel Skin Cardiac pulse Analysis time→
3. ⚫ Video-based HR estimation performance is limited ⚫ BVP signal is low signal-to-noise ratio 1. BVP signal is subtle signal (typically less than 2 bits) 2. Large noise due to fluctuations of ambient illumination Problem of video-based HR estimation 2 Camera Ambient light 1. Subtle BVP 2. Illumination noise Illumination Small amplitude Large amplitude BVP
4. ⚫ Modeling spatio-temporal BVP characteristics improve BVP signal extraction performance [1] • BVP has quasi-periodic temporal characteristics → BVP can be modeled using conservative systems • BVP propagates over the facial region with the same timing → BVP can be modeled as spatially similar Previous method 1: Physiological modeling of BVP 3 video observation time Temporal characteristic Cardiac pulse Spatial characteristic BVP propagation spatial [1] Kurihara+, Spatio-Temporal Structure Extraction of Blood Volume Pulse Using Dynamic Mode Decomposition for Heart Rate Estimation, Access 2023
5. ⚫ Using RGB and near-infrared (NIR) videos • Captured with invisible NIR light → Stable NIR signals independent to illumination noise ⚫ Flexible integration of RGB/NIR videos based on background illumination analysis [2] → Achieve robustness in a various illumination scene Previous method 2: Ambient illumination modeling 4 RGB video NIR video Controlled illumination RGB video NIR video Uncontrolled illumination [2] Kurihara+, Non-Contact Heart Rate Estimation via Adaptive RGB/NIR Signal Fusion, TIP 2021
6. ⚫ Perform physiological and illumination modeling • Comprehensive modeling can enhance HR estimation performance Our key idea 5 RGB/NIR camera Ambient Physiological modeling Illumination modeling NIR light Illumination Large BVP Small
7. ⚫ Bayesian inference-based HR estimation incorporating physiological and illumination model Overview of our algorithm 6 NIR video RGB video Flexible integration based on illumination model RGB/NIR weight calculation patch group NIR HR likelihood computation based on physiological model RGB HR likelihood computation based on physiological model Integrated HR likelihood HR posterior at 𝜏-1 HR posterior at 𝜏 MAP estimation State transition
8. ⚫ Extract RGB/NIR signals from the facial patches • RGB signals: project onto the chrominance space → Mitigate influences of the noise components Obtaining multiple observation signal 7 RGB video RGB Observation RGB signal ←patch ←patch Chrominance analysis[6] B R G NIR video NIR Observation ←patch time→ time→ time→ patches
9. ⚫ Utilize HR estimation method based on spatio-temporal BVP modeling (BVPDMD) [1] • Dynamic Mode Decomposition (DMD): spatio-temporal dynamical analysis method • Incorporating BVP dynamical knowledge into DMD enable accurate BVP signal extraction and HR estimation HR candidate extraction: overview 8 BVP Temporal dynamics BVP Spatial dynamics spatial time temporal freq. amplitude BVPDMD Mode spectral Peak analysis BVP signal & HR candidate Select mode Observation signal [1] Kurihara+, Spatio-Temporal Structure Extraction of Blood Volume Pulse Using Dynamic Mode Decomposition for Heart Rate Estimation, Access 2023
10. ⚫ Perform BVPDMD to neighbor patch group • Obtain RGB/NIR HR candidate & confidence score Calculating HR candidates & confidence scores 9 RGB Observation NIR Observation time→ time→ ←patch ←patch neighbor patch group #1 BVPDMD RGB HR candidate & confidence score #1 … BVPDMD NIR HR candidate & confidence score #1 … temporal freq. amplitude BVPDMD Mode spectral HR candidates BVP signal & HR candidate Confidence score Selected mode
11. ⚫ Compute RGB/NIR HR likelihood using HR candidates & confidence score • Aggregate HR candidates using weighted kernel density estimation RGB/NIR HR likelihood computation 10 time→ time→ ←patch ←patch … #1 #2 Weighted kernel density estimation RGB HR likelihood … #1 #2 Weighted kernel density estimation NIR HR likelihood RGB NIR
12. ⚫ Flexible RGB/NIR integration based on correlation analysis between face/background • Assess whether RGB or NIR signals is more reliable [2] » Uncontrolled scene : High correlations → NIR is reliable because of stable NIR light » Controlled scene : Less correlations → RGB is reliable because of skin and blood characteristics Flexible RGB/NIR integration: overview 11 Face Back Controlled : RGB is reliable Face Back Uncontrolled : NIR is reliable [2] Kurihara+, Non-Contact Heart Rate Estimation via Adaptive RGB/NIR Signal Fusion, TIP 2021
13. ⚫ Perform Pearson correlation analysis on RGB signals in face and background regions • In uncontrolled scene: coefficient becomes high RGB facial signal is difficult to estimate HR → Assign more weight on NIR likelihood Flexible RGB/NIR integration: compute flexible weight 12 RGB likelihood (priority in controlled) NIR likelihood (priority in uncontrolled) Flexibly integrated likelihood RGB video G face G back Pearson Correlation Correlation analysis × ×
14. ⚫ Infer latent HR based on MAP estimation • Calculate HR posterior using integrated HR likelihood and HR prior based on particle filter framework HR estimation based on MAP inference 13 HR HR posterior at 𝜏 − 1 HR posterior at 𝜏 HR prior at 𝜏 State transition RGB/NIR integrated HR likelihood Apply Bayes' theorem MAP estimation
15. ⚫ Experimental setting (11 subjects, 4 scenes) • Bright (600 lx), Low (0.4 lx), Varying (1 lx), Theater (1 lx) ⚫ Comparison methods: • RGB method [3,4,5] • NIR method [6,7] • RGB/NIR method [8,9] Experiments using real data 14 Low High Theater RGB NIR Varying During experiment (Theater) Video [3] Kumar+, DistancePPG: Robust non-contact vital signs monitoring using a camera, Biomedical Optics Express 2015 [4] Tulyakov+, Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions, CVPR 2016 [5] Yu+, Physformer: Facial video-based physiological measurement with temporal difference transformer, CVPR 2022 [6] Martinez+, Non-contact photoplethysmogram and instantaneous heart rate estimation from infrared face video, ICIP 2019 [7] Nowara+, Sparseppg: Towards driver monitoring using camera-based vital signs estimation in near-infrared, CVPRW 2018 [8] Kado+, Remote heart rate measurement from RGB-NIR video based on spatial and spectral face patch selection, EMBC 2018 [9] Liu+, Information-enhanced network for noncontact heart rate estimation from facial videos, TCSVT 2023 RGB/NIR camera subject
16. ⚫ Camera: RGB/NIR camera （JAI AD-130GE） • A two-plate sensor enables simultaneous acquisition of RGB and NIR video • Frame rate: 30fps • Bit depth: 8bit • Resolution: 1296×966 Experimentation details 15 Internal structure Wavelength (nm) R G B NIR 400 Sensitivity 600 800 1000 RGB sensor NIR sensor Hot Mirror Spectral sensitivity
17. ⚫ Evaluation metrics: • Mean absolute error [bpm] between ground-truth HRs and estimated HRs Quantitative comparison 16 Bright Low Vary Theater Ave. RGB Distance [3] 5.2 157.3 28.5 35.2 56.6 SAMC [4] 4.1 87.1 23.1 26.9 35.3 Physformer [5] 19.5 74.0 27.9 53.8 43.8 NIR NatalIR [6] 27.6 24.2 26.0 28.1 26.5 SparsePPG [7] 45.4 46.4 59.7 22.6 43.5 RGB/ NIR Random [8] 5.6 13.5 22.3 17.7 14.8 MDEF [9] 40.3 39.5 30.0 38.0 37.0 Ours 2.2 5.5 6.4 6.8 5.2
18. ⚫ Analyzing the impact of each module • Physiological BVP modeling • Flexible integration of RGB/NIR ⚫ Evaluation metrics: success rate [%] • Success is defined as an error within ±5 bpm Analysis of each component 17 Bright Low Vary Theater Ave. RGB wo/Phys 15.8 0.0 0.2 1.8 4.5 RGB w/Phys 91.0 7.3 0.3 16.7 28.8 NIR wo/Phys 16.5 18.7 14.1 22.0 17.8 NIR w/Phys 61.7 63.2 58.6 63.3 61.7 Flex wo/Phys 15.7 15.2 12.4 21.8 16.3 Flex w/Phys 90.9 61.1 54.0 62.2 67.1 Bright Low Vary Theater Ave. RGB wo/Phys 15.8 0.0 0.2 1.8 4.5 RGB w/Phys 91.0 7.3 0.3 16.7 28.8 NIR wo/Phys 16.5 18.7 14.1 22.0 17.8 NIR w/Phys 61.7 63.2 58.6 63.3 61.7 Flex wo/Phys 15.7 15.2 12.4 21.8 16.3 Flex w/Phys 90.9 61.1 54.0 62.2 67.1
19. ⚫ Video-based HR estimation based on physiological and illumination modeling • Comprehensive modeling can enhance performance Summary 18 RGB/NIR camera Ambient Illumination modeling RGB NIR NIR light Physiological modeling