Interpretable Deep Learning Model for Decoding Hypnotic Experience from Raw EEG Data

Transcript

1. Interpretable for Decoding Hypnotic Experience from Yeganeh Farahzadi, Morteza Ansarina, Zoltan Kekecs Raw EEG Data Deep Learning Model SCEH 74rd Annual Workshops & Scientific Program (October 2023) Eötvös Loránd University, University of Luxembourg
2. Outline • Introduction: Deep Learning and its potential application in understanding the cognitive neuroscience of hypnosis. • Methods: Implementing neural network using • Results and interpreting the implications of this study • Limitations and future directions Deep Learning Raw EEG Data
3. Introduction: Application of Deep Learning in Cognitive Neuroscience of Hypnosis
4. Bridge the Gap between Mind & Brain Jensen et al., (2017). New directions in hypnosis research: strategies for advancing the cognitive and clinical neuroscience of hypnosis. Kihlstrom, J. F. (2013). Neuro-hypnotism: Prospects for hypnosis and neuroscience. Cortex, 49(2), 365-374. • What are the neural mechanisms that support responses to hypnotic induction and suggestions? • What are the neurophysiological correlates of hypnotic suggestibility?
5. Bridge the Gap between Mind & Brain Jensen et al., (2017). New directions in hypnosis research: strategies for advancing the cognitive and clinical neuroscience of hypnosis. Kihlstrom, J. F. (2013). Neuro-hypnotism: Prospects for hypnosis and neuroscience. Cortex, 49(2), 365-374. • What are the neural mechanisms that support responses to hypnotic induction and suggestions? • What are the neurophysiological correlates of hypnotic suggestibility?
6. Bridge the Gap between Mind & Brain Data
7. Bridge the Gap between Mind & Brain Data De-noising
8. Bridge the Gap between Mind & Brain Data De-noising
9. Bridge the Gap between Mind & Brain Useful of data for the task at hand Data Representation De-noising
10. Bridge the Gap between Mind & Brain Useful of data for the task at hand Data Representation Task Deep or Superficial? De-noising
11. Bridge the Gap between Mind & Brain Useful of data for the task at hand Data Representation Task Deep or Superficial? De-noising A different way to look at data, to represent or encode the data. Time: Visual Representation Exact Representation Time elapsing measurement Quick inspection Arithmetic calculation Visual Representation
12. Task Representation Data
13. Task Data Representation & Rules
14. Is the circle Pink or Blue? Task Data Representation & Rules
15. Is the circle Pink or Blue? Task Data Rotation Representation & Rules
16. Is the circle Pink or Blue? Task Data X > 0 X < 0 Rotation Representation & Rules
17. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. Rotation Positive or Negative? Representation & Rules
18. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Positive or Negative? Representation & Rules
19. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Affective Norms for English Words (AFINN) Positive or Negative? Representation & Rules
20. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Affective Norms for English Words (AFINN) Positive or Negative? 0 4 0 3 4 0 0 0 -3 Representation & Rules
21. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Affective Norms for English Words (AFINN) Positive or Negative? 0 4 0 3 4 0 0 0 -3 Sum > 0 Sum < 0 Representation & Rules
22. Is the circle Pink or Blue? Task Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Affective Norms for English Words (AFINN) I went into a deep trance, but it was an unsettling experience. Positive or Negative? 0 4 0 3 4 0 0 0 -3 Sum > 0 Sum < 0 Representation & Rules
23. Is the circle Pink or Blue? Task Is the person in Deep hypnosis or Superficial? Data X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Affective Norms for English Words (AFINN) I went into a deep trance, but it was an unsettling experience. Positive or Negative? 0 4 0 3 4 0 0 0 -3 Sum > 0 Sum < 0 Representation & Rules
24. Is the circle Pink or Blue? Task Is the person in Deep hypnosis or Superficial? Data Gamma X > 0 X < 0 I felt an incredible sense of calm and relaxation Is the sentence I had a difficult time getting into a trance. "difficult," “time,” “get,” “trance,” Tokenisation, Removal of stop words "feel," “incredible,” “sense,” “calm,” “relaxation.” Rotation Processing Affective Norms for English Words (AFINN) I went into a deep trance, but it was an unsettling experience. Positive or Negative? 0 4 0 3 4 0 0 0 -3 Sum > 0 Sum < 0 Representation & Rules
25. Solution… Data Positive or Negative? Representation & Rules Task I felt an incredible sense of calm and relaxation
26. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation
27. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation
28. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation
29. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation
30. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation
31. Solution… Data Representation & Rules Positive or Negative? Task I felt an incredible sense of calm and relaxation Predict Masked Words
32. Solution… Data Representation & Rules Positive or Negative? Downstream Tasks Fine-tuning Task I felt an incredible sense of calm and relaxation Predict Masked Words
33. Solution… Data Representation & Rules Downstream Tasks Fine-tuning Task
34. Solution… Data Representation & Rules Downstream Tasks Fine-tuning Task Predict Masked Signal Values Deep or Superficial?
35. Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer Chollet, F. (2021). Deep learning with Python.
36. Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
37. w1 w2 w3 x1 x2 x3 Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
38. w1 w2 w3 x1 x2 x3 Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
39. w1 w2 w3 x1 x2 x3 Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
40. w1 w2 w3 x1 x2 x3 Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
41. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi Chollet, F. (2021). Deep learning with Python.
42. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation Chollet, F. (2021). Deep learning with Python.
43. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ̂ y Chollet, F. (2021). Deep learning with Python.
44. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score ̂ y Chollet, F. (2021). Deep learning with Python.
45. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python.
46. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python.
47. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python.
48. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python.
49. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python. Recurrent Layers Encode temporal information
50. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python. Recurrent Layers Encode temporal information Convolutional Layers (Conv2D) Encode spatial information
51. w1 w2 w3 x1 x2 x3 The “Deep” in deep learning stands for this idea of successive learning of representations. Learnable Parameters (weights) Input Layer 1st Hidden Layer 2nd Hidden Layer Output Layer 3 ∑ i=1 wi xi 1st layer of representation 2nd layer of representation ytrue Loss Score Feedback Signal ̂ y Chollet, F. (2021). Deep learning with Python. Recurrent Layers Encode temporal information (Conv1D) Encode spatial information Convolutional Layers (Conv2D) Encode spatial information
52. Objective & Data Data Representation & Rules Downstream Tasks Task Predict Masked Signal Values
53. Objective & Data Data Representation & Rules Downstream Tasks Task Predict Masked Signal Values 52 Participants listening to a neutral hypnosis induction (Elkins, 2014) Level of Hypnosis Depth 11-points Likert Scale
54. Objective & Data Data Representation & Rules Superficial or Deep? => 5 5 < Downstream Tasks Task Predict Masked Signal Values 52 Participants listening to a neutral hypnosis induction (Elkins, 2014) Level of Hypnosis Depth 11-points Likert Scale
55. Objective & Data Data Representation & Rules Superficial or Deep? => 5 5 < Downstream Tasks Task Predict Masked Signal Values 52 Participants listening to a neutral hypnosis induction (Elkins, 2014) Level of Hypnosis Depth 11-points Likert Scale EEG downsampled to 128 Hz, and normalised
56. Model H Task Predict Masked Signal Values Representation & Rules Data
57. Model Time Channels H Task Predict Masked Signal Values Representation & Rules Data
58. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| H Task Predict Masked Signal Values Representation & Rules Data
59. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| H Segm ents Task Predict Masked Signal Values Representation & Rules Data
60. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents Task Predict Masked Signal Values Representation & Rules Data
61. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Task Predict Masked Signal Values Representation & Rules Data
62. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Task Predict Masked Signal Values Representation & Rules Data
63. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Task Predict Masked Signal Values Representation & Rules Data Binary Classification Shallow or Deep? => 5 5 <
64. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Implementation Details Train/test split: 80% train, 20% validation Python Module: PyTorch (v2.0.1). Batch Size: 256 Learning rate: 1e-2 Optimiser: Adam Loss function: MSE, Cross-Entropy Task Predict Masked Signal Values Representation & Rules Data Binary Classification Shallow or Deep? => 5 5 <
65. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Implementation Details Train/test split: 80% train, 20% validation Python Module: PyTorch (v2.0.1). Batch Size: 256 Learning rate: 1e-2 Optimiser: Adam Loss function: MSE, Cross-Entropy Task Predict Masked Signal Values Representation & Rules Data Binary Classification Shallow or Deep? => 5 5 <
66. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Train Implementation Details Train/test split: 80% train, 20% validation Python Module: PyTorch (v2.0.1). Batch Size: 256 Learning rate: 1e-2 Optimiser: Adam Loss function: MSE, Cross-Entropy Task Predict Masked Signal Values Representation & Rules Data Binary Classification Shallow or Deep? => 5 5 <
67. Model Time Channels Spatial Encoder (Conv1D) Temporal Encoder (Recurrent) XCNN ℝ|segments|×|time|×|features| ℝ|segments|×|time|×|channels| ℝ|segments|×|features| H Segm ents ̂ XCNN Temporal Decoder (Recurrent) Spatial Decoder (Conv1D) Train Val Implementation Details Train/test split: 80% train, 20% validation Python Module: PyTorch (v2.0.1). Batch Size: 256 Learning rate: 1e-2 Optimiser: Adam Loss function: MSE, Cross-Entropy Task Predict Masked Signal Values Representation & Rules Data Binary Classification Shallow or Deep? => 5 5 <
68. Results
69. MSE Loss decreases in both training and test set after 20000 steps of training
70. Classification accuracy improved compared to traditional Machine Learning pipeline
71. Classification accuracy improved compared to traditional Machine Learning pipeline
72. Classification accuracy improved compared to traditional Machine Learning pipeline Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models.
73. Classification accuracy improved compared to traditional Machine Learning pipeline Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models.
74. Classification accuracy improved compared to traditional Machine Learning pipeline Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models. Implications • highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
75. Classification accuracy improved compared to traditional Machine Learning pipeline Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models. Train/test Split across time axis Implications • highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
76. Classification accuracy improved compared to traditional Machine Learning pipeline Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models. Train/test Split across time axis Train/test Split across subject axis Implications • highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
77. Implications Limitations • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
78. Implications • The train/test split was done along time axis. Limitations • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
79. Implications • The train/test split was done along time axis. • Small sample size. Limitations • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
80. Implications • The train/test split was done along time axis. • Small sample size. • Lack of any hyper-parameter Optimisations. Limitations • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
81. Implications • The train/test split was done along time axis. • Small sample size. • Lack of any hyper-parameter Optimisations. Limitations Future Directions • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
82. • Optimize hyperparamers of the model and incorporate larger sample size. Implications • The train/test split was done along time axis. • Small sample size. • Lack of any hyper-parameter Optimisations. Limitations Future Directions • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
83. • Optimize hyperparamers of the model and incorporate larger sample size. • Employ AI interpretability methods to understand the internal representations of the model. Implications • The train/test split was done along time axis. • Small sample size. • Lack of any hyper-parameter Optimisations. Limitations Future Directions • Highlights the effectiveness of deep learning in handling raw EEG data with minimal preprocessing. • Takes us a step closer to addressing the multifaceted neural correlates of hypnosis.
84. References Jensen, M. P., Jamieson, G. A., Lutz, A., Mazzoni, G., McGeown, W. J., Santarcangelo, E. L., ... & Terhune, D. B. (2017). New directions in hypnosis research: strategies for advancing the cognitive and clinical neuroscience of hypnosis. Neuroscience of consciousness, 2017(1), nix004. Kihlstrom, J. F. (2013). Neuro-hypnotism: Prospects for hypnosis and neuroscience. Cortex, 49(2), 365-374. Goodfellow, I. (2016). Deep Learning-Ian Goodfellow, Yoshua Bengio, Aaron Courville. Adapt. Comput. Mach. Learn. Chollet, F. (2021). Deep learning with Python. Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., & Faubert, J. (2019). Deep learning-based electroencephalography analysis: a systematic review. Journal of neural engineering, 16(5), 051001. Thomas, A. W., Ré, C., & Poldrack, R. A. (2023). Benchmarking explanation methods for mental state decoding with deep learning models. NeuroImage, 273, 120109. Farahzadi, Y., Alldredge, C., & Kekecs, Z. (2023). Neural Correlates of Hypnosis: Insights from EEG-Based Machine Learning Models.
85. Behavioral Medicine and Research Credibility Laboratory
86. Thank You :-) SECH Conference, 2023 yeganeh.farahzadi@ppk.elte.hu