Review: Fast Sequences of Non-spatial State Representations in Humans

A 2-hour review talk of the following article, at Neuroimaging Journal Club, School of Cognitive Sciences, IPM, Tehran, Iran:

Kurth-Nelson, Z., Economides, M., Dolan, R. J., & Dayan, P. (2016). Fast Sequences of Non-spatial State Representations in Humans. Neuron, 91(1), 194–204. http://doi.org/10.1016/j.neuron.2016.05.028

Transcript

1. Fast Sequences of Non-spatial
 State Representations in
 Humans Zeb Kurth-Nelson,
 Marcos Economides,
 Raymond J. Dolan,
 and Peter Dayan Morteza Ansarinia Institute for Cognitive Science Studies Neuroimaging Journal Club
 School of Cognitive Sciences, IPM September 6, 2016
2. Outline ✦ Introduction and Overview ‣ MEG ✦ Design of Experiment ✦ Non-Spatial Task ✦ Object Decoding ✦ Multivariate Analysis ✦ Results ✦ Discussions ✦ Conclusion Kurth-Nelson, et al. (2016). Fast Sequences of Non-spatial State Representations in Humans. Neuron, 91(1), 194–204. 2
3. Overview ✦ Decoupled from sensory and motor signals, neural activities can also represent and replay past and possible future states: ‣ In memory consolidation, ‣ imaginations, ‣ learning, ‣ planning and looking ahead, ‣ and decision making. ✦ Studies in rodent hippocampus show decoupled neural activities occur in form of internally generated coherent sequences that encode trajectories through past or future states of spatial tasks. ✦ These fast sequences may be a fundamental feature of neural computation across spatial and non-spatial tasks. However, these sequences have only been studied in the context of spatial tasks in rodents, and never in humans. Wikenheiser & Redish (2015), Pezzulo et al. (2014), and Moser (2013) 3
4. Overview ✦ They show that spontaneous MEG activity plays out fast sequences of state representations, after participants have learned a non-spatial navigation task. ✦ These sequences formed trajectories of up to 4 states that presented backward through the connections of the task, with state-to-state lag of 40 milliseconds. So the whole four-step trajectory lasted on the order of 120 milliseconds. ✦ It is the first time such sequences are observed in humans, as well as the first time in non-spatial task setting. ✦ These results suggest that fast non-local sequences may a fundamental neural mechanism in decision making that is conserved across species and across problem domains. Wikenheiser & Redish (2015), Pezzulo et al. (2014), Moser (2013), and Takahashi (2015) 4
5. Introduction ✦ Most brain areas not only are involved in encoding current inputs, context, and motor outputs, but also they (decoupled from inputs) encode past experiences and possible future states. ✦ Place cells in rodents spontaneously play out (“replay”) sequences of other positions. Studies suggest these sequences occurs in two main contexts: ‣ Within sharp-wave ripple events (SWRs), ‣ and nested in theta waves. ✦ These fast sequences observed in sleep and wakefulness, and also in various kind of spatial tasks. Mnih et al. (2016), Foster & Wilson (2006), Wikenheiser & Redish (2015), van der Meer et al. (2010), and Kurth-Nelson (2015). 5
6. Introduction ✦ Possible functions: ‣ Learning: consolidating or maintaining knowledge in cortex; and temporal compression of events to bring them within a time frame of synaptic plasticity (e.g. for credit assignment). ‣ Decision Making & Planning: in online/offline planning or predictions in animals, sequences help them predict the path they will run in the immediate future. ✦ The goal of the study was to investigate spontaneous neural sequences in a non-spatial context in healthy human volunteers. ✦ Authors used multivariate analysis of MEG data to decode time- resolved representations of visual objects that were not currently being experienced. Therefore they reasoned it might be possible to detect fast sequences using MEG signals of decodable visual but absent objects. Mnih et al. (2016), Foster & Wilson (2006), Wikenheiser & Redish (2015), van der Meer et al. (2010), and Kurth-Nelson (2015). 6
7. Magnetoencephalography (MEG) ✦ Magnetic fields are produced by synchronized post-synaptic currents of dendrites of pyramidal neuron. ✦ Signals are amplified, then measured using flux transformers plugged to SQUIDs. ✦ Instead of magnetometers, MEG uses axial or planar gradiometers and reference sensors. ✦ 150-300 sensors are needed to cover all the head, to record at 1kHz. ✦ Compared to EEG and fMRI: Non-invasive, high spatiotemporal resolutions, unaffected by scalp distortions, more coils and sensors, shows absolute neural activities, can be recorded for sleeping subjects, no operational noise, does not require stillness, and its helmet is so cool. SQUID: Supper Conducting Quantum Interface Device 7
8. Magnetoencephalography (MEG) Hillebrand & Barnes (2002). 8
9. Magnetoencephalography (MEG) Image Courtesy of Human Connectome Project. © 2016 Human Connectome Project. R-fMRI vs. R-MEG 9
10. Non-spatial Seq. and Hippocampus ✦ An agent should build abstract cognitive maps for non-spatial as well as spatial tasks. In computational view, these cognitive mapping is done by hippocampus and associated brain areas. As a result, hippocampus should express sequences in non-spatial tasks. ✦ MEG contains signals from hippocampus, but they are relatively difficult to detect, rendering it unlikely that recorded sequences arose directly from hippocampus: ‣ Cortical activities dominates other MEG signals. ‣ Classifiers were trained on visual responses shortly after stimulus onset, and might reflect cortical visual processing. ‣ Regressions were based on mostly occipital and posterior temporal sensors. ✦ Another possible scenario: The observed sequences corresponded to reactivation of visual representations during retrieval or remembering. ✦ It is possible hippocampus generates sequences (perhaps during SWRs) that drives cortical activities detected in this article. ‣ The coupling between hippocampus and cortex during SWRs was observed before. ‣ However, their data provides no direct evidence for such an upstream role for hippocampus, and the sequences stem from intrinsic cortical dynamics. Kuhn & Chun (2014), Tanaka et al. (2014), Ji and Watson (2007), Tolman (1984), and Euston et al. (207). 10
11. Hippocampus Nature Reviews Neuroscience 15, 655–669 (2014). Copyright © 1993 Wiley-Liss, Inc., and Nature Publishing Group. 11
12. Hippocampus Copyleft 2016. Morteza Ansarinia. 12
13. Design of Experiment — Maze ✦ Participants performed a novel 6-state non-spatial navigation task. ✦ Each state was defined by a unique visual object, and a varying reward. ✦ Available choices: UP and DOWN, leading to different state ✦ Before scanning: Subjects were trained to learn structure of the task. ✦ During scanning: ‣ Initiated from a random state, they were asked for to enter a sequence of 4 moves (i.e. a path around a maze). ‣ Goal: Collect as much as reward as possible. ‣ Simple stimulus-response learning (RL) strategy was discouraged by: - Randomized reward by ±1. - Informed negation states that flipped cumulative reward by -1. - No bird’s-eye view of the maze. ‣ No subjective spatial structure were perceived, when asked during debriefing. Next
 Page 13
14. Design of Experiment — Maze Randomized
 Rewards 14 Design of Experiment — Maze Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors. 14
15. Design of Experiment ✦ To verify the neural sequences only occur if participants learned the transition: ‣ During training phase, subjects achieved 100% accuracy on a set of questions, in which knowledge of transition structure was probed. ‣ In debriefing, participants subjectively reported if they deployed their knowledge of transitions to plan during scans (self-reports). ‣ Behavioral data strongly favored using knowledge of transitions for planning over simple stimulus-response (reinforcement learning) reactions. They compared four policies including Plan, Qfirst, Qall, and Greedy reactions. Next Page 15
16. Behavioral Model Comparisons Group Level 16 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors.
17. ✦ Starting state was revealed and participant was allowed 60s to plan ahead (planning time), by pressing UP or DOWN keys. ✦ Task performance (collected reward) was higher
 for 9 out of 12 subjects: ‣ Longer planning time 㱻 More earning ‣ More money 㱻 Shorter planning time Design of Experiment 17 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors. 17
18. Multivariate Models ✦ To recognize spatial pattern of MEG elicited by direct visual object presentation, each object and subject trained to a model by a lasso-regularized logistic regression, one for each object. ‣ Lasso penalty encouraged sparsity (made most coefficients zero) and to select occipital and posterior temporal sensors. ‣ Training Data: gathered from a secondary task (the object was presented multiple times in random order). Based on Kurth-Nelson (2015) only MEG data 200ms after onset was recorded for training. ✦ Models were cross-validated to confirm they captured essential object-related features in the MEG signals, and repeated with randomization for 100 times. Overall, all models predicted probability of a label for an object correctly (decoded object as expected by training phase). ✦ Prediction accuracy: The set of models together was used to as making a categorical prediction about a class of the left-out data (the one with highest probability). ‣ Cross-validated Prediction Accuracy = 53.7% ± 3.8% ‣ Chance = 16.7% Kurth-Nelson et al. (2015) Check out next page for the figures 18
19. Multivariate Models — Decoding ✦ The effect of lasso penalty to make most coefficients zero, and
 make occipital and posterior temporal sensors salient of a
 single trial. ✦ Of the 18×6=108 models used in analysis, shows the fraction
 where each sensor was non-zero. ✦ Example of the six probability time series output by the six
 regression models. ✦ From top row: raw data, learned betas, and
 transformed probability. 19 19 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors.
20. Decode Object Representations ✦ Data collected in the the planning phase of each trial, and fed to train a regression model. ✦ Each 10 milliseconds time bin of MEG data was independently input to each of six models (one for each object), yielding six time series of probabilities (of time t for object k). ✦ P(k,t) quantifies the degree to which the spatial pattern of MEG activity at time t resembles the evoked neural response to visual object k. ✦ Sequenceness: A measure that quantifies whether a decoded neural representation of state S1 was likely followed by a decoded representation of S2; or in case of reverse sequences, whether S2 would be followed by S1. It was modeled using cross-correlation measure. ✦ Method: Study time series for possible paths in the behavioral task measured by the sequenceness metric. See “Experimental Procedures” in supplementary materials for details (Kurth-Nelson, 2016). 20
21. Decode Object Representations ✦ Sequences could either be forward (S1➝S2) or reversed (S2➝S1). They observer a peak in reversed sequenceness at 40 milliseconds of lag;
 means at t+40 the same or a connected
 second state is expected. ✦ The peak was statistically non-zero, tested
 using a multilevel model with a
 fixed+random intercept. ✦ Small sample size was rationalized by
 consistency of sequenceness between
 participants. Moreover, they used both parametric and non-parametric tests, and cross-validations to ensure free parameter of logistic regression does not affect the results. See “Experimental Procedures” in supplementary materials for details (Kurth-Nelson, 2016). 21 Check out next page for the figures
22. Decode Object Representations 22 22 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors.
23. Decode Object Representations 23 23 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors.
24. Length of Sequences ✦ By using same multimodal and modifying intercepts,
 they concluded that spontaneous state
 representations tended to occur as fast sequences
 of up to 4 consecutive states. ✦ Given 40 milliseconds as the strongest state-to-state
 lag, they estimated a 4-state sequence lasted
 120 milliseconds. ✦ The sequences temporally compressed in time relative to the real experience by factor of 25-100 (Visual cross-fade between objects took 350 milliseconds). ✦ Similar, hippocampal sequence events in rodents last on the order of 50-200 milliseconds both in theta and SWRs. Diba & Buzaski (2007), Dragon & Tongawa (2011), and Wikenheiser and Redish (2015). 24
25. Negative Results ✦ On average (mean statistic) there was no ‣ significant relationship between planning time and earning, ‣ significant relationship between planning time and sequenceness, ‣ or significant relationship between earning and sequenceness. ✦ However absence of significant differences could be due to limited sample size (low power). ✦ No significant trial-by-trial relationship (correlation) between magnitude of sequenceness against earnings. ✦ No significant trial-by-trial relationship (correlation) between magnitude of sequenceness against planning time on the same trial. ✦ No greater magnitude of sequenceness for a specific state-tuple, in a chosen or unchosen setting of sequences, and even compared to the previous trials. Statistical Techniques: ANOVA, regression on subject means, linear mixed effects, and cross-correlations. 25
26. Forward vs. Reverse Sequences ✦ Analysis was based on subtracting forward from reverse sequenceness. ✦ At 40 milliseconds lag, there was much stronger expression of reverse compared to forward sequenceness. ‣ Either increase in reverse sequenceness, ‣ or reduction in forward sequenceness. ✦ Authors argue reduction in forward sequenceness is unlikely due to implying consistent positive amount not only at 40 milliseconds by all latencies. 26 Neuron 91, 194-204 (2016), DOI: 10.1016/j.neuron.2016.05.028. Copyright © 2016 The Authors. 26
27. Conclusion ✦ The power of multivariate analysis of MEG data to trace the trajectories of fast-evolving neural representations in human. ✦ Non-invasive, and ability to track representations across wide cortical area. ✦ They claim it was the first evidence for fast spatial and non-spatial spontaneous sequences of state representations in human brain (thus in a wide range of cognitive domains). 27
28. Thank You :-)