A Computational Model of Control Allocation Based on the Expected Value of Control (ICPS 2019)

Transcript

1. Sebastian Musslick Princeton Neuroscience Institute, Princeton University ICPS 2019 Symposium - Neural Mechanisms of Effort Mobilization and Cognitive Control
2. Cognitive control – reconfigure information processing away from default (automatic) settings (Cohen et al., 1990; Botvinick & Cohen, 2015) read email follow talk
3. A Computational Model of Control Allocation Based on the Expected Value of Control A. The Theory B. The Model C. Simulations & Predictions D. Estimating Mental Effort from Behavior
4. A Computational Model of Control Allocation Based on the Expected Value of Control A. The Theory B. The Model C. Simulations & Predictions D. Estimating Mental Effort from Behavior
5. RED
6. A Theory of Control Allocation: Expected Value of Control GREEN EVC(signal,state) = Pr(outcome i i ∑ | signal,state)⋅Value(outcome i ) # $ % & ' (−Cost(signal) Shenhav, Botvinick, & Cohen (2013)
7. A Theory of Control Allocation: Expected Value of Control Control Signal Intensity
8. A Theory of Control Allocation: Expected Value of Control Control Signal Intensity Payoff
9. A Theory of Control Allocation: Expected Value of Control Control Signal Intensity Payoff Cost
10. A Theory of Control Allocation: Expected Value of Control Control Signal Intensity Payoff Cost EVC
11. A Theory of Control Allocation: Expected Value of Control Control Signal Intensity Payoff Cost EVC
12. A Computational Model of Control Allocation Based on the Expected Value of Control A. The Theory B. The Model C. Simulations & Predictions D. Estimating Mental Effort from Behavior
13. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift = driftCONTROL + driftAUTOMATIC
14. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift = driftCONTROL + driftAUTOMATIC
15. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence AUTOMATIC drift = driftCONTROL + ! WORD + ! COLOR
16. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence AUTOMATIC drift = driftCONTROL + ! WORD + ! COLOR
17. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence AUTOMATIC drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR CONTROL
18. Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = · + · + +
19. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = · + · 1 + + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence
20. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = · + · 1 + + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence
21. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = ·(−10)+ · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence
22. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = 0 ·(−10)+ 0 · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence
23. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = 0 ·(−10)+ 0 · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift = −9 t 0
24. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = 0 ·(−10)+ 0 · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift =
25. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = 0 ·(−10)+ 15 · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift =
26. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR drift = 0 ·(−10)+ 15 · 1 + (−10) + 1 Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence drift = 6
27. drift = ! WORD ·" WORD + ! COLOR ·" COLOR + " WORD + " COLOR Computational Model How is Control Implemented? Agent Task Environment Drift Diffusion Model (DDM; Ratcliff, 1978; Bogacz; 2006) t 0 “red” “green” accumulated evidence #(%&''()*|,-., 0) 23(%&''()*|,-., 0) drift = 0 ·(−10)+ 15 · 1 + (−10) + 1 drift = 6
28. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) Control Allocation
29. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 1. Simulate performance Control Allocation
30. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance &'( !, * +,- = .(0122345| * +,-, 7) · :3;<2= − 01?5(7) Control Allocation
31. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance &'( !, * +,- = .(0122345| * +,-, 7) · :3;<2= − 01?5(7) Control Allocation
32. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance Control Allocation &'( !, * +,- = .(0122345| * +,-, 7) · :3;<2= − 01?5(7)
33. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) !WORD = 0 !COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance Control Allocation &'()*'*+,-,./+ 0/1,(!) ! 4*5/+6.7!8-,./+ 0/1,(!) 9! + ;<= !, ? @+A = B(0/88*5,| ? @+A, D) · 4*F-8G − 0/1,(D)
34. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance Control Allocation &'( !, * +,- = .(0122345| * +,-, 7) · :3;<2= − 01?5(7) 3. Select control signal that maximizes EVC 7∗ = max D &'( * +,-, 7
35. Computational Model How is Control Allocated? Agent Task Environment t 0 “red” “green” accumulated evidence Control Implementation Musslick, Shenhav, Botvinick & Cohen (2015, RLDM) ! WORD = 0 ! COLOR = 15 t 0 2. Compute Expected Value of Control (EVC) 1. Simulate performance Control Allocation &'( !, * +,- = .(0122345| * +,-, 7) · :3;<2= − 01?5(7) 3. Select control signal that maximizes EVC 7∗ = max D &'( * +,-, 7
36. A Computational Model of Control Allocation Based on the Expected Value of Control A. The Theory B. The Model C. Simulations & Predictions D. Estimating Mental Effort from Behavior
37. Cognitive Control Phenomena Basic Phenomena Incongruency Costs (Stroop, 1935) Switch Costs (Allport, 1994) Effects of Incentives on Task Performance Distractor Interference/Facilitation (Padmala & Pessoa) Reward & Switch Costs (Umemoto & Holroyd, 2015) Effects of Incentives on Task Choice Demand Avoidance (Kool et al., 2010) Cognitive Effort Discounting (Westbrook & Braver, 2015) Reward and Voluntary Task Switching (Arrington & Braun, 2017) Adaptation to Task Difficulty Congruency Sequence Effect (Gratton, 1992) Proportion Congruency Effect (Logan & Zbrodoff, 1979) Non-Monotonicity in Task Engagement (Gilzenrat, 2010) Stability-Flexibility Tradeoff (Goschke, 2000)
38. Cognitive Control Phenomena Basic Phenomena Incongruency Costs (Stroop, 1935) Switch Costs (Allport, 1994) Effects of Incentives on Task Performance Distractor Interference/Facilitation (Padmala & Pessoa) Reward & Switch Costs (Umemoto & Holroyd, 2015) Effects of Incentives on Task Choice Demand Avoidance (Kool et al., 2010) Cognitive Effort Discounting (Westbrook & Braver, 2015) Reward and Voluntary Task Switching (Arrington & Braun, 2017) Adaptation to Task Difficulty Congruency Sequence Effect (Gratton, 1992) Proportion Congruency Effect (Logan & Zbrodoff, 1979) Non-Monotonicity in Task Engagement (Gilzenrat, 2010) Stability-Flexibility Tradeoff (Goschke, 2000)
39. Effects of Incentives on Task Performance Padmala & Pessoa (2011)
40. Effects of Incentives on Task Performance Padmala & Pessoa (2011) Data
41. Effects of Incentives on Task Performance Padmala & Pessoa (2011) Data
42. Effects of Incentives on Task Performance Padmala & Pessoa (2011) Data EVC Model
43. Effects of Incentives on Task Performance Padmala & Pessoa (2011) Data EVC Model
44. Effects of Incentives on Task Choice Arrington & Braun (2018)
45. Effects of Incentives on Task Choice Arrington & Braun (2018) Data
46. Effects of Incentives on Task Choice Arrington & Braun (2018) Data
47. Effects of Incentives on Task Choice Arrington & Braun (2018) Data
48. Effects of Incentives on Task Choice Arrington & Braun (2018) Data EVC Model
49. Effects of Incentives on Task Choice Arrington & Braun (2018) Data EVC Model
50. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape
51. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape
52. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape Data
53. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape Data EVC Model
54. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape
55. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape Human Participants
56. Adaptation to Task Difficulty Gilzenrat et al. (2010) Task Difficulty Trial Escape EVC Model Human Participants
57. A Computational Model of Control Allocation Based on the Expected Value of Control A. The Theory B. The Model C. Simulations & Predictions D. Estimating Mental Effort from Behavior
58. Estimating Mental Effort from Behavior Limitations in Ability to Recover Individual Differences Validity of estimated costs: Correlation between true cost of control and estimated cost of control 0 5 10 Standard Deviation of Reward Sensitivity v 0 0.5 1 Correlation Between True and Estimated Control Costs reward sensitivity Musslick, Cohen & Shenhav (2018, CogSci)
59. Estimating Mental Effort from Behavior Spurious Correlations Paradigm A Paradigm B Proxy A for Cost of Cognitive Control Proxy B for Cost of Cognitive Control ? 0 0.5 1 Correlation Between Reward Sensitivity v Across Experiments -1 -0.5 0 0.5 1 Correlation Between Control Costs Across Experiments True Correlation reward sensitivity Musslick, Cohen & Shenhav (2018, CogSci)
60. A Theory of Control Allocation: Expected Value of Control GREEN EVC(signal,state) = Pr(outcome i i ∑ | signal,state)⋅Value(outcome i ) # $ % & ' (−Cost(signal) Shenhav, Botvinick, & Cohen (2013)
61. Jonathan Cohen Amitai Shenhav Thank you! Acknowledgements Matthew Botvinick