Tweet
Tweet

Slide 1

Slide 1 text

Structured Decision-making and Adaptive Management For The Control Of Infectious Disease Christopher Fonnesbeck Vanderbilt University Matthew Ferrari Penn State University Katriona Shea Penn State University Michael Tildesley University of Edinburgh Michael Runge USGS Patuxent Petra Klepac Princeton University Dylan George US Department of Defense Scott Isard Penn State University Andrew Flack Defense Threat Reduction Agency Thursday, March 1, 12

Slide 2

Slide 2 text

Thursday, March 1, 12

Slide 3

Slide 3 text

Thursday, March 1, 12

Slide 4

Slide 4 text

Thursday, March 1, 12

Slide 5

Slide 5 text

Thursday, March 1, 12

Slide 6

Slide 6 text

Thursday, March 1, 12

Slide 7

Slide 7 text

Thursday, March 1, 12

Slide 8

Slide 8 text

Thursday, March 1, 12

Slide 9

Slide 9 text

Decisions under Uncertainty Thursday, March 1, 12

Slide 10

Slide 10 text

Foot-and-mouth Disease Tildesley et al., Nature 2006 Thursday, March 1, 12

Slide 11

Slide 11 text

2001 UK outbreak Thursday, March 1, 12

Slide 12

Slide 12 text

“What should we do?” Thursday, March 1, 12

Slide 13

Slide 13 text

Thursday, March 1, 12

Slide 14

Slide 14 text

Imperial College Model Deterministic, fast, approximates spatial structure Thursday, March 1, 12

Slide 15

Slide 15 text

Keeling Model Stochastic, spatial and flexible, but difficult to parameterize Thursday, March 1, 12

Slide 16

Slide 16 text

DEFRA Interspread Model “Black box” model for predictive Swine Fever in New Zealand vels of culling e the spatial se hotspots Vet Record, 2001. Thursday, March 1, 12

Slide 17

Slide 17 text

Thursday, March 1, 12

Slide 18

Slide 18 text

Thursday, March 1, 12

Slide 19

Slide 19 text

‣6.5 million or more livestock destroyed ‣Loss of export markets ‣Closure of markets, shows and footpaths ‣£2 billion direct, £3 billion indirect costs ‣Public distress (60 suicides) ‣Political upheaval ‣Debate and recriminations Costs of 2001 Outbreak Thursday, March 1, 12

Slide 20

Slide 20 text

Objective? Stop epidemic as quickly as possible? Minimize non-livestock economic losses? Minimize losses to farmers? Minimize political impact? Thursday, March 1, 12

Slide 21

Slide 21 text

What keeps us from making optimal decisions? Thursday, March 1, 12

Slide 22

Slide 22 text

Structured Decision-making Thursday, March 1, 12

Slide 23

Slide 23 text

1 Objectives Thursday, March 1, 12

Slide 24

Slide 24 text

2 Decision Alternatives Thursday, March 1, 12

Slide 25

Slide 25 text

3 Valuation of Outcomes Thursday, March 1, 12

Slide 26

Slide 26 text

4 Models of System Response to Actions Thursday, March 1, 12

Slide 27

Slide 27 text

Uncertainty Thursday, March 1, 12

Slide 28

Slide 28 text

aleatoric uncertainty Thursday, March 1, 12

Slide 29

Slide 29 text

epistemic uncertainty Thursday, March 1, 12

Slide 30

Slide 30 text

State Action Observation Stochasticity New State Thursday, March 1, 12

Slide 31

Slide 31 text

State Action Observation Stochasticity New State stochasticity Thursday, March 1, 12

Slide 32

Slide 32 text

State Action Observation Stochasticity New State Process Uncertainty process uncertainty Thursday, March 1, 12

Slide 33

Slide 33 text

partial observability State Action Observation Stochasticity New State Process Uncertainty Partial Observability Thursday, March 1, 12

Slide 34

Slide 34 text

partial controllability State Action Observation Stochasticity New State Process Uncertainty Partial Controllability Partial Observability Thursday, March 1, 12

Slide 35

Slide 35 text

Thursday, March 1, 12

Slide 36

Slide 36 text

“...as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns - the ones we don't know we don't know..." - D. Rumsfeld Thursday, March 1, 12

Slide 37

Slide 37 text

Sequential Decision Analysis Thursday, March 1, 12

Slide 38

Slide 38 text

passive management Thursday, March 1, 12

Slide 39

Slide 39 text

reactive management Thursday, March 1, 12

Slide 40

Slide 40 text

passive adaptive management Thursday, March 1, 12

Slide 41

Slide 41 text

active adaptive management Thursday, March 1, 12

Slide 42

Slide 42 text

Decide Models Objectives Alternative Actions Thursday, March 1, 12

Slide 43

Slide 43 text

Decide Monitor Models Objectives Alternative Actions Thursday, March 1, 12

Slide 44

Slide 44 text

Decide Monitor Learn Models Objectives Alternative Actions Update Thursday, March 1, 12

Slide 45

Slide 45 text

Adaptive Resource Management Thursday, March 1, 12

Slide 46

Slide 46 text

“Optimal decision-making in the face of uncertainty, with an aim to reducing uncertainty using informed management” Thursday, March 1, 12

Slide 47

Slide 47 text

State Reward Action New State Thursday, March 1, 12

Slide 48

Slide 48 text

Pr(xt+1 = j | xt = i,at (xt ) = d), d 2 A t State (t) State (t+1) State (t+2) Reward (t) Reward (t+1) Reward (t+2) Random Effects (t) Random Effects (t+1) Random Effects (t+2) Action (t) Action (t+1) Action (t+2) Thursday, March 1, 12

Slide 49

Slide 49 text

State (t) State (t+1) State (t+2) Reward (t) Reward (t+1) Reward (t+2) Random Effects (t) Random Effects (t+1) Random Effects (t+2) Action (t) Action (t+1) Action (t+2) Information State (t) Thursday, March 1, 12

Slide 50

Slide 50 text

= 0 x0 = 0 x1 = 1 x1 = 1 x1 = 0 x1 = 1 a0 = 0 a0 vaccinate no vaccinate no outbreak New Cases Cost 0 0 100 100 0 0 100 1000 1-p p 1-p p state action state Thursday, March 1, 12

Slide 51

Slide 51 text

= 0 x0 = 0 x1 = 1 x1 = 1 x1 = 0 x1 = 1 a0 = 0 a0 vaccinate no vaccinate no outbreak New Cases Cost 0 0 100 100 0 0 100 1000 1-p p 1-p p = 0.111 p∗ state action state Thursday, March 1, 12

Slide 52

Slide 52 text

= 0 x0 = 0 x1 = 1 x1 = 1 x1 = 0 x1 = 1 a0 = 0 a0 1-p p 1-p p = 0 x0 = 0 x1 = 1 x1 = 1 x1 = 0 x1 = 1 a0 = 0 a0 1-p p 1-p p Model 1 Model 2 Thursday, March 1, 12

Slide 53

Slide 53 text

Single Model Optimal Control Thursday, March 1, 12

Slide 54

Slide 54 text

action-value function Policy Thursday, March 1, 12

Slide 55

Slide 55 text

( , ) = E[ r( | ) ∣ ] Qπ st at ∑ i=t T ai si st total expected reward Thursday, March 1, 12

Slide 56

Slide 56 text

( , ) Qπ st at = E[r( | ) + r( | ) ∣ ] at st ∑ i=t+1 T ai si st = r( | ) + p( | , ) at st ∑ st+1 st+1 st at ×E[ r( | ) ∣ ] ∑ i=t+1 T ai si st+1 = r( | ) + p( | , ) ( , ) at st ∑ st+1 st+1 st at Qπ st+1 at+1 Thursday, March 1, 12

Slide 57

Slide 57 text

Bellman equation ( , ) = r( | ) + p( | , ) ( , ) Qπ st at at st ∑ st+1 st+1 st at Qπ st+1 at+1 current reward expected future reward Thursday, March 1, 12

Slide 58

Slide 58 text

optimal policy Thursday, March 1, 12

Slide 59

Slide 59 text

( , ) Q∗ st at max at = r( | ) + p( | , ) ( , ) ⎡ ⎣ at st ∑ st+1 st+1 st at Q∗ st+1 at+1 ⎤ ⎦ optimal policy Thursday, March 1, 12

Slide 60

Slide 60 text

Optimality For a given pathway to be optimal, all subsections of that path must also be optimal Thursday, March 1, 12

Slide 61

Slide 61 text

Multiple Models Optimal Control Thursday, March 1, 12

Slide 62

Slide 62 text

Average cumulative reward ( , |p) Q ˉπ st at = E[ (t) r( | ) ∣ ] ∑ m pm ∑ i=t T ai si st = (t) ( , ) ∑ m pm Qπ st at model weight Thursday, March 1, 12

Slide 63

Slide 63 text

( , |p) Q ˉπ st at = (t) r( | ) + p( | , ) ( , ) ∑ m pm ⎡ ⎣ ∑ i=t T ai si ∑ st+1 st+1 st at Qπ st+1 at+1 ⎤ ⎦ = ( | , p) + (t)p( | , ) ( , ) r ˉ ai si ∑ m ∑ st+1 pm st+1 st at Qπ st+1 at+1 Thursday, March 1, 12

Slide 64

Slide 64 text

Bayes’ Theorem (t + 1) pm = (t)p( | , ) pm st+1 st at (t)p( | , ) ∑ m pm st+1 st at = (t)p( | , ) pm st+1 st at ( | , ) p ˉ st+1 st at Thursday, March 1, 12

Slide 65

Slide 65 text

Optimal policy ( , |p) = ( | , p) + ( | , ) ( , ) Q ˉ∗ st at max at ⎡ ⎣ r ˉ ai si ∑ st+1 p ˉ st+1 st at Q ˉ∗ st+1 at+1 ⎤ ⎦ Thursday, March 1, 12

Slide 66

Slide 66 text

Finding an Optimal Policy Thursday, March 1, 12

Slide 67

Slide 67 text

Exhaustive Search Thursday, March 1, 12

Slide 68

Slide 68 text

States 3 Levels 10 Decision 2 Levels 5 Stochastic 2 Levels 5 Discretized Problem Thursday, March 1, 12

Slide 69

Slide 69 text

1.23 e172 yr using exhaustive search 1 Thursday, March 1, 12

Slide 70

Slide 70 text

Dynamic Programming Thursday, March 1, 12

Slide 71

Slide 71 text

Stochastic DP V⇤(xT )=max a2A xT E h rT (aT | xT ) i V⇤(xT 1 )= max a2A xT 1 E h r(aT 1 | xT 1 )+γV⇤(xT ) i . . . V⇤(xt )=max a2A xt E h r(at | xt )+γV⇤(xt+1 ) i T = Time horizon Thursday, March 1, 12

Slide 72

Slide 72 text

7.38 hours using dynamic programming Thursday, March 1, 12

Slide 73

Slide 73 text

States 6 Levels 30 Decision 9 Levels 5 Stochastic 3 Levels 9 More Complex Problem Thursday, March 1, 12

Slide 74

Slide 74 text

2.43 e12 yr using dynamic programming Thursday, March 1, 12

Slide 75

Slide 75 text

Alternatives? Thursday, March 1, 12

Slide 76

Slide 76 text

reinforcement learning Sutton & Barto 1998 Thursday, March 1, 12

Slide 77

Slide 77 text

Optimal strategy is learned by receiving reinforcement from a dynamic environment Thursday, March 1, 12

Slide 78

Slide 78 text

Q0(st,at ) = Q(st,at )+α[rt+1 +γQ(st+1,at+1 ) Q(st,at )] learning rate temporal difference learning difference between estimates Thursday, March 1, 12

Slide 79

Slide 79 text

Q (st, at ) = (1 )Q(st, at ) + [rt+1 + ⇥Q(st+1, at+1 )] Thursday, March 1, 12

Slide 80

Slide 80 text

exploitation exploration random action optimal action ϵ 1 − ϵ Thursday, March 1, 12

Slide 81

Slide 81 text

Initialize Q(s,a) Initialize s0 Choose initial action a0 from π Execute a Observe s' Choose next action a' from π Update Q(s',a') with r Advance s,a = s',a' Q(s,a) = Q(s',a') SARSA Repeat until convergence Initialization Thursday, March 1, 12

Slide 82

Slide 82 text

Initialize Q(s,a) Initialize s0 Choose initial action a0 from π Execute a Observe s' Choose next action a' from π Update Q(s',a') with r Advance s,a = s',a' Q(s,a) = Q(s',a') SARSA Repeat until convergence Initialization Q (s, a) = Q(s, a) + [r + ⇥Q(s , a ) Q(s, a)] Thursday, March 1, 12

Slide 83

Slide 83 text

Learning Thursday, March 1, 12

Slide 84

Slide 84 text

Value of Information Thursday, March 1, 12

Slide 85

Slide 85 text

Expected Value of Perfect Information Thursday, March 1, 12

Slide 86

Slide 86 text

EVPI = (t) ( ) − ( , ) ∑ i pi V∗ i xt V ˉ∗ xt pt Thursday, March 1, 12

Slide 87

Slide 87 text

Foot-and-mouth Disease Thursday, March 1, 12

Slide 88

Slide 88 text

Objective minimize cost of cattle + cost of vaccination Thursday, March 1, 12

Slide 89

Slide 89 text

IP Infected Premises DC Dangerous Contacts CP Contiguous Premises RC Ring Culling V Vaccination Decision Alternatives Thursday, March 1, 12

Slide 90

Slide 90 text

Kernel Models Distance from Source Transmission Risk Fat & Shallow UK Thin & Steep Thursday, March 1, 12

Slide 91

Slide 91 text

Vaccine Effectiveness 50% effective 10% susceptibility, 80% transmission 90% effective Thursday, March 1, 12

Slide 92

Slide 92 text

Belief Kernel Kernel IP IP/DC IP/DC/CP IP/DC/RC IP/DC/V Worst Best 25.0% 1 9.94 5.07 3.42 2.99 2.2 9.94 2.2 50.0% 2 5.1 1.9 1.41 2.59 1.90 5.1 1.41 25.0% 3 5.11 0.54 0.71 1.6 1.29 5.11 0.54 Average 6.31 2.35 1.74 2.44 1.82 6.31 1.74 Weighted minimum cost Weighted minimum cost Weighted minimum cost 1.39 Partial EVPI Partial EVPI 0.3475 (20%) 0.3475 (20%) 0.3475 (20%) Kernel EVPI Thursday, March 1, 12

Slide 93

Slide 93 text

Belief Effectiveness IP IP/DC IP/DC/CP IP/DC/RC IP/DC/V Worst Best 33.3% 90% 6.31 2.35 1.74 2.44 1.64 6.31 1.64 33.3% 10%/80% 6.31 2.35 1.74 2.44 1.70 6.31 1.70 33.3% 50% 6.31 2.35 1.74 2.44 2.14 6.31 1.74 Average 6.31 2.35 1.74 2.44 1.82 6.31 1.74 Weighted minimum cost Weighted minimum cost Weighted minimum cost 1.69 Partial EVPI Partial EVPI 0.0475 (2.7%) 0.0475 (2.7%) 0.0475 (2.7%) Vaccine Effectiveness EVPI Thursday, March 1, 12

Slide 94

Slide 94 text

Belief Kernel Vaccine IP IP/DC IP/DC/CP IP/DC/RC IP/DC/V Worst Best 8.3% 1 90% 9.94 5.07 3.42 2.99 2.19 9.94 2.19 16.7% 2 90% 5.1 1.9 1.41 2.59 1.68 5.1 1.41 8.3% 3 90% 5.11 0.54 0.71 1.6 1 5.11 0.54 8.3% 1 10/80% 9.94 5.07 3.42 2.99 2.19 9.94 2.19 16.7% 2 10/80% 5.1 1.9 1.41 2.59 1.7 5.1 1.41 8.3% 3 10/80% 5.11 0.54 0.71 1.6 1.19 5.11 0.54 8.3% 1 50% 9.94 5.07 3.42 2.99 2.22 9.94 2.22 16.7% 2 50% 5.1 1.9 1.41 2.59 2.33 5.1 1.41 8.3% 3 50% 5.11 0.54 0.71 1.6 1.68 5.11 0.54 Average 6.31 2.35 1.74 2.44 1.82 6.31 1.74 Weighted minimum cost Weighted minimum cost Weighted minimum cost 1.39 Partial EVPI Partial EVPI 0.3475 (20%) 0.3475 (20%) 0.3475 (20%) Combined EVPI Thursday, March 1, 12

Slide 95

Slide 95 text

Measles Thursday, March 1, 12

Slide 96

Slide 96 text

Thursday, March 1, 12

Slide 97

Slide 97 text

Thursday, March 1, 12

Slide 98

Slide 98 text

SEIR Model Susceptible Exposed Infectious Recovered Vaccination Thursday, March 1, 12

Slide 99

Slide 99 text

Objective min f( , ) ∑ t=1 T casest costt = min w( ) ∑ t=1 T casest +costt Thursday, March 1, 12

Slide 100

Slide 100 text

V0 V5 V10 V15 Management Actions No vaccination Vaccinate <5 years Vaccinate <10 years Vaccinate <15 years $0 $100 $175 $250 Cost Thursday, March 1, 12

Slide 101

Slide 101 text

Who Acquires Infection From Whom (WAIFW) ⎡ ⎣ 6 2 1 2 6 2 1 2 6 ⎤ ⎦ ⎡ ⎣ 3 3 3 3 3 3 3 3 3 ⎤ ⎦ assortative non-assortative Thursday, March 1, 12

Slide 102

Slide 102 text

Susceptibility Vaccination Strategy Vaccination Strategy Vaccination Strategy Model weight < 5 years < 10 years < 15 years Best Action < 5 years at risk .5 100 57 40 100 < 10 years at risk .25 25 86 60 86 < 15 years at risk .25 -12.5 42.8 70 70 Expected Benefit Expected Benefit 53.1 60.7 52.5 88.9 Thursday, March 1, 12

Slide 103

Slide 103 text

Best Static Strategy risk <5 risk <10 risk <15 0.8 0.2 0.8 0.6 0.4 0.6 0.4 0.6 0.4 0.2 0.8 0.2 Thursday, March 1, 12

Slide 104

Slide 104 text

Thursday, March 1, 12

Slide 105

Slide 105 text

Thursday, March 1, 12

Slide 106

Slide 106 text

Adaptive Monitoring Thursday, March 1, 12

Slide 107

Slide 107 text

When is adaptive management useful? Thursday, March 1, 12

Slide 108

Slide 108 text

When is adaptive management useful? ➊ Sequential decision-making Thursday, March 1, 12

Slide 109

Slide 109 text

When is adaptive management useful? ➊ Sequential decision-making ➋ Decisions influence system behavior Thursday, March 1, 12

Slide 110

Slide 110 text

When is adaptive management useful? ➊ Sequential decision-making ➋ Decisions influence system behavior ➌ There is uncertainty regarding the system and the expected consequences of decisions Thursday, March 1, 12

Slide 111

Slide 111 text

Limitations of AM ➊ Requires informative monitoring ➋ Management body involved in all steps ➌ Institutional challenges ➍ Requires flexibility Thursday, March 1, 12

Slide 112

Slide 112 text

Promise of AM ➊ Forces articulation of objectives, constraints, costs ➋ Explicit quantification of uncertainty ➌ Transparent mechanism for choosing among decision alternatives Thursday, March 1, 12