Slide 1
Slide 1 text
June 2019
SAVE Close Down Event
Trial evaluation
Slide 2
Slide 2 text
Trial evaluation
• Experimental design
◦ SAVE: best practice trial design
◦ Power analysis and sample size
◦ Recruitment outcomes
• Trial evaluation challenges
◦ Initial and revised analysis methods
◦ Timescales and reference points
◦ Attrition
• Summary and recommendations
Slide 3
Slide 3 text
What is ‘best practice’?
3
Slide 4
Slide 4 text
0
2
4
6
8
10
12
14
200 400 600 800 1000 1200 1400
Detectable % effect (p = 0.05)
Trial Group Size Required
Designed effect size
Required trial group size
Source: UoS analysis of Irish CER Domestic Demand Response pre-trial consumption data
Mean kWh 16:00 – 20:00 (“Evening peak”)
p = 0.05, P = 0.8
Þ Each
trial group > 1000
Þ
Control + 3 trial groups
Þ
Total sample > 4,000 households
Statistical power and sample size
Slide 5
Slide 5 text
5
Geographical location of SAVE
project trial participants
• Hampshire, Isle of Wight,
Southampton, Portsmouth
• Sampling stratified by the
random selection of Census OAs
within deprivation quintiles
• Random selection of 50
addresses from each OA
• Random allocation to treatment
groups
Sampling
4,318 households
32,000 letters
Slide 6
Slide 6 text
6
§ Income § Environmental attitudes
Error bars: 95% Confidence Intervals
Source: UoS analysis of SAVE vs Understanding Society Wave 4 sample for South East England
(weighted for non-response)
Recruitment outcomes: representative?
Slide 7
Slide 7 text
7
§ Electricity consumption § Environmental attitudes
Error bars: 95% Confidence Intervals
Source: UoS analysis of SAVE vs Understanding Society Wave 4 sample for South East England
(weighted for non-response)
Recruitment outcomes: biased?
Slide 8
Slide 8 text
Large sample
size Statistically robust
Random
allocation to trial
groups
Equivalent groups: differences
in consumption can be
attributed to intervention
Random,
representative
sample
Results are generalisable to
the wider population
Recruitment outcomes
Slide 9
Slide 9 text
Consumption Wh
Pre-intervention:
(t0
)
Post-intervention:
(t1
)
Observed trend:
control group
Observed trend:
treatment group
Treatment effect
Analysis method – equivalent trial groups
Slide 10
Slide 10 text
Consumption Wh
Pre-intervention:
(t0
)
Post-intervention:
(t1
)
Analysis method – asymmetrical groups
Constant
difference
Treatment effect
Slide 11
Slide 11 text
Timescales – short and long-term effects
Slide 12
Slide 12 text
12
Q1 Q2 Q4
Q3 Q1 Q2 Q4
Q3
2017 2018
Control TG1
Treatment TG2
Treatment TG4
Treatment TG3
TP1 TP2 TP3
LED lighting upgrades
Treatment Group
Trial Periods
Sample attrition
Slide 13
Slide 13 text
Sample attrition
Slide 14
Slide 14 text
Extended evaluation period,
however this resulted in:
• Smaller sample
• Increased uncertainty in
estimated treatment effects
• Difficulty in evaluating the
maximum savings
Sample attrition
800 households
550 households
Slide 15
Slide 15 text
SAVE delivered a robust, best practice trial design to provide industry-leading evidence
base for estimating and modelling demand response
• Even with careful design and implementation, the project faced evaluation challenges:
◦ Small asymmetries between groups required a new analytical approach
◦ Understanding responses to interventions required analysis across multiple time
scales
◦ Attrition and communications issues over the trial increased uncertainty
• Recommendations:
◦ Plan for asymmetry in trial groups even for RCTs with equivalent trial groups at
trial start
◦ Be realistic about timescales around recruitment and interventions
◦ Adapt analysis approaches to each intervention
◦ Sample size: plan for attrition and communication issues
Summary and recommendations