SAVE Trial Design and Evaluation

Presentation on the design and evaluation of the Solent Achieving Value from Efficiency (SAVE) project trials. The presentation was delivered at the project closedown event at Central Hall, Westminster on 6th June 2019.

Transcript

1. June 2019 SAVE Close Down Event Trial evaluation
2. Trial evaluation • Experimental design ◦ SAVE: best practice trial design ◦ Power analysis and sample size ◦ Recruitment outcomes • Trial evaluation challenges ◦ Initial and revised analysis methods ◦ Timescales and reference points ◦ Attrition • Summary and recommendations
3. What is ‘best practice’? 3
4. 0 2 4 6 8 10 12 14 200 400 600 800 1000 1200 1400 Detectable % effect (p = 0.05) Trial Group Size Required Designed effect size Required trial group size Source: UoS analysis of Irish CER Domestic Demand Response pre-trial consumption data Mean kWh 16:00 – 20:00 (“Evening peak”) p = 0.05, P = 0.8 Þ Each trial group > 1000 Þ Control + 3 trial groups Þ Total sample > 4,000 households Statistical power and sample size
5. 5 Geographical location of SAVE project trial participants • Hampshire, Isle of Wight, Southampton, Portsmouth • Sampling stratified by the random selection of Census OAs within deprivation quintiles • Random selection of 50 addresses from each OA • Random allocation to treatment groups Sampling 4,318 households 32,000 letters
6. 6 § Income § Environmental attitudes Error bars: 95% Confidence Intervals Source: UoS analysis of SAVE vs Understanding Society Wave 4 sample for South East England (weighted for non-response) Recruitment outcomes: representative?
7. 7 § Electricity consumption § Environmental attitudes Error bars: 95% Confidence Intervals Source: UoS analysis of SAVE vs Understanding Society Wave 4 sample for South East England (weighted for non-response) Recruitment outcomes: biased?
8. Large sample size Statistically robust Random allocation to trial groups Equivalent groups: differences in consumption can be attributed to intervention Random, representative sample Results are generalisable to the wider population Recruitment outcomes
9. Consumption Wh Pre-intervention: (t0 ) Post-intervention: (t1 ) Observed trend: control group Observed trend: treatment group Treatment effect Analysis method – equivalent trial groups
10. Consumption Wh Pre-intervention: (t0 ) Post-intervention: (t1 ) Analysis method – asymmetrical groups Constant difference Treatment effect
11. Timescales – short and long-term effects
12. 12 Q1 Q2 Q4 Q3 Q1 Q2 Q4 Q3 2017 2018 Control TG1 Treatment TG2 Treatment TG4 Treatment TG3 TP1 TP2 TP3 LED lighting upgrades Treatment Group Trial Periods Sample attrition
13. Sample attrition
14. Extended evaluation period, however this resulted in: • Smaller sample • Increased uncertainty in estimated treatment effects • Difficulty in evaluating the maximum savings Sample attrition 800 households 550 households
15. SAVE delivered a robust, best practice trial design to provide industry-leading evidence base for estimating and modelling demand response • Even with careful design and implementation, the project faced evaluation challenges: ◦ Small asymmetries between groups required a new analytical approach ◦ Understanding responses to interventions required analysis across multiple time scales ◦ Attrition and communications issues over the trial increased uncertainty • Recommendations: ◦ Plan for asymmetry in trial groups even for RCTs with equivalent trial groups at trial start ◦ Be realistic about timescales around recruitment and interventions ◦ Adapt analysis approaches to each intervention ◦ Sample size: plan for attrition and communication issues Summary and recommendations
16. Thank you for listening. t.w.rushby@soton.ac.uk @tom_rushby #SAVEClosedown