Harnessing uncertainty: The role of probabilistic time series forecasting in the renewable energy transition

How can probabilistic forecasting accelerate the renewable energy transition? The rapid growth of non-steerable and intermittent wind and solar power requires accurate forecasts and the ability to plan under uncertainty. In this talk, we will make a case for using probabilistic forecasts over deterministic forecasts. We will cover methods for generating and evaluating probabilistic forecasts, and discuss how probabilistic price and wind power forecasts can be combined to derive optimal short-term power trading strategies.

Transcript

1. 1 Harnessing uncertainty The role of probabilistic time series forecasting in the renewable energy transition Alexander Backus Data Science Manager www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash Amsterdam 15 September 2023
2. Climate change: Our planet's most pressing challenge 2 Photo by William Bossen on Unsplash * ourworldindata.org/ Monthly temperature anomaly* ℃
3. The energy sector accounts for over 73% of greenhouse gas emissions 3 Photo by Michal Pech on Unsplash * ourworldindata.org/ *
4. The energy transition: Pivoting towards renewables 4 Photo by Nicholas Doherty on Unsplash * ourworldindata.org/ TWh Renewable electricity generation in NL*
5. A balancing act on the energy grid: Supply needs to equal demand at any moment 5 supply t demand t =
6. 6 Renewable generation is non-steerable and intermittent Power (MWh) Time (days) 0 1 2 3 4
7. 7 Energy markets match supply and demand through price Time 1 day before (12:00) Time of delivery Market Day-ahead market Balancing market Power 100 MWh (sold) 80 MWh 🔽(generated) Price 100 EUR/MWh 300 EUR/MWh 🔼 imbalance cost = 𝚫power · 𝚫price = (100 - 80 MWh) ✕ (300 - 100 EUR/MWh) = 4000 EUR Shortage on the grid! 👎 By hypothetical renewable energy supplier Timeline t 0 t -1
8. 8 Making renewable energy more predictable and profitable Founded in 2016, Amsterdam ~50 FTE and growing Dexter Energy provides short-term power forecasting and trade optimization for renewable portfolios
9. 9 Two key unknown quantities: volume and price Power forecast Price forecast Optimization Recommended trade imbalance cost = 𝚫power · 𝚫price
10. 10 Time 1 day before (12:00) Time of delivery Market Day-ahead market Balancing market Power 75 MWh (sold) 🔽 80 MWh (generated) Power forecast NA 80 MWh Price 120 EUR/MWh 🔼 220 EUR/MWh 🔽 Price forecast 100 EUR/MWh 300 EUR/MWh = -5 MWh ✕ 100 EUR/MWh = -500 EUR Lower 𝚫price and less grid imbalance! 👍 How do we help balance the grid? Sell: 🙋 A) As much as we can B) Nothing C) A bit more D A bit less How price forecasts can help balance the grid imbalance cost = 𝚫power · 𝚫price
11. 11 Time 1 day before (12:00) Time of delivery Market Day-ahead market Balancing market Power 0 MWh (sold) 🔽🔽 80 MWh (generated) Power forecast NA 80 MWh Price 150 EUR/MWh 🔼🔼 -10 EUR/MWh 🔽🔽 Price forecast 100 EUR/MWh 300 EUR/MWh How do we help balance the grid? Sell: 🙋 A) As much as we can B Nothing C) A bit more D A bit less = -80 MWh ✕ -160 EUR/MWh = 12800 EUR Market flipped into a surplus state! 👎 How price forecasts can help balance the grid imbalance cost = 𝚫power · 𝚫price ⚠ Price maker effect
12. 12 Quantify uncertainty to allow risk-based steering t -1 Time Price (EUR/MWh) Actuals Forecast Point forecast q80 q60 q40 q20 P price ( y | x, t ) Point forecast q80 Price (EUR/MWh) Probability density slice
13. 13 Quantify uncertainty to allow risk-based steering t -1 Time Price (EUR/MWh) Actuals Forecast Point forecast q80 q60 q40 q20 Point forecast Price (EUR/MWh) Probability density P price ( y | x, t )
14. 14 Volume forecast Price forecast Optimization Recommended trade Market Weather Time series regression problem Trading actions How to obtain probabilistic forecasts? X f ŷ Features ⚠ Price taker assumption Model
15. 15 Residuals of a calibration set give a baseline estimate of the typical uncertainty profile Training Testing 1. Train point forecast model ŷ = f ( X ) Calibration 2. Predict on calibration set and get error distribution Error 3. Overlay error distribution on new point predictions ŷ Price
16. 16 Calibration and sharpness define the quality of a probabilistic forecast Sharpness Calibration Theoretical quantile Sample quantile q99 q99 0 Price (EUR/MWh) Probability density Actual
17. 17 Continuous Ranked Probability Score is a single metric to evaluate probabilistic forecasts Price (EUR/MWh) x Cumulative probability 0 1 Price (EUR/MWh) Probability density Actual Actual y CRPS https://www.lokad.com/continuous-ranked-probability-score/ https://datumorphism.leima.is/cards/time-series/crps/
18. 18 Continuous Ranked Probability Score is a single metric to evaluate probabilistic forecasts Price (EUR/MWh) Probability density Actual Point forecast Cumulative probability 0 1 💡CRPS = MAE Price (EUR/MWh) x Actual y https://www.lokad.com/continuous-ranked-probability-score/ https://datumorphism.leima.is/cards/time-series/crps/
19. 19 Three methods to obtain conditional uncertainty estimates Quantile loss Quantile forest Quantile binning
20. 20 Quantile loss: asymmetrically weight errors during model training Error Weight q50 (median) Underforecast Overforecast ɑ-1 ɑ q20 q20 q80 q80 Probability density function ▶ lightgbm.LGBMRegressor( objective=‘quantile’, alpha=0.2)
21. 21 Quantile forest: aggregate ensemble predictions Training data Bootstrap sample … Decision trees ▶ sklearn.ensemble.RandomForestRegressor( max_depth=None, min_samples_leaf=1) Meinshausen (2006) “Quantile regression forests.” J Mach Learn Res 7:983–999
22. 22 Quantile forest: aggregate ensemble predictions Test data Decision trees ŷ 1 ŷ 2 ŷ n Probability density function Aggregate Meinshausen (2006) “Quantile regression forests.” J Mach Learn Res 7:983–999
23. 23 Quantile binning: reduce quantile regression problem to a classification problem 1. Bin continuous target into intervals 2. Train a multiclass classifier 3. Predict and combine Features Target y y ∈ [0, 10) y ∈ [10, 20) y ∈ [20, 30) X 1 23 0 1 0 X 2 8 1 0 0 … … … … … X n+1 ? 0.04 0.87 0.09 Probability density function ▶ sklearn.preprocessing.KBinsDiscretizer ▶ sklearn.multiclass.OneVsRestClassifier
24. 24 Conformal prediction can be used to obtain calibrated uncertainty estimates 3. Add correction value to original quantile prediction to ensure coverage ŷ q40 = f q40 ( X ) + c Testing 1. Train quantile regression model Training f q40 ( X ) 2. Predict on calibration set and extract correction value from error distribution q40 Sample quantile Calibration Error c ▶ mapie.time_series_regression. MapieTimeSeriesRegressor More on this topic: https://github.com/valeman/awesome-conformal-prediction Chen Xu and Yao Xie (2021) “Conformal Prediction Interval for Dynamic Time-Series.” ICML
25. 25 Combining probabilistic price with power forecasts Power forecast Price forecast Optimization Recommended trade P power ( y | x ) P price ( y | x ) Monte Carlo simulation cost = 𝚫power · 𝚫price More information and different approaches see: Pinson (2023) “Distributionally robust trading strategies for renewable energy producers.” IEEE Transactions on Energy Markets, Policy and Regulation 1(1), pp. 37-47
26. Key takeaways 26 Existing machine learning models can be extended to estimate quantiles Probabilistic forecasting can help accelerate the energy transition Probabilistic forecasting adds value by enabling risk-based trading strategies
27. 27 Thank you www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash