Inge van den Ende
December 04, 2023
340

Leveraging conformal prediction for calibrated probabilistic time series forecasts to accelerate the renewable energy transition

Presentation given at Pydata Eindhoven at 30th of November 2023

Inge van den Ende

December 04, 2023

Transcript

1. 1 Leveraging conformal prediction for calibrated probabilistic time series forecasts

to accelerate the renewable energy transition Inge van den Ende Data Scientist www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash Eindhoven 30 November 2023
2. 2 A balancing act on the energy grid: Supply needs

to equal demand at any moment supply t demand t =

4. 4 Forecasting the uncertainty explicitly enables decision making supply demand

price

t 0 Time y Actuals Forecast Point forecast ?Uncertainty?

uncertainty t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y
7. 7 Conformal prediction can create a prediction interval for any

point forecast t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y Conformal prediction
8. 8 A calibration set is hold out from the train

set Train set Calibration set Test set

10. Train the point forecast model on the train set as

usual Training Calibration Prediction 1. Train point forecast model on the training set ŷ = f ( X )
11. 11 The calibration set is used to compute the prediction

interval Training Calibration Prediction 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value Create a sorted list with absolute conformity scores Conformity scores Number of scores Select the quantile of the prediction interval quantile Conformity scores Number of scores
12. 12 The calibration set is used to compute the prediction

interval Training Calibration Prediction Create a sorted list with absolute conformity scores |e| 1 |e| 2 … |e| N-1 |e| N |e| 1 |e| 2 … |e| N-1 |e| N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 e 2 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value e 3 e 1
13. 13 Add the prediction interval to every prediction Training Calibration

Prediction 3. Predict with point forecast model y Prediction set Point forecast y Prediction set Point forecast = y Point forecast + selected conformity score = y + |e| 2 Add prediction interval based on conformity scores Point forecast - selected conformity score = y - |e| 2
14. 14 Residuals of a calibration set determine the prediction interval

Training Calibration Prediction 1. Train point forecast model ŷ = f ( X ) 2. Predict on calibration set and get conformity scores 3. Predict with point forecast model & add prediction interval based on conformity scores y Calibration set Point forecast Actual value y Prediction set Point forecast Prediction interval
15. 15 Python packages for conformal prediction MAPIE: Model Agnostic Prediction

Interval Estimator Crepes: Conformal classifiers, regressors and predictive systems
16. 16 Forecasting with prediction interval with the crepes package ▶

crepes_model = WrapRegressor(baseline_model) ▶ crepes_model.fit(X_prop_train, y_prop_train) ▶ crepes_model.calibrate(X_cal, y_cal) ▶ crepes_point_prediction = crepes_model.predict(X_test) ▶ crepes_prediction_cp = crepes_model.predict_int(X_test, confidence=0.90)

some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution
18. The prediction interval is constant over the prediction set t

0 Time Actuals Forecast 90% interval Do we expect the same uncertainty for these points?
19. 19 A prediction interval provides less information then a probabilistic

distribution t 0 Time Price (EUR/MWh) Actuals Forecast q95 Probability density Probability density q05 q95 q05 90% interval

some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution Solution will be given in next slides
21. 21 Calibrating a probabilistic forecast creates a well-calibrated full distribution

that is specific over samples Part 1: Create prediction interval Part 2: Calibrate probabilistic forecast
22. We can use the same three steps to calibrate a

probabilistic forecast Training Calibration Prediction
23. Train a probabilistic forecast model on the train set Training

Calibration Prediction 1. Train a probabilistic forecast model on the training set [ŷ q01 …ŷ q01 ]= f ( X ) t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 y Probability density slice For example: conformalized quantile regression
24. Quantile regression: fit a model per quantile that you predict

Intermezzo: probabilistic forecasting t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 slice q80 q20 Probability density function
25. 25 Quantile regression: asymmetrically weight errors during model training Error

Weight q50 (median) Underforecast Overforecast ɑ-1 ɑ q20 q80 q80 q20 Probability density function ▶ lightgbm.LGBMRegressor( objective=‘quantile’, alpha=0.2) Intermezzo: probabilistic forecasting
26. Why do we need conformalized quantile regression? Intermezzo: probabilistic forecasting

Quantile regression Conformal prediction Asymptotically consistent Statistical guarantee of valid coverage Takes into account local variability of the input space Basic application does not adapt to input space Conformalized quantile regression Statistical guarantee of valid coverage Takes into account local variability of the input space 26
27. 27 The calibration set is used to compute a correction

factor Training Calibration Prediction Per interval of the probabilistic prediction 2. Predict on calibration set and get conformity scores y Calibration set Mean forecast Actual value 70% forecast 30% forecast 27 Create a sorted list with the conformity scores = residuals e 1 e 2 … e N-1 e N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 Correction can be positive or negative: Positive = wider distribution Negative = more narrow e 1 e 2 … e N-1 e N
28. 28 Calibrate every prediction interval Training Calibration Prediction 3. Predict

with probabilistic forecast model Calibrate distribution based on conformity scores Calibrated probabilistic forecast Probabilistic forecast Per interval of the probabilistic prediction
29. 29 Residuals of a calibration set are used to calibrate

the forecasted distribution Training Calibration Prediction 1. Train probabilistic forecast model [ŷ q01 …ŷ q01 ]= f ( X ) 2. Predict on calibration set and get conformity scores for every quantile 3. Predict on test set and calibrate that distribution y Calibration set y Prediction set

can be used Advantages Statistical guarantee: valid coverage No distribution assumption needed Varies over the prediction set A distribution provides more information then a single prediction interval Assumption: exchangeability Disadvantage

Time y
32. Key takeaways about conformal prediction 32 Conditional when calibrating probabilistic

forecast Helps to accelerate the renewable energy transition Simple method with statistical guarantee
33. At the start of 2022 the interest in conformal prediction

started to rise Google trend worldwide show increase from start of 2022