Slide 1

Slide 1 text

1 Leveraging conformal prediction for calibrated probabilistic time series forecasts to accelerate the renewable energy transition Inge van den Ende Data Scientist www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash Eindhoven 30 November 2023

Slide 2

Slide 2 text

2 A balancing act on the energy grid: Supply needs to equal demand at any moment supply t demand t =

Slide 3

Slide 3 text

3 Uncertainty in energy generation forecast increases

Slide 4

Slide 4 text

4 Forecasting the uncertainty explicitly enables decision making supply demand price

Slide 5

Slide 5 text

5 Point forecasts don’t give any information about this uncertainty t 0 Time y Actuals Forecast Point forecast ?Uncertainty?

Slide 6

Slide 6 text

6 A prediction interval gives us more information about the uncertainty t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y

Slide 7

Slide 7 text

7 Conformal prediction can create a prediction interval for any point forecast t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y Conformal prediction

Slide 8

Slide 8 text

8 A calibration set is hold out from the train set Train set Calibration set Test set

Slide 9

Slide 9 text

Three steps to forecast with prediction interval Training Calibration Prediction

Slide 10

Slide 10 text

Train the point forecast model on the train set as usual Training Calibration Prediction 1. Train point forecast model on the training set ŷ = f ( X )

Slide 11

Slide 11 text

11 The calibration set is used to compute the prediction interval Training Calibration Prediction 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value Create a sorted list with absolute conformity scores Conformity scores Number of scores Select the quantile of the prediction interval quantile Conformity scores Number of scores

Slide 12

Slide 12 text

12 The calibration set is used to compute the prediction interval Training Calibration Prediction Create a sorted list with absolute conformity scores |e| 1 |e| 2 … |e| N-1 |e| N |e| 1 |e| 2 … |e| N-1 |e| N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 e 2 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value e 3 e 1

Slide 13

Slide 13 text

13 Add the prediction interval to every prediction Training Calibration Prediction 3. Predict with point forecast model y Prediction set Point forecast y Prediction set Point forecast = y Point forecast + selected conformity score = y + |e| 2 Add prediction interval based on conformity scores Point forecast - selected conformity score = y - |e| 2

Slide 14

Slide 14 text

14 Residuals of a calibration set determine the prediction interval Training Calibration Prediction 1. Train point forecast model ŷ = f ( X ) 2. Predict on calibration set and get conformity scores 3. Predict with point forecast model & add prediction interval based on conformity scores y Calibration set Point forecast Actual value y Prediction set Point forecast Prediction interval

Slide 15

Slide 15 text

15 Python packages for conformal prediction MAPIE: Model Agnostic Prediction Interval Estimator Crepes: Conformal classifiers, regressors and predictive systems

Slide 16

Slide 16 text

16 Forecasting with prediction interval with the crepes package ▶ crepes_model = WrapRegressor(baseline_model) ▶ crepes_model.fit(X_prop_train, y_prop_train) ▶ crepes_model.calibrate(X_cal, y_cal) ▶ crepes_point_prediction = crepes_model.predict(X_test) ▶ crepes_prediction_cp = crepes_model.predict_int(X_test, confidence=0.90)

Slide 17

Slide 17 text

Advantages 17 This simple method has great advantages, but also some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution

Slide 18

Slide 18 text

The prediction interval is constant over the prediction set t 0 Time Actuals Forecast 90% interval Do we expect the same uncertainty for these points?

Slide 19

Slide 19 text

19 A prediction interval provides less information then a probabilistic distribution t 0 Time Price (EUR/MWh) Actuals Forecast q95 Probability density Probability density q05 q95 q05 90% interval

Slide 20

Slide 20 text

Advantages 20 This simple method has great advantages, but also some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution Solution will be given in next slides

Slide 21

Slide 21 text

21 Calibrating a probabilistic forecast creates a well-calibrated full distribution that is specific over samples Part 1: Create prediction interval Part 2: Calibrate probabilistic forecast

Slide 22

Slide 22 text

We can use the same three steps to calibrate a probabilistic forecast Training Calibration Prediction

Slide 23

Slide 23 text

Train a probabilistic forecast model on the train set Training Calibration Prediction 1. Train a probabilistic forecast model on the training set [ŷ q01 …ŷ q01 ]= f ( X ) t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 y Probability density slice For example: conformalized quantile regression

Slide 24

Slide 24 text

Quantile regression: fit a model per quantile that you predict Intermezzo: probabilistic forecasting t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 slice q80 q20 Probability density function

Slide 25

Slide 25 text

25 Quantile regression: asymmetrically weight errors during model training Error Weight q50 (median) Underforecast Overforecast ɑ-1 ɑ q20 q80 q80 q20 Probability density function ▶ lightgbm.LGBMRegressor( objective=‘quantile’, alpha=0.2) Intermezzo: probabilistic forecasting

Slide 26

Slide 26 text

Why do we need conformalized quantile regression? Intermezzo: probabilistic forecasting Quantile regression Conformal prediction Asymptotically consistent Statistical guarantee of valid coverage Takes into account local variability of the input space Basic application does not adapt to input space Conformalized quantile regression Statistical guarantee of valid coverage Takes into account local variability of the input space 26

Slide 27

Slide 27 text

27 The calibration set is used to compute a correction factor Training Calibration Prediction Per interval of the probabilistic prediction 2. Predict on calibration set and get conformity scores y Calibration set Mean forecast Actual value 70% forecast 30% forecast 27 Create a sorted list with the conformity scores = residuals e 1 e 2 … e N-1 e N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 Correction can be positive or negative: Positive = wider distribution Negative = more narrow e 1 e 2 … e N-1 e N

Slide 28

Slide 28 text

28 Calibrate every prediction interval Training Calibration Prediction 3. Predict with probabilistic forecast model Calibrate distribution based on conformity scores Calibrated probabilistic forecast Probabilistic forecast Per interval of the probabilistic prediction

Slide 29

Slide 29 text

29 Residuals of a calibration set are used to calibrate the forecasted distribution Training Calibration Prediction 1. Train probabilistic forecast model [ŷ q01 …ŷ q01 ]= f ( X ) 2. Predict on calibration set and get conformity scores for every quantile 3. Predict on test set and calibrate that distribution y Calibration set y Prediction set

Slide 30

Slide 30 text

Advantages 30 A remaining disadvantage: exchangeability Model agnostic: Any model can be used Advantages Statistical guarantee: valid coverage No distribution assumption needed Varies over the prediction set A distribution provides more information then a single prediction interval Assumption: exchangeability Disadvantage

Slide 31

Slide 31 text

31 Exchangeability does not always hold ▶ mapie.time_series_regression.MapieTimeSeriesRegressor t 0 Time y

Slide 32

Slide 32 text

Key takeaways about conformal prediction 32 Conditional when calibrating probabilistic forecast Helps to accelerate the renewable energy transition Simple method with statistical guarantee

Slide 33

Slide 33 text

At the start of 2022 the interest in conformal prediction started to rise Google trend worldwide show increase from start of 2022

Slide 34

Slide 34 text

On the awesome-conformal-prediction github you can find more information Started in 2022 QR code to awesome-conformal-prediction github

Slide 35

Slide 35 text

35 Thank you www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash Meet-up in January