Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging conformal prediction for calibrated ...

Leveraging conformal prediction for calibrated probabilistic time series forecasts to accelerate the renewable energy transition

Presentation given at Pydata Eindhoven at 30th of November 2023

Inge van den Ende

December 04, 2023
Tweet

Other Decks in Science

Transcript

  1. 1 Leveraging conformal prediction for calibrated probabilistic time series forecasts

    to accelerate the renewable energy transition Inge van den Ende Data Scientist www.dexterenergy.ai Photo by Nicholas Doherty on Unsplash Eindhoven 30 November 2023
  2. 2 A balancing act on the energy grid: Supply needs

    to equal demand at any moment supply t demand t =
  3. 5 Point forecasts don’t give any information about this uncertainty

    t 0 Time y Actuals Forecast Point forecast ?Uncertainty?
  4. 6 A prediction interval gives us more information about the

    uncertainty t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y
  5. 7 Conformal prediction can create a prediction interval for any

    point forecast t 0 Time Actuals Forecast Point forecast t 0 Time Actuals Forecast 90% interval y y Conformal prediction
  6. 8 A calibration set is hold out from the train

    set Train set Calibration set Test set
  7. Train the point forecast model on the train set as

    usual Training Calibration Prediction 1. Train point forecast model on the training set ŷ = f ( X )
  8. 11 The calibration set is used to compute the prediction

    interval Training Calibration Prediction 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value Create a sorted list with absolute conformity scores Conformity scores Number of scores Select the quantile of the prediction interval quantile Conformity scores Number of scores
  9. 12 The calibration set is used to compute the prediction

    interval Training Calibration Prediction Create a sorted list with absolute conformity scores |e| 1 |e| 2 … |e| N-1 |e| N |e| 1 |e| 2 … |e| N-1 |e| N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 e 2 2. Predict on calibration set and get conformity scores y Calibration set Point forecast Actual value e 3 e 1
  10. 13 Add the prediction interval to every prediction Training Calibration

    Prediction 3. Predict with point forecast model y Prediction set Point forecast y Prediction set Point forecast = y Point forecast + selected conformity score = y + |e| 2 Add prediction interval based on conformity scores Point forecast - selected conformity score = y - |e| 2
  11. 14 Residuals of a calibration set determine the prediction interval

    Training Calibration Prediction 1. Train point forecast model ŷ = f ( X ) 2. Predict on calibration set and get conformity scores 3. Predict with point forecast model & add prediction interval based on conformity scores y Calibration set Point forecast Actual value y Prediction set Point forecast Prediction interval
  12. 15 Python packages for conformal prediction MAPIE: Model Agnostic Prediction

    Interval Estimator Crepes: Conformal classifiers, regressors and predictive systems
  13. 16 Forecasting with prediction interval with the crepes package ▶

    crepes_model = WrapRegressor(baseline_model) ▶ crepes_model.fit(X_prop_train, y_prop_train) ▶ crepes_model.calibrate(X_cal, y_cal) ▶ crepes_point_prediction = crepes_model.predict(X_test) ▶ crepes_prediction_cp = crepes_model.predict_int(X_test, confidence=0.90)
  14. Advantages 17 This simple method has great advantages, but also

    some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution
  15. The prediction interval is constant over the prediction set t

    0 Time Actuals Forecast 90% interval Do we expect the same uncertainty for these points?
  16. 19 A prediction interval provides less information then a probabilistic

    distribution t 0 Time Price (EUR/MWh) Actuals Forecast q95 Probability density Probability density q05 q95 q05 90% interval
  17. Advantages 20 This simple method has great advantages, but also

    some disadvantages Model agnostic: Any model can be used Disadvantages Statistical guarantee: valid coverage No distribution assumption needed Constant over the prediction set A single prediction interval provides less information then a distribution Solution will be given in next slides
  18. 21 Calibrating a probabilistic forecast creates a well-calibrated full distribution

    that is specific over samples Part 1: Create prediction interval Part 2: Calibrate probabilistic forecast
  19. We can use the same three steps to calibrate a

    probabilistic forecast Training Calibration Prediction
  20. Train a probabilistic forecast model on the train set Training

    Calibration Prediction 1. Train a probabilistic forecast model on the training set [ŷ q01 …ŷ q01 ]= f ( X ) t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 y Probability density slice For example: conformalized quantile regression
  21. Quantile regression: fit a model per quantile that you predict

    Intermezzo: probabilistic forecasting t -1 Time y Actuals Forecast Point forecast q80 q60 q40 q20 slice q80 q20 Probability density function
  22. 25 Quantile regression: asymmetrically weight errors during model training Error

    Weight q50 (median) Underforecast Overforecast ɑ-1 ɑ q20 q80 q80 q20 Probability density function ▶ lightgbm.LGBMRegressor( objective=‘quantile’, alpha=0.2) Intermezzo: probabilistic forecasting
  23. Why do we need conformalized quantile regression? Intermezzo: probabilistic forecasting

    Quantile regression Conformal prediction Asymptotically consistent Statistical guarantee of valid coverage Takes into account local variability of the input space Basic application does not adapt to input space Conformalized quantile regression Statistical guarantee of valid coverage Takes into account local variability of the input space 26
  24. 27 The calibration set is used to compute a correction

    factor Training Calibration Prediction Per interval of the probabilistic prediction 2. Predict on calibration set and get conformity scores y Calibration set Mean forecast Actual value 70% forecast 30% forecast 27 Create a sorted list with the conformity scores = residuals e 1 e 2 … e N-1 e N Select the quantile of the prediction interval index = ((1 - ⍺) * n) - 1 Correction can be positive or negative: Positive = wider distribution Negative = more narrow e 1 e 2 … e N-1 e N
  25. 28 Calibrate every prediction interval Training Calibration Prediction 3. Predict

    with probabilistic forecast model Calibrate distribution based on conformity scores Calibrated probabilistic forecast Probabilistic forecast Per interval of the probabilistic prediction
  26. 29 Residuals of a calibration set are used to calibrate

    the forecasted distribution Training Calibration Prediction 1. Train probabilistic forecast model [ŷ q01 …ŷ q01 ]= f ( X ) 2. Predict on calibration set and get conformity scores for every quantile 3. Predict on test set and calibrate that distribution y Calibration set y Prediction set
  27. Advantages 30 A remaining disadvantage: exchangeability Model agnostic: Any model

    can be used Advantages Statistical guarantee: valid coverage No distribution assumption needed Varies over the prediction set A distribution provides more information then a single prediction interval Assumption: exchangeability Disadvantage
  28. Key takeaways about conformal prediction 32 Conditional when calibrating probabilistic

    forecast Helps to accelerate the renewable energy transition Simple method with statistical guarantee
  29. At the start of 2022 the interest in conformal prediction

    started to rise Google trend worldwide show increase from start of 2022
  30. On the awesome-conformal-prediction github you can find more information Started

    in 2022 QR code to awesome-conformal-prediction github