Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High dimensional time series analysis

Rob J Hyndman
October 31, 2017

High dimensional time series analysis

Keynote presentation for ACEMS retreat

Rob J Hyndman

October 31, 2017
Tweet

More Decks by Rob J Hyndman

Other Decks in Research

Transcript

  1. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 2
  2. Sub-daily me series analysis How to visualize many series of

    sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? 4
  3. Sub-daily me series analysis How to visualize many series of

    sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? 4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec M T W T F S S M T W T F S S M T W T F S S M T W T F S S
  4. Sub-daily me series analysis How to visualize many series of

    sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? Di Cook Earo Wang Mitchell O’Hara-Wild 5
  5. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 6
  6. Time series feature analysis Can we use me series features

    for fast iden fica on of forecas ng models? How to generate new me series with specified feature vectors? What can we say about the feature space of me series? 8
  7. Time series feature analysis Can we use me series features

    for fast iden fica on of forecas ng models? How to generate new me series with specified feature vectors? What can we say about the feature space of me series? Kate Smith-Miles George Athanasopoulos Thiyanga Talagala 8
  8. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 9
  9. Time series anomaly detec on How to iden fy anomalous

    behaviour within streaming data? How to define an anomaly in a large mul variate data set? 12
  10. Time series anomaly detec on How to iden fy anomalous

    behaviour within streaming data? How to define an anomaly in a large mul variate data set? Kate Smith-Miles Mario Andr´ es Mu˜ noz Acosta Sevvandi Kandanaarachchi Dilini Talagala 12
  11. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 13
  12. Electricity demand 15 Monday Tuesday Wednesday Thursday Friday Saturday Sunday

    1539 1549 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 0 2 4 6 Time of day Demand (kWh) Percentile 10 25 50 75 90
  13. Electricity demand How to forecast future demand by household? How

    to reconcile household demand forecasts with state and na onal demand forecasts? How to iden fy unusual demand pa erns? How to measure forecast accuracy when forecasts are probability distribu ons within a hierarchy? 16
  14. Electricity demand How to forecast future demand by household? How

    to reconcile household demand forecasts with state and na onal demand forecasts? How to iden fy unusual demand pa erns? How to measure forecast accuracy when forecasts are probability distribu ons within a hierarchy? Souhaib Ben Taieb Cameron Roach 16
  15. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 17
  16. Forecast reconcilia on Huawei sales by division, group, sub-group, etc.

    Australian tourism demand by state, region, zone. 18 Total A AA AAA AAB AAC AB ABA ABB ABC AC ACA ACB ACC B BA BAA BAB BAC BB BBA BBB BBC BC BCA BCB BCC C CA CAA CAB CAC CB CBA CBB CBC CC CCA CCB CCC
  17. Forecast reconcilia on Forecasts at all nodes must be coherent

    Bo om level typically has thousands or millions of me series How to define coherence probabilis cally? How to visualize so many me series? 19 George Athanasopoulos Anastasios Panagiotelis Shanika Wickramasuriya Puwasala Gamakumara Earo Wang
  18. Australian tourism demand 20 Quarterly data on visitor night from

    1998:Q1 – 2013:Q4 From Na onal Visitor Survey, based on annual interviews of 120,000 Australians aged 15+, collected by Tourism Research Australia. Split by 7 states, 27 zones and 76 regions (a geographical hierarchy) Also split by purpose of travel Holiday Visi ng friends and rela ves (VFR) Business Other 304 bo om-level series
  19. Hierarchical me series Total A B C 21 yt :

    observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.
  20. Hierarchical me series Total A B C yt = [yt

    , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1        yA,t yB,t yC,t   21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.
  21. Hierarchical me series Total A B C yt = [yt

    , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.
  22. Hierarchical me series Total A B C yt = [yt

    , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   bt 21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.
  23. Hierarchical me series Total A B C yt = [yt

    , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   bt yt = Sbt 21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.
  24. Disaggregated me series Every collec on of me series with

    aggrega on constraints can be wri en as yt = Sbt where yt is a vector of all series at me t bt is a vector of the most disaggregated series at me t S is a “summing matrix” containing the aggrega on constraints. 22
  25. Forecas ng nota on Let ˆ yn (h) be vector

    of ini al h-step forecasts, made at me n, stacked in same order as yt. 23
  26. Forecas ng nota on Let ˆ yn (h) be vector

    of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) 23
  27. Forecas ng nota on Let ˆ yn (h) be vector

    of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. 23
  28. Forecas ng nota on Let ˆ yn (h) be vector

    of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. P projects base forecasts ˆ yn (h) to bo om level. 23
  29. Forecas ng nota on Let ˆ yn (h) be vector

    of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. P projects base forecasts ˆ yn (h) to bo om level. S adds them up 23
  30. General proper es: bias and variance ˜ yn (h) =

    SPˆ yn (h) Bias Reconciled forecasts are unbiased iff SPS = S. 24
  31. General proper es: bias and variance ˜ yn (h) =

    SPˆ yn (h) Bias Reconciled forecasts are unbiased iff SPS = S. Variance Let error variance of h-step base forecasts ˆ yn (h) be Wh = Var[yn+h − ˆ yn (h) | y1 , . . . , yn ] Then error variance of the reconciled forecasts is Var[yn+h − ˜ yn (h) | y1 , . . . , yn ] = SPWh P S 24
  32. Op mal forecast reconcilia on ˜ yn (h) = SPˆ

    yn (h) Theorem: MinT Reconcilia on If P sa sfies SPS = S, then minP = trace[SPWh P S ] has solu on P = (S W−1 h S)−1S W−1 h . 25
  33. Op mal forecast reconcilia on ˜ yn (h) = SPˆ

    yn (h) Theorem: MinT Reconcilia on If P sa sfies SPS = S, then minP = trace[SPWh P S ] has solu on P = (S W−1 h S)−1S W−1 h . Reconciled forecasts Base forecasts 25 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)
  34. Op mal forecast reconcilia on Reconciled forecasts Base forecasts 26

    ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)
  35. Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume

    that Wh = kh W1 to simplify computa ons. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)
  36. Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume

    that Wh = kh W1 to simplify computa ons. WLS solu on Approximate W1 by its diagonal. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)
  37. Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume

    that Wh = kh W1 to simplify computa ons. WLS solu on Approximate W1 by its diagonal. GLS solu on Es mate W1 using shrinkage to the diagonal. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)
  38. Australian tourism 27 Hierarchy: States (7) Zones (27) Regions (82)

    Base forecasts ETS (exponen al smoothing) models
  39. Base forecasts Domestic tourism forecasts: Total Year Visitor nights 1998

    2000 2002 2004 2006 2008 60000 65000 70000 75000 80000 85000 28
  40. Base forecasts Domestic tourism forecasts: NSW Year Visitor nights 1998

    2000 2002 2004 2006 2008 18000 22000 26000 30000 28
  41. Base forecasts Domestic tourism forecasts: VIC Year Visitor nights 1998

    2000 2002 2004 2006 2008 10000 12000 14000 16000 18000 28
  42. Base forecasts Domestic tourism forecasts: Nth.Coast.NSW Year Visitor nights 1998

    2000 2002 2004 2006 2008 5000 6000 7000 8000 9000 28
  43. Base forecasts Domestic tourism forecasts: Metro.QLD Year Visitor nights 1998

    2000 2002 2004 2006 2008 8000 9000 11000 13000 28
  44. Base forecasts Domestic tourism forecasts: Sth.WA Year Visitor nights 1998

    2000 2002 2004 2006 2008 400 600 800 1000 1200 1400 28
  45. Base forecasts Domestic tourism forecasts: X201.Melbourne Year Visitor nights 1998

    2000 2002 2004 2006 2008 4000 4500 5000 5500 6000 28
  46. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q time
  47. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  48. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  49. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  50. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  51. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  52. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  53. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  54. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  55. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  56. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  57. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  58. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  59. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  60. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  61. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  62. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  63. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  64. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  65. Forecast evalua on 29 Training sets Test sets h =

    1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  66. Forecast evalua on 29 Training sets Test sets h =

    2 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  67. Forecast evalua on 29 Training sets Test sets h =

    3 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  68. Forecast evalua on 29 Training sets Test sets h =

    4 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  69. Forecast evalua on 29 Training sets Test sets h =

    5 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  70. Forecast evalua on 29 Training sets Test sets h =

    6 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time
  71. Hierarchy: states, zones, regions Forecast horizon RMSE h = 1

    h = 2 h = 3 h = 4 h = 5 h = 6 Ave Australia Base 1762.04 1770.29 1766.02 1818.82 1705.35 1721.17 1757.28 Bo om 1736.92 1742.69 1722.79 1752.74 1666.73 1687.43 1718.22 WLS 1705.21 1715.87 1703.75 1729.56 1627.79 1661.24 1690.57 GLS 1704.64 1715.60 1705.31 1729.04 1626.36 1661.64 1690.43 States Base 399.77 404.16 401.92 407.26 395.38 401.17 401.61 Bo om 404.29 406.95 404.96 409.02 399.80 401.55 404.43 WLS 398.84 402.12 400.71 405.03 394.76 398.23 399.95 GLS 398.84 402.16 400.86 405.03 394.59 398.22 399.95 Regions Base 93.15 93.38 93.45 93.79 93.50 93.56 93.47 Bo om 93.15 93.38 93.45 93.79 93.50 93.56 93.47 WLS 93.02 93.32 93.38 93.72 93.39 93.53 93.39 GLS 92.98 93.27 93.34 93.66 93.34 93.46 93.34 30
  72. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 31
  73. Temporal hierarchies Annual Semi-Annual1 Q1 Q2 Semi-Annual2 Q3 Q4 Basic

    idea: ¯ Forecast series at each available frequency. ¯ Op mally reconcile forecasts within the same year. 32
  74. Monthly series Annual Semi-Annual1 Q1 M1 M2 M3 Q2 M4

    M5 M6 Semi-Annual2 Q3 M7 M8 M9 Q4 M10 M11 M12 k = 2, 4, 12 nodes 33
  75. Monthly series Annual FourM1 BiM1 M1 M2 BiM2 M3 M4

    FourM2 BiM3 M5 M6 BiM4 M7 M8 FourM3 BiM5 M9 M10 BiM6 M11 M12 k = 2, 4, 12 nodes k = 3, 6, 12 nodes 33
  76. Monthly series Annual FourM1 BiM1 M1 M2 BiM2 M3 M4

    FourM2 BiM3 M5 M6 BiM4 M7 M8 FourM3 BiM5 M9 M10 BiM6 M11 M12 k = 2, 4, 12 nodes k = 3, 6, 12 nodes Why not k = 2, 3, 4, 6, 12 nodes? 33
  77. Monthly data        

                                      A SemiA1 SemiA2 FourM1 FourM2 FourM3 Q1 . . . Q4 BiM1 . . . BiM6 M1 . . . M12                                           (28×1) =                                           1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 . . . 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 . . . 0 0 0 0 0 0 0 0 0 0 1 1 I12                                           S                                     M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12                                     bt 34
  78. In general For a me series y1 , . .

    . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. 35
  79. In general For a me series y1 , . .

    . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. A single unique hierarchy is only possible when there are no coprime pairs in F(m). 35
  80. In general For a me series y1 , . .

    . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. A single unique hierarchy is only possible when there are no coprime pairs in F(m). Mk = m/k is seasonal period of aggregated series. 35
  81. UK Accidents and Emergency Demand 1 2 3 4 5

    6 5100 5300 5500 Annual (k=52) Forecast 2 4 6 8 10 12 2500 2600 2700 2800 2900 Semi−annual (k=26) Forecast 5 10 15 20 25 1250 1350 1450 Quarterly (k=13) Forecast 20 40 60 80 360 380 400 420 440 460 Monthly (k=4) Forecast 50 100 150 180 190 200 210 220 230 Bi−weekly (k=2) Forecast 50 100 150 200 250 300 90 95 100 105 110 Weekly (k=1) Forecast 36 – – – – base reconciled
  82. Outline 1 Sub-daily me series analysis 2 Time series feature

    analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 37