Slide 1

Slide 1 text

High-dimensional me series analysis Rob J Hyndman 1 November 2017

Slide 2

Slide 2 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 2

Slide 3

Slide 3 text

Pedestrian counts 3

Slide 4

Slide 4 text

Pedestrian counts 3 −37.83 −37.82 −37.81 −37.80 −37.79 144.93 144.94 144.95 144.96 144.97 144.98 Longitude Latitude

Slide 5

Slide 5 text

Sub-daily me series analysis How to visualize many series of sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? 4

Slide 6

Slide 6 text

Sub-daily me series analysis How to visualize many series of sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? 4 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec M T W T F S S M T W T F S S M T W T F S S M T W T F S S

Slide 7

Slide 7 text

Sub-daily me series analysis How to visualize many series of sub-daily data over several years? How to iden fy unusual pa erns/incidents? How to forecast sub-daily data taking account of public holidays and special events? Di Cook Earo Wang Mitchell O’Hara-Wild 5

Slide 8

Slide 8 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 6

Slide 9

Slide 9 text

Walmart weekly sales data 7

Slide 10

Slide 10 text

Time series feature analysis Can we use me series features for fast iden fica on of forecas ng models? How to generate new me series with specified feature vectors? What can we say about the feature space of me series? 8

Slide 11

Slide 11 text

Time series feature analysis Can we use me series features for fast iden fica on of forecas ng models? How to generate new me series with specified feature vectors? What can we say about the feature space of me series? Kate Smith-Miles George Athanasopoulos Thiyanga Talagala 8

Slide 12

Slide 12 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 9

Slide 13

Slide 13 text

Security monitoring 10

Slide 14

Slide 14 text

Security monitoring 11

Slide 15

Slide 15 text

Time series anomaly detec on How to iden fy anomalous behaviour within streaming data? How to define an anomaly in a large mul variate data set? 12

Slide 16

Slide 16 text

Time series anomaly detec on How to iden fy anomalous behaviour within streaming data? How to define an anomaly in a large mul variate data set? Kate Smith-Miles Mario Andr´ es Mu˜ noz Acosta Sevvandi Kandanaarachchi Dilini Talagala 12

Slide 17

Slide 17 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 13

Slide 18

Slide 18 text

Electricity demand <2>{ } 14

Slide 19

Slide 19 text

Electricity demand 15 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 1539 1549 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 0 2 4 6 Time of day Demand (kWh) Percentile 10 25 50 75 90

Slide 20

Slide 20 text

Electricity demand How to forecast future demand by household? How to reconcile household demand forecasts with state and na onal demand forecasts? How to iden fy unusual demand pa erns? How to measure forecast accuracy when forecasts are probability distribu ons within a hierarchy? 16

Slide 21

Slide 21 text

Electricity demand How to forecast future demand by household? How to reconcile household demand forecasts with state and na onal demand forecasts? How to iden fy unusual demand pa erns? How to measure forecast accuracy when forecasts are probability distribu ons within a hierarchy? Souhaib Ben Taieb Cameron Roach 16

Slide 22

Slide 22 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 17

Slide 23

Slide 23 text

Forecast reconcilia on Huawei sales by division, group, sub-group, etc. Australian tourism demand by state, region, zone. 18 Total A AA AAA AAB AAC AB ABA ABB ABC AC ACA ACB ACC B BA BAA BAB BAC BB BBA BBB BBC BC BCA BCB BCC C CA CAA CAB CAC CB CBA CBB CBC CC CCA CCB CCC

Slide 24

Slide 24 text

Forecast reconcilia on Forecasts at all nodes must be coherent Bo om level typically has thousands or millions of me series How to define coherence probabilis cally? How to visualize so many me series? 19 George Athanasopoulos Anastasios Panagiotelis Shanika Wickramasuriya Puwasala Gamakumara Earo Wang

Slide 25

Slide 25 text

Australian tourism demand 20

Slide 26

Slide 26 text

Australian tourism demand 20 Quarterly data on visitor night from 1998:Q1 – 2013:Q4 From Na onal Visitor Survey, based on annual interviews of 120,000 Australians aged 15+, collected by Tourism Research Australia. Split by 7 states, 27 zones and 76 regions (a geographical hierarchy) Also split by purpose of travel Holiday Visi ng friends and rela ves (VFR) Business Other 304 bo om-level series

Slide 27

Slide 27 text

Hierarchical me series Total A B C 21

Slide 28

Slide 28 text

Hierarchical me series Total A B C 21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.

Slide 29

Slide 29 text

Hierarchical me series Total A B C yt = [yt , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1        yA,t yB,t yC,t   21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.

Slide 30

Slide 30 text

Hierarchical me series Total A B C yt = [yt , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.

Slide 31

Slide 31 text

Hierarchical me series Total A B C yt = [yt , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   bt 21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.

Slide 32

Slide 32 text

Hierarchical me series Total A B C yt = [yt , yA,t , yB,t , yC,t ] =      1 1 1 1 0 0 0 1 0 0 0 1      S   yA,t yB,t yC,t   bt yt = Sbt 21 yt : observed aggregate of all series at me t. yX,t : observa on on series X at me t. bt : vector of all series at bo om level in me t.

Slide 33

Slide 33 text

Disaggregated me series Every collec on of me series with aggrega on constraints can be wri en as yt = Sbt where yt is a vector of all series at me t bt is a vector of the most disaggregated series at me t S is a “summing matrix” containing the aggrega on constraints. 22

Slide 34

Slide 34 text

Forecas ng nota on Let ˆ yn (h) be vector of ini al h-step forecasts, made at me n, stacked in same order as yt. 23

Slide 35

Slide 35 text

Forecas ng nota on Let ˆ yn (h) be vector of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) 23

Slide 36

Slide 36 text

Forecas ng nota on Let ˆ yn (h) be vector of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. 23

Slide 37

Slide 37 text

Forecas ng nota on Let ˆ yn (h) be vector of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. P projects base forecasts ˆ yn (h) to bo om level. 23

Slide 38

Slide 38 text

Forecas ng nota on Let ˆ yn (h) be vector of ini al h-step forecasts, made at me n, stacked in same order as yt. (In general, they will not “add up”.) Reconciled forecasts must be of the form: ˜ yn (h) = SPˆ yn (h) for some matrix P. P projects base forecasts ˆ yn (h) to bo om level. S adds them up 23

Slide 39

Slide 39 text

General proper es: bias and variance ˜ yn (h) = SPˆ yn (h) 24

Slide 40

Slide 40 text

General proper es: bias and variance ˜ yn (h) = SPˆ yn (h) Bias Reconciled forecasts are unbiased iff SPS = S. 24

Slide 41

Slide 41 text

General proper es: bias and variance ˜ yn (h) = SPˆ yn (h) Bias Reconciled forecasts are unbiased iff SPS = S. Variance Let error variance of h-step base forecasts ˆ yn (h) be Wh = Var[yn+h − ˆ yn (h) | y1 , . . . , yn ] Then error variance of the reconciled forecasts is Var[yn+h − ˜ yn (h) | y1 , . . . , yn ] = SPWh P S 24

Slide 42

Slide 42 text

Op mal forecast reconcilia on ˜ yn (h) = SPˆ yn (h) 25

Slide 43

Slide 43 text

Op mal forecast reconcilia on ˜ yn (h) = SPˆ yn (h) Theorem: MinT Reconcilia on If P sa sfies SPS = S, then minP = trace[SPWh P S ] has solu on P = (S W−1 h S)−1S W−1 h . 25

Slide 44

Slide 44 text

Op mal forecast reconcilia on ˜ yn (h) = SPˆ yn (h) Theorem: MinT Reconcilia on If P sa sfies SPS = S, then minP = trace[SPWh P S ] has solu on P = (S W−1 h S)−1S W−1 h . Reconciled forecasts Base forecasts 25 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)

Slide 45

Slide 45 text

Op mal forecast reconcilia on Reconciled forecasts Base forecasts 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)

Slide 46

Slide 46 text

Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume that Wh = kh W1 to simplify computa ons. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)

Slide 47

Slide 47 text

Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume that Wh = kh W1 to simplify computa ons. WLS solu on Approximate W1 by its diagonal. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)

Slide 48

Slide 48 text

Op mal forecast reconcilia on Reconciled forecasts Base forecasts Assume that Wh = kh W1 to simplify computa ons. WLS solu on Approximate W1 by its diagonal. GLS solu on Es mate W1 using shrinkage to the diagonal. 26 ˜ yn (h) = S(S W−1 h S)−1S W−1 h ˆ yn (h)

Slide 49

Slide 49 text

Australian tourism 27

Slide 50

Slide 50 text

Australian tourism 27 Hierarchy: States (7) Zones (27) Regions (82)

Slide 51

Slide 51 text

Australian tourism 27 Hierarchy: States (7) Zones (27) Regions (82) Base forecasts ETS (exponen al smoothing) models

Slide 52

Slide 52 text

Base forecasts Domestic tourism forecasts: Total Year Visitor nights 1998 2000 2002 2004 2006 2008 60000 65000 70000 75000 80000 85000 28

Slide 53

Slide 53 text

Base forecasts Domestic tourism forecasts: NSW Year Visitor nights 1998 2000 2002 2004 2006 2008 18000 22000 26000 30000 28

Slide 54

Slide 54 text

Base forecasts Domestic tourism forecasts: VIC Year Visitor nights 1998 2000 2002 2004 2006 2008 10000 12000 14000 16000 18000 28

Slide 55

Slide 55 text

Base forecasts Domestic tourism forecasts: Nth.Coast.NSW Year Visitor nights 1998 2000 2002 2004 2006 2008 5000 6000 7000 8000 9000 28

Slide 56

Slide 56 text

Base forecasts Domestic tourism forecasts: Metro.QLD Year Visitor nights 1998 2000 2002 2004 2006 2008 8000 9000 11000 13000 28

Slide 57

Slide 57 text

Base forecasts Domestic tourism forecasts: Sth.WA Year Visitor nights 1998 2000 2002 2004 2006 2008 400 600 800 1000 1200 1400 28

Slide 58

Slide 58 text

Base forecasts Domestic tourism forecasts: X201.Melbourne Year Visitor nights 1998 2000 2002 2004 2006 2008 4000 4500 5000 5500 6000 28

Slide 59

Slide 59 text

Base forecasts Domestic tourism forecasts: X402.Murraylands Year Visitor nights 1998 2000 2002 2004 2006 2008 0 100 200 300 28

Slide 60

Slide 60 text

Base forecasts Domestic tourism forecasts: X809.Daly Year Visitor nights 1998 2000 2002 2004 2006 2008 0 20 40 60 80 100 28

Slide 61

Slide 61 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 62

Slide 62 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 63

Slide 63 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 64

Slide 64 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 65

Slide 65 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 66

Slide 66 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 67

Slide 67 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 68

Slide 68 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 69

Slide 69 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 70

Slide 70 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 71

Slide 71 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 72

Slide 72 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 73

Slide 73 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 74

Slide 74 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 75

Slide 75 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 76

Slide 76 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 77

Slide 77 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 78

Slide 78 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 79

Slide 79 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 80

Slide 80 text

Forecast evalua on 29 Training sets Test sets h = 1 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 81

Slide 81 text

Forecast evalua on 29 Training sets Test sets h = 2 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 82

Slide 82 text

Forecast evalua on 29 Training sets Test sets h = 3 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 83

Slide 83 text

Forecast evalua on 29 Training sets Test sets h = 4 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 84

Slide 84 text

Forecast evalua on 29 Training sets Test sets h = 5 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 85

Slide 85 text

Forecast evalua on 29 Training sets Test sets h = 6 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q time

Slide 86

Slide 86 text

Hierarchy: states, zones, regions Forecast horizon RMSE h = 1 h = 2 h = 3 h = 4 h = 5 h = 6 Ave Australia Base 1762.04 1770.29 1766.02 1818.82 1705.35 1721.17 1757.28 Bo om 1736.92 1742.69 1722.79 1752.74 1666.73 1687.43 1718.22 WLS 1705.21 1715.87 1703.75 1729.56 1627.79 1661.24 1690.57 GLS 1704.64 1715.60 1705.31 1729.04 1626.36 1661.64 1690.43 States Base 399.77 404.16 401.92 407.26 395.38 401.17 401.61 Bo om 404.29 406.95 404.96 409.02 399.80 401.55 404.43 WLS 398.84 402.12 400.71 405.03 394.76 398.23 399.95 GLS 398.84 402.16 400.86 405.03 394.59 398.22 399.95 Regions Base 93.15 93.38 93.45 93.79 93.50 93.56 93.47 Bo om 93.15 93.38 93.45 93.79 93.50 93.56 93.47 WLS 93.02 93.32 93.38 93.72 93.39 93.53 93.39 GLS 92.98 93.27 93.34 93.66 93.34 93.46 93.34 30

Slide 87

Slide 87 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 31

Slide 88

Slide 88 text

Temporal hierarchies Annual Semi-Annual1 Q1 Q2 Semi-Annual2 Q3 Q4 32

Slide 89

Slide 89 text

Temporal hierarchies Annual Semi-Annual1 Q1 Q2 Semi-Annual2 Q3 Q4 Basic idea: ¯ Forecast series at each available frequency. ¯ Op mally reconcile forecasts within the same year. 32

Slide 90

Slide 90 text

Monthly series Annual Semi-Annual1 Q1 M1 M2 M3 Q2 M4 M5 M6 Semi-Annual2 Q3 M7 M8 M9 Q4 M10 M11 M12 k = 2, 4, 12 nodes 33

Slide 91

Slide 91 text

Monthly series Annual FourM1 BiM1 M1 M2 BiM2 M3 M4 FourM2 BiM3 M5 M6 BiM4 M7 M8 FourM3 BiM5 M9 M10 BiM6 M11 M12 k = 2, 4, 12 nodes k = 3, 6, 12 nodes 33

Slide 92

Slide 92 text

Monthly series Annual FourM1 BiM1 M1 M2 BiM2 M3 M4 FourM2 BiM3 M5 M6 BiM4 M7 M8 FourM3 BiM5 M9 M10 BiM6 M11 M12 k = 2, 4, 12 nodes k = 3, 6, 12 nodes Why not k = 2, 3, 4, 6, 12 nodes? 33

Slide 93

Slide 93 text

Monthly data                                           A SemiA1 SemiA2 FourM1 FourM2 FourM3 Q1 . . . Q4 BiM1 . . . BiM6 M1 . . . M12                                           (28×1) =                                           1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 . . . 0 0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 . . . 0 0 0 0 0 0 0 0 0 0 1 1 I12                                           S                                     M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12                                     bt 34

Slide 94

Slide 94 text

In general For a me series y1 , . . . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. 35

Slide 95

Slide 95 text

In general For a me series y1 , . . . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. A single unique hierarchy is only possible when there are no coprime pairs in F(m). 35

Slide 96

Slide 96 text

In general For a me series y1 , . . . , yT, observed at frequency m, we generate aggregate series y[k] j = jk t=1+(j−1)k yt , for j = 1, . . . , T/k k ∈ F(m) = {factors of m}. A single unique hierarchy is only possible when there are no coprime pairs in F(m). Mk = m/k is seasonal period of aggregated series. 35

Slide 97

Slide 97 text

UK Accidents and Emergency Demand 1 2 3 4 5 6 5100 5300 5500 Annual (k=52) Forecast 2 4 6 8 10 12 2500 2600 2700 2800 2900 Semi−annual (k=26) Forecast 5 10 15 20 25 1250 1350 1450 Quarterly (k=13) Forecast 20 40 60 80 360 380 400 420 440 460 Monthly (k=4) Forecast 50 100 150 180 190 200 210 220 230 Bi−weekly (k=2) Forecast 50 100 150 200 250 300 90 95 100 105 110 Weekly (k=1) Forecast 36 – – – – base reconciled

Slide 98

Slide 98 text

Outline 1 Sub-daily me series analysis 2 Time series feature analysis 3 Time series anomaly detec on 4 Probabilis c electricity demand analysis 5 Forecast reconcilia on 6 Temporal hierarchies 7 R packages 37

Slide 99

Slide 99 text

R packages 38 Papers, packages and slides available at robjhyndman.com