Probabilistic outlier detection and visualization of smart metre data

Standard time series analysis and visualization tools fail on smart metre data due to the sheer volume of available data (both in the time dimension and due to the large numbers of smart metres providing data). In addition, smart metre data is often messy, with missing observations, periods where some metres are offline, periods of relatively constant low level energy usage, occasional days of unusually high demand, and so on.

We introduce some new tools for exploring large collections of smart metre data based on the probability distribution of demand for each household, as it varies by the time of day and day of the week.

We are particularly interested in clustering households into groups with similar probability demand distributions, and in identifying households with unusual demand distributions. It is also of interest to estimate a typical household distribution.

Our approach involves computing the pairwise Jensen-Shannon distances between household probability distributions. Then we show that a kernel density estimate can be constructed on the distribution of distances, allowing us to cluster similar households, and find unusual households.

Transcript

1. 1 Probabilistic outlier detection and visualization of smart metre data Rob J Hyndman Cairns, June 2017
2. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 2
3. Irish smart metre data Figure: http://solutions.3m.com 500 households from smart metering trial Electricity consumption at 30-minute intervals between 14 July 2009 and 31 December 2010 Heating/cooling energy usage excluded 3
4. Irish smart metre data 0 2 4 6 0 200 400 Days Demand (kWh) Demand for ID: 1718 4
5. Irish smart metre data 0 1 2 3 4 5 0 200 400 Days Demand (kWh) Demand for ID: 1550 5
6. Irish smart metre data 0 2 4 6 0 200 400 Days Demand (kWh) Demand for ID: 1539 6
7. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 7
8. Quantiles conditional on time of week Compute sample quantiles at p = 0.01, 0.02, . . . , 0.99 for each household and each half-hour of the week. 168 probability distributions per household. Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Demand (kWh) Demand for ID: 1718 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 1 2 3 4 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 8
9. Quantiles conditional on time of week Compute sample quantiles at p = 0.01, 0.02, . . . , 0.99 for each household and each half-hour of the week. 168 probability distributions per household. Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 1 2 3 4 5 Demand (kWh) Demand for ID: 1550 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 1 2 3 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 9
10. Quantiles conditional on time of week Compute sample quantiles at p = 0.01, 0.02, . . . , 0.99 for each household and each half-hour of the week. 168 probability distributions per household. Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Demand (kWh) Demand for ID: 1539 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 10
11. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: 11
12. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros 11
13. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support 11
14. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness 11
15. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness Avoids missing data issues and variation in series length 11
16. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness Avoids missing data issues and variation in series length Avoids timing of household events, holidays, etc. 11
17. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness Avoids missing data issues and variation in series length Avoids timing of household events, holidays, etc. Allows clustering of households based on probabilistic behaviour rather than coincident behaviour. 11
18. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness Avoids missing data issues and variation in series length Avoids timing of household events, holidays, etc. Allows clustering of households based on probabilistic behaviour rather than coincident behaviour. Allows identification of anomalous households. 11
19. Quantiles conditional on time of week Sample quantiles better than kernel density estimate: presence of zeros non-negative support high skewness Avoids missing data issues and variation in series length Avoids timing of household events, holidays, etc. Allows clustering of households based on probabilistic behaviour rather than coincident behaviour. Allows identification of anomalous households. Allows estimation of typical household behaviour. 11
20. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 12
21. Pairwise distances The time series of 535 × 48 observations per household is mapped to a set of 7×48×99 quantiles giving a bivariate surface for each household. Can we compute pairwise distances between all households? 13 −→ ← ? → Distance
22. Jensen-Shannon distances Kullback-Leibler divergence between two densities D(p, q) = ∞ ∞ p(x) log p(x) q(x) dx 14
23. Jensen-Shannon distances Kullback-Leibler divergence between two densities D(p, q) = ∞ ∞ p(x) log p(x) q(x) dx Not symmetric: D(p, q) = D(q, p) 14
24. Jensen-Shannon distances Kullback-Leibler divergence between two densities D(p, q) = ∞ ∞ p(x) log p(x) q(x) dx Not symmetric: D(p, q) = D(q, p) Jensen-Shannon distance between two densities JS(p, q) = [D(p, r) + D(q, r)]/2 where r = (p + q)/2 14
25. Jensen-Shannon distances Kullback-Leibler divergence between two densities D(p, q) = ∞ ∞ p(x) log p(x) q(x) dx Not symmetric: D(p, q) = D(q, p) Jensen-Shannon distance between two densities JS(p, q) = [D(p, r) + D(q, r)]/2 where r = (p + q)/2 Distance between two households ∆ij = 7×48 t=1 JS(pt , qt) 14
26. Kernel matrix and density ranking Similarity between two households wij = exp(−∆2 ij /h2). 15
27. Kernel matrix and density ranking Similarity between two households wij = exp(−∆2 ij /h2). Row sums of the kernel matrix gives a scaled kernel density estimate of households: ˆ fi = n j=1 wij h is bandwidth in Gaussian kernel. Households can be ranked by density values. 15
28. Typical households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 2.5 5.0 7.5 Demand (kWh) Demand for ID: 1672 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 16
29. Typical households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Demand (kWh) Demand for ID: 1058 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 17
30. Typical households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 2.5 5.0 7.5 Demand (kWh) Demand for ID: 1183 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 18
31. Anomalous households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.5 1.0 Demand (kWh) Demand for ID: 1881 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.3 0.6 0.9 1.2 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 19
32. Anomalous households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.5 1.0 1.5 2.0 Demand (kWh) Demand for ID: 1607 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.5 1.0 1.5 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 20
33. Anomalous households Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.5 1.0 1.5 2.0 2.5 Demand (kWh) Demand for ID: 1821 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0.0 0.5 1.0 1.5 2.0 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 21
34. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 22
35. Laplacian eigenmaps Idea: Embed conditional densities in a 2d space where the distances are preserved “as far as possible”. 23
36. Laplacian eigenmaps Idea: Embed conditional densities in a 2d space where the distances are preserved “as far as possible”. Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij L = D − W (the Laplacian matrix). 23
37. Laplacian eigenmaps Idea: Embed conditional densities in a 2d space where the distances are preserved “as far as possible”. Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij L = D − W (the Laplacian matrix). Solve generalized eigenvector problem: Le = λDe. 23
38. Laplacian eigenmaps Idea: Embed conditional densities in a 2d space where the distances are preserved “as far as possible”. Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij L = D − W (the Laplacian matrix). Solve generalized eigenvector problem: Le = λDe. Let ek be eigenvector corresponding to kth smallest eigenvalue. 23
39. Laplacian eigenmaps Idea: Embed conditional densities in a 2d space where the distances are preserved “as far as possible”. Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij L = D − W (the Laplacian matrix). Solve generalized eigenvector problem: Le = λDe. Let ek be eigenvector corresponding to kth smallest eigenvalue. Then e2 and e3 create an embedding of households in 2d space. 23
40. Key property of Laplacian embedding Let yi = (e2,i , e3,i) be the embedded point corresponding to household i. Then the Laplacian eigenmap minimizes ij wij(yi − yj)2 = y Ly such that y Dy = 1. 24
41. Key property of Laplacian embedding Let yi = (e2,i , e3,i) be the embedded point corresponding to household i. Then the Laplacian eigenmap minimizes ij wij(yi − yj)2 = y Ly such that y Dy = 1. the most similar points are as close as possible. 24
42. Key property of Laplacian embedding Let yi = (e2,i , e3,i) be the embedded point corresponding to household i. Then the Laplacian eigenmap minimizes ij wij(yi − yj)2 = y Ly such that y Dy = 1. the most similar points are as close as possible. First eigenvalue is 0 due to translation invariance. 24
43. Key property of Laplacian embedding Let yi = (e2,i , e3,i) be the embedded point corresponding to household i. Then the Laplacian eigenmap minimizes ij wij(yi − yj)2 = y Ly such that y Dy = 1. the most similar points are as close as possible. First eigenvalue is 0 due to translation invariance. Equivalent to optimal embedding using Laplace-Beltrami operator on manifolds. 24
44. Outliers shown in embedded space q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1881 1607 1821 −4 −2 0 2 −2 −1 0 1 2 Comp1 Comp2 HDRs q q q q 1 50 99 >99 Laplacian embedding (HDRs on original space) 25
45. Outliers shown in embedded space q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1 2 3 4 5 6 7 8 9 10 −4 −2 0 2 −2 −1 0 1 2 Comp1 Comp2 HDRs q q q q 1 50 99 >99 Laplacian embedding (HDRs on original space) 26
46. Outliers computed in embedded space: q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1881 1607 1816 −4 −2 0 2 −2 −1 0 1 2 Comp1 Comp2 HDRs q q q q 1 50 99 >99 Laplacian embedding (HDRs on embedded space) 27
47. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 28
48. Spectral clustering Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij A = D−1/2WD−1/2 (the affinity matrix). Top k eigenvectors of A are clustered using k-means. 29
49. Spectral clustering Let W = [wij] where wij = exp(−∆2 ij /h2). D = diag(ˆ fi) where ˆ fi = n j=1 wij A = D−1/2WD−1/2 (the affinity matrix). Top k eigenvectors of A are clustered using k-means. Not necessarily same h as used for embedding or density-ranking. 29
50. Clustering shown in embedded space q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q −4 −2 0 2 −2 −1 0 1 2 Comp1 Comp2 Cluster q q q 1 2 3 Laplacian embedding with spectral clustering 30
51. Clustering shown in embedded space q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1081 1058 1810 −4 −2 0 2 −2 −1 0 1 2 Comp1 Comp2 Cluster q q q 1 2 3 Laplacian embedding with spectral clustering 31
52. Typical household in cluster 1 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Demand (kWh) Demand for ID: 1081 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 32
53. Typical household in cluster 2 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Demand (kWh) Demand for ID: 1058 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 33
54. Typical household in cluster 3 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 8 Demand (kWh) Demand for ID: 1810 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 6 12 18 24 0 2 4 6 Time of day Quantiles 0.1 0.3 0.5 0.7 0.9 Probability 34
55. Outline 1 Irish smart metre data 2 Quantiles conditional on time of week 3 Finding typical and unusual households 4 Visualization via embedding 5 Spectral clustering and embedding 6 Features and limitations 35
56. Features and limitations Features of approach Converting time series to quantile surfaces conditional on time of week. Using pairwise distances between households Using kernel matrices for density ranking, embedding and clustering 36
57. Features and limitations Features of approach Converting time series to quantile surfaces conditional on time of week. Using pairwise distances between households Using kernel matrices for density ranking, embedding and clustering Unresolved issues Need to select the bandwidth h in constructing the similarity matrix. Three different uses of bandwidth: density-ranking, embedding, clustering. Different bandwidth in each case? The use of pairwise distances makes it hard to scale this algorithm. 36