Time Series Analysis by JavaScript @LLMatsuri2013

Transcript

1. Time Series analysis by JavaScript LL matsuri 2013/08/24 1
2. Time Series analysis by JavaScript લॲཧ LL matsuri 2013/08/24 2
3. {twitter: “muddydixon”} ✓Data Mining ✓Data Visualization ✓Love D3.js ✓working @ NIFTY 3
4. NIFTY Cloud 4
5. Visualization ✓What is Visualization? communicating information clearly and effectively 5
6. Visualization on Web ✓Profit ✓Watch KGI / KPI continuously ✓Share visualization via URL ✓empowered representation by HTML5 / css3 -> D3.js 6
7. Visualization on Web : D3.js 7
8. Time Series Visualization ✓Visualize “time series” of ✓KGI / KP of services ✓Metrics of system resources 8
9. Sample {"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427} {"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494} {"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888} {"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641} {"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452} {"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346} {"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358} {"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969} {"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213} {"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109} {"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633} {"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777} {"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732} {"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024} {"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203} {"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324} {"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145} {"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285} {"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979} {"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581} {"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261} {"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419} {"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493} {"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272} {"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287} {"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856} {"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998} {"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702} {"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021} {"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679} {"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319} 9
10. ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ σΠϦʔͰ ച্σʔλݟͤͯ ྃղ͠·ͨ͠ 10
11. ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ ΍ͬͺΓɺ݄࣍ ͱि࣍Ͱ ʜྃղ͠·ͨ͠ 11
12. ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ ͋ͱച্ͷฏۉ ͱ෼ࢄ΋ ͏ͬɺͭΒ͍ 12
13. Summarize keyvalues = {} for d in data time = new Date(d.time.getFullYear(), d.time.getMonth(), d.time.getDate(), d.time.getHours(), d.time.getMinutes()) keyvalues[time] = [] unless keyvalues[time] keyvalues[time].push d aggregated = [] for time, values of keyvalues obj = {time: time, cpu: 0, memory: 0} for d in values obj.cpu += d.cpu obj.memory += d.memory aggregated.push obj console.log aggregated 13
14. Problems Large Time perspectives / Statistics Avoid to get data each time Hourly / Daily / Weekly / Monthly / Yearly Summary / Average / Stdev 14
15. Problems ϝϯυΫαΠ 15
16. Series.js 16
17. Series.js ✓Providing utility methods for time series ✓sum, sum square, mean, median, quantile ✓variance, standard devience, covariance, correlation ✓auto covariance, auto correlation ✓Providing aggregation ✓minutes, hour, day, week, month, year ✓Coming soon ✓auto regression, spectrum ✓moving average, holt-winters, exponential moving average 17
18. Series.js APIs example # class methods Series.sum([1,2,3,4,5]) #=> 15 Series.mean([1,2,3,4,5]) #=> 3 # instance methods s = new Series() s.put d for d in [1,2,3,4,5] s.mean() #=> 3 18
19. Series.js APIs example # accessor Series.y((d)-> d.v).sum([{v:1},{v:2},{v:3},{v:4},{v: 5}]) #=> 15 s = new Series().y((d)-> d.v) s.put d for d in [{v:1}, {v:2}, {v:3}, {v:4}, {v:5}] s.mean() #=> 3 19
20. Series.js APIs example # aggregation data = [{t: 1, v:1}, {t: 1, v:2}, {t:2, v:3}, {t:2, v:4}, {t:1, v:5}] Series.y((d)-> d.v).aggregation(Series.sum).key((d)- > d.t)(data) #=> [{t:1, v: 8}, {t: 2, v: 7}] 20
21. Sample {"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427} {"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494} {"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888} {"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641} {"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452} {"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346} {"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358} {"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969} {"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213} {"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109} {"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633} {"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777} {"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732} {"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024} {"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203} {"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324} {"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145} {"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285} {"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979} {"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581} {"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261} {"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419} {"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493} {"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272} {"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287} {"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856} {"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998} {"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702} {"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021} {"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679} {"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319} 21
22. Summarize with Series.js Series.y((d)-> d.cpu).aggregation(Series.sum) .minute((d)-> d.time)(data) #=> [{ t: 'Sun Jan 13 2013 23:29:00 GMT+0900 (JST)', y: 2.0095 }, { t: 'Sun Jan 13 2013 23:30:00 GMT+0900 (JST)', y: 3.4370 }, { t: 'Sun Jan 13 2013 23:31:00 GMT+0900 (JST)', y: 1.58180 }, { t: 'Sun Jan 13 2013 23:32:00 GMT+0900 (JST)', y: 2.2742 }, { t: 'Sun Jan 13 2013 23:33:00 GMT+0900 (JST)', y: 3.4541 }, { t: 'Sun Jan 13 2013 23:34:00 GMT+0900 (JST)', y: 4.6035 }, { t: 'Sun Jan 13 2013 23:35:00 GMT+0900 (JST)', y: 2.1165 }, { t: 'Sun Jan 13 2013 23:36:00 GMT+0900 (JST)', y: 1.6177 }, 22
23. TODO Performance Refactoring Document 23
24. TODO Client side 24
25. TODO Client side ͖ͬ͞QVTI͠·ͨ͠ 25
26. CROSS 2014 26
27. ҎԼɺϏʔϧΛҿΜͰ সإͳٕज़ऀͷإ͕ ଓ͍͍ͯΔͱ ࢥ͍ͬͯͩ͘͞ 27