Tweet
Tweet

Slide 1

Slide 1 text

Time Series analysis by JavaScript LL matsuri 2013/08/24 1

Slide 2

Slide 2 text

Time Series analysis by JavaScript લॲཧ LL matsuri 2013/08/24 2

Slide 3

Slide 3 text

{twitter: “muddydixon”} ✓Data Mining ✓Data Visualization ✓Love D3.js ✓working @ NIFTY 3

Slide 4

Slide 4 text

NIFTY Cloud 4

Slide 5

Slide 5 text

Visualization ✓What is Visualization? communicating information clearly and effectively 5

Slide 6

Slide 6 text

Visualization on Web ✓Profit ✓Watch KGI / KPI continuously ✓Share visualization via URL ✓empowered representation by HTML5 / css3 -> D3.js 6

Slide 7

Slide 7 text

Visualization on Web : D3.js 7

Slide 8

Slide 8 text

Time Series Visualization ✓Visualize “time series” of ✓KGI / KP of services ✓Metrics of system resources 8

Slide 9

Slide 9 text

Sample {"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427} {"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494} {"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888} {"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641} {"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452} {"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346} {"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358} {"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969} {"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213} {"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109} {"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633} {"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777} {"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732} {"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024} {"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203} {"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324} {"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145} {"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285} {"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979} {"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581} {"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261} {"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419} {"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493} {"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272} {"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287} {"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856} {"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998} {"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702} {"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021} {"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679} {"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319} 9

Slide 10

Slide 10 text

ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ σΠϦʔͰ ച্σʔλݟͤͯ ྃղ͠·ͨ͠ 10

Slide 11

Slide 11 text

ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ ΍ͬͺΓɺ݄࣍ ͱि࣍Ͱ ʜྃղ͠·ͨ͠ 11

Slide 12

Slide 12 text

ՄࢹԽΤϯδχΞ Ϛωʔδϟʔ ͋ͱച্ͷฏۉ ͱ෼ࢄ΋ ͏ͬɺͭΒ͍ 12

Slide 13

Slide 13 text

Summarize keyvalues = {} for d in data time = new Date(d.time.getFullYear(), d.time.getMonth(), d.time.getDate(), d.time.getHours(), d.time.getMinutes()) keyvalues[time] = [] unless keyvalues[time] keyvalues[time].push d aggregated = [] for time, values of keyvalues obj = {time: time, cpu: 0, memory: 0} for d in values obj.cpu += d.cpu obj.memory += d.memory aggregated.push obj console.log aggregated 13

Slide 14

Slide 14 text

Problems Large Time perspectives / Statistics Avoid to get data each time Hourly / Daily / Weekly / Monthly / Yearly Summary / Average / Stdev 14

Slide 15

Slide 15 text

Problems ϝϯυΫαΠ 15

Slide 16

Slide 16 text

Series.js 16

Slide 17

Slide 17 text

Series.js ✓Providing utility methods for time series ✓sum, sum square, mean, median, quantile ✓variance, standard devience, covariance, correlation ✓auto covariance, auto correlation ✓Providing aggregation ✓minutes, hour, day, week, month, year ✓Coming soon ✓auto regression, spectrum ✓moving average, holt-winters, exponential moving average 17

Slide 18

Slide 18 text

Series.js APIs example # class methods Series.sum([1,2,3,4,5]) #=> 15 Series.mean([1,2,3,4,5]) #=> 3 # instance methods s = new Series() s.put d for d in [1,2,3,4,5] s.mean() #=> 3 18

Slide 19

Slide 19 text

Series.js APIs example # accessor Series.y((d)-> d.v).sum([{v:1},{v:2},{v:3},{v:4},{v: 5}]) #=> 15 s = new Series().y((d)-> d.v) s.put d for d in [{v:1}, {v:2}, {v:3}, {v:4}, {v:5}] s.mean() #=> 3 19

Slide 20

Slide 20 text

Series.js APIs example # aggregation data = [{t: 1, v:1}, {t: 1, v:2}, {t:2, v:3}, {t:2, v:4}, {t:1, v:5}] Series.y((d)-> d.v).aggregation(Series.sum).key((d)- > d.t)(data) #=> [{t:1, v: 8}, {t: 2, v: 7}] 20

Slide 21

Slide 21 text

Sample {"time":"2013-01-02T08:54:31.000Z","cpu":0.6283,"memory":0.4427} {"time":"2013-01-02T08:54:22.000Z","cpu":0.6103,"memory":0.6494} {"time":"2013-01-02T08:54:24.000Z","cpu":0.6096,"memory":0.4888} {"time":"2013-01-02T08:54:25.000Z","cpu":0.6811,"memory":0.641} {"time":"2013-01-02T08:54:14.000Z","cpu":0.6028,"memory":0.6452} {"time":"2013-01-02T08:54:06.000Z","cpu":0.5565,"memory":0.4346} {"time":"2013-01-02T08:54:10.000Z","cpu":0.4161,"memory":0.358} {"time":"2013-01-02T08:55:01.000Z","cpu":0.5256,"memory":0.5969} {"time":"2013-01-02T08:55:47.000Z","cpu":0.6145,"memory":0.3213} {"time":"2013-01-02T08:56:24.000Z","cpu":0.6896,"memory":0.5109} {"time":"2013-01-02T08:56:35.000Z","cpu":0.443,"memory":0.633} {"time":"2013-01-02T08:56:11.000Z","cpu":0.4746,"memory":0.4777} {"time":"2013-01-02T08:56:42.000Z","cpu":0.503,"memory":0.5732} {"time":"2013-01-02T08:56:38.000Z","cpu":0.4731,"memory":0.4024} {"time":"2013-01-02T08:57:01.000Z","cpu":0.405,"memory":0.5203} {"time":"2013-01-02T08:57:54.000Z","cpu":0.4496,"memory":0.5324} {"time":"2013-01-02T08:57:04.000Z","cpu":0.3528,"memory":0.3145} {"time":"2013-01-02T08:57:49.000Z","cpu":0.6659,"memory":0.4285} {"time":"2013-01-02T08:57:19.000Z","cpu":0.3237,"memory":0.6979} {"time":"2013-01-02T08:58:31.000Z","cpu":0.4045,"memory":0.6581} {"time":"2013-01-02T08:58:45.000Z","cpu":0.3134,"memory":0.3261} {"time":"2013-01-02T08:58:44.000Z","cpu":0.4382,"memory":0.419} {"time":"2013-01-02T08:58:38.000Z","cpu":0.6983,"memory":0.3493} {"time":"2013-01-02T08:58:34.000Z","cpu":0.6645,"memory":0.6272} {"time":"2013-01-02T08:59:54.000Z","cpu":0.604,"memory":0.3287} {"time":"2013-01-02T08:59:25.000Z","cpu":0.574,"memory":0.4856} {"time":"2013-01-02T08:59:19.000Z","cpu":0.4775,"memory":0.3998} {"time":"2013-01-02T08:59:34.000Z","cpu":0.5047,"memory":0.6702} {"time":"2013-01-02T08:59:46.000Z","cpu":0.672,"memory":0.5021} {"time":"2013-01-02T08:59:53.000Z","cpu":0.3278,"memory":0.6679} {"time":"2013-01-02T08:59:29.000Z","cpu":0.4494,"memory":0.3319} 21

Slide 22

Slide 22 text

Summarize with Series.js Series.y((d)-> d.cpu).aggregation(Series.sum) .minute((d)-> d.time)(data) #=> [{ t: 'Sun Jan 13 2013 23:29:00 GMT+0900 (JST)', y: 2.0095 }, { t: 'Sun Jan 13 2013 23:30:00 GMT+0900 (JST)', y: 3.4370 }, { t: 'Sun Jan 13 2013 23:31:00 GMT+0900 (JST)', y: 1.58180 }, { t: 'Sun Jan 13 2013 23:32:00 GMT+0900 (JST)', y: 2.2742 }, { t: 'Sun Jan 13 2013 23:33:00 GMT+0900 (JST)', y: 3.4541 }, { t: 'Sun Jan 13 2013 23:34:00 GMT+0900 (JST)', y: 4.6035 }, { t: 'Sun Jan 13 2013 23:35:00 GMT+0900 (JST)', y: 2.1165 }, { t: 'Sun Jan 13 2013 23:36:00 GMT+0900 (JST)', y: 1.6177 }, 22

Slide 23

Slide 23 text

TODO Performance Refactoring Document 23

Slide 24

Slide 24 text

TODO Client side 24

Slide 25

Slide 25 text

TODO Client side ͖ͬ͞QVTI͠·ͨ͠ 25

Slide 26

Slide 26 text

CROSS 2014 26

Slide 27

Slide 27 text

ҎԼɺϏʔϧΛҿΜͰ সإͳٕज़ऀͷإ͕ ଓ͍͍ͯΔͱ ࢥ͍ͬͯͩ͘͞ 27