Elastic for Time Series Data and Predictive Analytics

Slide 1

Slide 1 text

1 ChristophWurm, Solutions Architect January 2016 ELASTIC FOR TIME SERIES DATA PREDICTIVE ANALYTICS

Slide 2

Slide 2 text

2 The Elastic Community 40,000 Community members 35,000 Commits against Elastic stack to-date

Slide 3

Slide 3 text

3 Viral Adoption Mar’15 Oct’12 Apr’13 Apr’14 Oct’13 20. Millions of Downloads 10. 40+ Million Downloads Cumulative across Elastic products to date Nov’15 40. Sept’14

Slide 4

Slide 4 text

4 What is Elastic? Platformaround a distributed data store By developers for developers Massive amounts of structured and unstructured data Real-time at scale

Slide 5

Slide 5 text

5 Elastic stack Logstash Collect, parse and enrich data Elasticsearch Store, search, analyze Hadoop Ecosystem Hadoop connector Beats Tap into your wire data Shield Security Watcher Scheduler Marvel - Monitoring Found Scale in the cloud Kibana Visualize and explore data Training Professional Services Support Subscriptions BUILT FOR TODAY’S SCALABLE, DISTRIBUTED SYSTEMS

Slide 6

Slide 6 text

6 What is Time Series Data? Has a timestamp Older and newer data Older data is less important Very old data will be deleted Random variation Trends and predictions

Slide 7

Slide 7 text

7 Time Series Architecture Filebeat Log files Packetbeat Packet sniffing Topbeat Server metrics Execbeat Arbitrary commands logstash-input-* JDBC, Twitter, *MQ, etc. Roll your own! Java, .NET, Python, etc. Logstash ES ES ES Kibana Timelion Custom

Slide 8

Slide 8 text

8 Demo #1 TIMELION

Slide 9

Slide 9 text

9 New in Elasticsearch2.0 Pipeline Aggregations “Aggregations on top of other aggregations” Derivatives Moving average Holt-Winters (prediction / anomaly detection) Custom

Slide 10

Slide 10 text

10 Moving Average

Slide 11

Slide 11 text

11 Linear Trend

Slide 12

Slide 12 text

12 Cyclic Trends (Holt-Winters)

Slide 13

Slide 13 text

13 In-Depth MOVING AVERAGE

Slide 14

Slide 14 text

14 { model: simple window: 180 } Simple, unweighted moving average (basically the mean)

Slide 15

Slide 15 text

15 { model: simple window: 720 } Simple, unweighted moving average (basically the mean)

Slide 16

Slide 16 text

16 Simple, unweighted moving average (basically the mean) { model: simple window: 10 }

Slide 17

Slide 17 text

17 Simple, unweighted moving average (basically the mean) { model: simple window: 100 }

Slide 18

Slide 18 text

18 { model: linear window: 180 } Linear weighted moving average

Slide 19

Slide 19 text

19 { model: ewma window: 180 } Exponential weighted moving average (Overfitting?)

Slide 20

Slide 20 text

20 { model: holt window: 180 } Holt-Linear double exponential weighted moving average Trend

Slide 21

Slide 21 text

21 { model: holt_winters window: 360 predict: 120 settings: { type: mult period: 120 } } Holt-Winters triple exponential weighted moving average Prediction

Slide 22

Slide 22 text

22 Demo #2 PREDICTIVE ANALYTICS

Slide 23

Slide 23 text

23 MADRID, Spain January 19 - 21 BERLIN,Germany January 25 - 28 COPENHAGEN, Denmark January 26 - 29 PARIS, France February 1 - 4 LONDON, United Kingdom February 3 - 5 AMSTERDAM, Netherlands February 8 - 11 training.elastic.co