Elastic for Time Series Data and Predictive Analytics

Transcript

1. 1
   ChristophWurm, Solutions Architect
   January 2016
   ELASTIC FOR TIME SERIES DATA
   PREDICTIVE ANALYTICS
   

   View Slide

2. 2
   The Elastic Community
   40,000
   Community
   members
   35,000
   Commits against
   Elastic stack to-date
   

   View Slide

3. 3
   Viral Adoption
   Mar’15
   Oct’12 Apr’13 Apr’14
   Oct’13
   20.
   Millions of Downloads
   10.
   40+ Million Downloads
   Cumulative across Elastic products to date
   Nov’15
   40.
   Sept’14
   

   View Slide

4. 4
   What is Elastic?
   Platformaround a distributed data store
   By developers for developers
   Massive amounts of structured and unstructured data
   Real-time at scale
   

   View Slide

5. 5
   Elastic stack
   Logstash
   Collect, parse
   and enrich data
   Elasticsearch
   Store, search, analyze
   Hadoop Ecosystem
   Hadoop connector Beats
   Tap into your
   wire data
   Shield
   Security
   Watcher
   Scheduler
   Marvel - Monitoring
   Found
   Scale in the cloud
   Kibana
   Visualize and explore data
   Training Professional Services Support Subscriptions
   BUILT FOR TODAY’S SCALABLE, DISTRIBUTED SYSTEMS
   

   View Slide

6. 6
   What is Time Series Data?
   Has a timestamp
   Older and newer data
   Older data is less important
   Very old data will be deleted
   Random variation
   Trends and predictions
   

   View Slide

7. 7
   Time Series Architecture
   Filebeat
   Log files
   Packetbeat
   Packet sniffing
   Topbeat
   Server metrics
   Execbeat
   Arbitrary commands
   logstash-input-*
   JDBC, Twitter, *MQ, etc.
   Roll your own!
   Java, .NET, Python, etc.
   Logstash
   ES
   ES
   ES
   Kibana
   Timelion
   Custom
   

   View Slide

8. 8
   Demo #1
   TIMELION
   

   View Slide

9. 9
   New in Elasticsearch2.0
   Pipeline Aggregations
   “Aggregations on top of other aggregations”
   Derivatives
   Moving average
   Holt-Winters (prediction / anomaly detection)
   Custom
   

   View Slide

10. 10
    Moving Average
    

    View Slide

11. 11
    Linear Trend
    

    View Slide

12. 12
    Cyclic Trends (Holt-Winters)
    

    View Slide

13. 13
    In-Depth
    MOVING AVERAGE
    

    View Slide

14. 14
    {
    model: simple
    window: 180
    }
    Simple, unweighted moving average (basically the mean)
    

    View Slide

15. 15
    {
    model: simple
    window: 720
    }
    Simple, unweighted moving average (basically the mean)
    

    View Slide

16. 16
    Simple, unweighted moving average (basically the mean)
    {
    model: simple
    window: 10
    }
    

    View Slide

17. 17
    Simple, unweighted moving average (basically the mean)
    {
    model: simple
    window: 100
    }
    

    View Slide

18. 18
    {
    model: linear
    window: 180
    }
    Linear weighted moving average
    

    View Slide

19. 19
    {
    model: ewma
    window: 180
    }
    Exponential weighted moving average (Overfitting?)
    

    View Slide

20. 20
    {
    model: holt
    window: 180
    }
    Holt-Linear double exponential weighted moving average
    Trend
    

    View Slide

21. 21
    {
    model: holt_winters
    window: 360
    predict: 120
    settings: {
    type: mult
    period: 120
    }
    }
    Holt-Winters triple exponential weighted moving average
    Prediction
    

    View Slide

22. 22
    Demo #2
    PREDICTIVE ANALYTICS
    

    View Slide

23. 23
    MADRID, Spain January 19 - 21
    BERLIN,Germany January 25 - 28
    COPENHAGEN, Denmark January 26 - 29
    PARIS, France February 1 - 4
    LONDON, United Kingdom February 3 - 5
    AMSTERDAM, Netherlands February 8 - 11
    training.elastic.co
    

    View Slide

24. View Slide

25. 25
    Q&A
    ASK ME ANYTHING
    

    View Slide