Milan Meetup June 2017 - Machine Learning

Transcript

1. ‹#› Abdon Pijpelink Education Engineer Milan Elastic Meetup, June 7th, 2017 Machine Learning
2. Machine Learning History • Prelert founded in 2010 • Prelert joined Elastic in September 2016 • Elastic Stack 5.4: Machine Learning beta • Elastic Stack 5.5: Machine Learning GA
3. The Elastic Stack
4. Why? 4
7. Why? - Problem 1: Information Overload • Manually search for “interesting” events in the data • Visually scrutinise (“eyeball”) for “abnormal” behaviour • Attempt to create thresholds/rules for alerting 7 abnormal? related? related?
8. Traditional Approaches to Anomaly Detection • Manually search for interesting events in the data • Visually scrutinize (“eyeball”) for abnormal behavior • Attempt to create thresholds/rules for alerting 8
9. Why? - Problem 1: Information Overload 9 You have to know what you are looking for.
10. Why? - Problem 2: Eyeball Deception 10 Where is the anomaly?
11. Why? - Problem 2: Eyeball Deception 11 Machine Learning model overlaid
12. Why? - Problem 3: Thresholds/Rules too rigid 12
13. Why? - Problem 3: Thresholds/Rules too rigid 13
14. How? 14
15. What would an ideal solution look like? Magic? any data actionable
 results Black
 Magic
16. What would an ideal solution look like? Math! any data actionable
 results
17. What is the best Model For this data based on changing criteria 17 • Use sophisticated machine-learning techniques to best-fit the right statistical model for your data. • Better models = better outlier detection = less false alarms
18. Periodicity – Self Learned (Daily) 18 Machine Learning model overlaid
19. Periodicity – Self Learned (Daily) 19 Machine Learning model becomes more focused with time & data
20. Demo 20
21. Architecture 21
22. Machine Learning Architecture
23. Scaling a Cluster node.master: true node.data: true node.ingest: true
24. Scaling a Cluster with Machine Learning node.master: true node.data: true node.ingest: true node.ml: true
25. Questions? 25