Well, That Escalated Quickly: Anomaly Detection with Elastic Machine Learning

Mike Barretta, Solution Architect, gave this talk at DeveloperWeek NYC on June 20

Transcript

1. Well, That Escalated Quickly: 
 Anomaly Detection with Elastic Machine Learning Mike Barretta Solution Architect
2. Technically… 2 Copyright © 2004 DreamWorks LLC All Rights Reserved
3. About Me 3 A generalist and information technologist • Customer support • Software development • System administration • “Data science” • Consulting • Sales
 A husband and father • Married 13 years • 4 kids • 1 dog • 1 hamster An intoxicated (and intoxicating!), chess playing, movie quoting literate • Wine is fine • Your king is mine • I’ve got the line “I am no one to be trifled with; that is all you ever need know” -- Name the movie! Copyright © 1987 20th Century Fox All Rights Reserved
4. Statistics since 2012, founding of Elastic 4 85,000+ Community Members 3,000+ Subscription Customers By the Numbers 100,000,000+ Product Downloads
5. Tech Finance Telco Consumer 5 Enterprise Customers in Every Industry Government
6. Massive Startup Adoption 6
7. 7 Elastic Stack Scalable, near real-time search, discovery, and analytics 100% open source
8. 8 X-Pack Extensions for the Elastic Stack Subscription pricing Single install 30 day free trial Security Alerting Monitoring Reporting Graph Machine Learning
9. 9 Elastic Cloud Hosted Elasticsearch & Kibana Includes X-Pack features Starts at $45/mo cloud.elastic.co
10. 10 Elastic Cloud Enterprise On-Premise Elastic Cloud Centrally Provision & Manage Clusters
11. 11 Support Included in Elastic Subscription Packages; More than traditional technical support Architecture / Index / Shard Design Cluster Management & Fine Tuning Query Performance Optimization Dev to Production Migration & Upgrades Elastic Stack and X-Pack Best Practices Experienced Support Engineers Training Consulting Become an Elastic Stack expert. Hands on technical and operational courses. Public courses around the world. Private courses on request. Accelerate your team’s success. Work with our consultants at any phase of your implementation – prototype, design, development, migration, and optimization.
12. Machine Learning Beta
 
 It catches what you miss, all by itself 12 PLATINUM
13. Anomalies in your data could indicate trouble 13 Spiked 404 errors Web attack IT Operational Analytics Security Analytics IoT / SCADA Unusual DNS activity Data exfiltration Rare log messages Failing sensor
14. Detecting (noteworthy) anomalies is hard! 14 Where’s the anomaly? Visual inspection is not practical…
15. Detecting (noteworthy) anomalies is hard! 15 What’s the right threshold? Rule-based alerts are insufficient…
16. 16 “Let me sum this up for you: you don’t know who you are, you don’t know what you want, and you don’t know what the hell is going on!” -- Name the movie! Copyright © 1988 Paramount Pictures All Rights Reserved
17. Machine Learning Beta 17 It catches what you miss, all by itself
18. Some History 18 2016 2017 Feb 2016 Prelert announces commercial plugin for the Elastic Stack @ elastic{on} 16 Sept 2016 Elastic acquires Prelert + May 2017 Prelert tech renamed “Machine Learning” and added to X-Pack “Mmm, that sounds good, I’ll have that.” -- Name the movie! Copyright © 1994 New Line Productions, Inc. All Rights Reserved
19. 19 What is it: Anomaly detection on time series data contained within Elasticsearch indices • Simple to use! • Scalable: ES & Kibana plugin • Mature, in-production capability • Online, unsupervised, with Bayesian model selection and periodicity detection • Fully automated: baselines, detection, and scoring What is it not: Anything else* * The plan is to add additional ML-driven features over tim
20. The Process: 
 It’s as easy as 1, 2, 3…4 20
21. Step 1: Feature Selection == Detector Definition 21 Deviations in 
 Counts or Values “high_sum(purchase_amt)”
 “count by error_type” “max(responsetime) by airline” Rare Events “rare by event_id” “rare by process” Unusual vs. Population “count by error_type over user” “sum(bytes) over job_title”
22. Step 2: Modeling via Observation 22 Sophisticated ML techniques to best-fit the right statistical model for the feature(s) you selected Better models ! better outlier detection ! less false positives “Well maybe the real God uses tricks, you know? Maybe he’s not omnipotent. He’s just been around so long he knows everything.” -- Name the movie! Copyright © 1993 Columbia Pictures Industries, Inc. All Rights Reserved
23. Step 3: Periodicity Detection 23 After 3 full days, daily periodicity has been learned. After 3 occurrences, weekly periodicity has been learned.
24. Step 4: Scoring of Unusualness 24
25. Sequence Diagram 25 _index: models _index: results _index: data get data get model update model save results get results
26. 26 The Demo
27. I, for one, am wary of our new AI overlords 27 “This mission is too important for me to allow you to jeopardize it.” “General, you are listening to a machine! Do the world a favor and don't act like one.” Copyright © 1966 Metro-Goldwyn-Mayer, Inc. All Rights Reserved Copyright © 1982 United Artists Corporation All Rights Reserved
28. For More Information 28 YouTube tutorials: https://goo.gl/eciQu2 www.elastic.co/products/x-pack/machine-learning mike.barretta@elastic.co
29. THANK YOU @elastic www.elastic.co