Save 37% off PRO during our Black Friday Sale! »

Well, That Escalated Quickly: 
Anomaly Detection with Elastic Machine Learning

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
June 29, 2017

Well, That Escalated Quickly: 
Anomaly Detection with Elastic Machine Learning

Mike Barretta, Solution Architect, gave this talk at DeveloperWeek NYC on June 20


Elastic Co

June 29, 2017


  1. Well, That Escalated Quickly: 
 Anomaly Detection with Elastic Machine

    Learning Mike Barretta Solution Architect
  2. Technically… 2 Copyright © 2004 DreamWorks LLC All Rights Reserved

  3. About Me 3 A generalist and information technologist • Customer

    support • Software development • System administration • “Data science” • Consulting • Sales
 A husband and father • Married 13 years • 4 kids • 1 dog • 1 hamster An intoxicated (and intoxicating!), chess playing, movie quoting literate • Wine is fine • Your king is mine • I’ve got the line “I am no one to be trifled with; that is all you ever need know” -- Name the movie! Copyright © 1987 20th Century Fox All Rights Reserved
  4. Statistics since 2012, founding of Elastic 4 85,000+ Community Members

    3,000+ Subscription Customers By the Numbers 100,000,000+ Product Downloads
  5. Tech Finance Telco Consumer 5 Enterprise Customers in Every Industry

  6. Massive Startup Adoption 6

  7. 7 Elastic Stack Scalable, near real-time search, discovery, and analytics

    100% open source
  8. 8 X-Pack Extensions for the Elastic Stack Subscription pricing Single

    install 30 day free trial Security Alerting Monitoring Reporting Graph Machine Learning
  9. 9 Elastic Cloud Hosted Elasticsearch & Kibana Includes X-Pack features

    Starts at $45/mo
  10. 10 Elastic Cloud Enterprise On-Premise Elastic Cloud Centrally Provision &

    Manage Clusters
  11. 11 Support Included in Elastic Subscription Packages; More than traditional

    technical support Architecture / Index / Shard Design Cluster Management & Fine Tuning Query Performance Optimization Dev to Production Migration & Upgrades Elastic Stack and X-Pack Best Practices Experienced Support Engineers Training Consulting Become an Elastic Stack expert. Hands on technical and operational courses. Public courses around the world. Private courses on request. Accelerate your team’s success. Work with our consultants at any phase of your implementation – prototype, design, development, migration, and optimization.
  12. Machine Learning Beta
 It catches what you miss, all

    by itself 12 PLATINUM
  13. Anomalies in your data could indicate trouble 13 Spiked 404

    errors Web attack IT Operational Analytics Security Analytics IoT / SCADA Unusual DNS activity Data exfiltration Rare log messages Failing sensor
  14. Detecting (noteworthy) anomalies is hard! 14 Where’s the anomaly? Visual

    inspection is not practical…
  15. Detecting (noteworthy) anomalies is hard! 15 What’s the right threshold?

    Rule-based alerts are insufficient…
  16. 16 “Let me sum this up for you: you don’t

    know who you are, you don’t know what you want, and you don’t know what the hell is going on!” -- Name the movie! Copyright © 1988 Paramount Pictures All Rights Reserved
  17. Machine Learning Beta 17 It catches what you miss, all

    by itself
  18. Some History 18 2016 2017 Feb 2016 Prelert announces commercial

    plugin for the Elastic Stack @ elastic{on} 16 Sept 2016 Elastic acquires Prelert + May 2017 Prelert tech renamed “Machine Learning” and added to X-Pack “Mmm, that sounds good, I’ll have that.” -- Name the movie! Copyright © 1994 New Line Productions, Inc. All Rights Reserved
  19. 19 What is it: Anomaly detection on time series data

    contained within Elasticsearch indices • Simple to use! • Scalable: ES & Kibana plugin • Mature, in-production capability • Online, unsupervised, with Bayesian model selection and periodicity detection • Fully automated: baselines, detection, and scoring What is it not: Anything else* * The plan is to add additional ML-driven features over tim
  20. The Process: 
 It’s as easy as 1, 2, 3…4

  21. Step 1: Feature Selection == Detector Definition 21 Deviations in

 Counts or Values “high_sum(purchase_amt)”
 “count by error_type” “max(responsetime) by airline” Rare Events “rare by event_id” “rare by process” Unusual vs. Population “count by error_type over user” “sum(bytes) over job_title”
  22. Step 2: Modeling via Observation 22 Sophisticated ML techniques to

    best-fit the right statistical model for the feature(s) you selected Better models ! better outlier detection ! less false positives “Well maybe the real God uses tricks, you know? Maybe he’s not omnipotent. He’s just been around so long he knows everything.” -- Name the movie! Copyright © 1993 Columbia Pictures Industries, Inc. All Rights Reserved
  23. Step 3: Periodicity Detection 23 After 3 full days, daily

    periodicity has been learned. After 3 occurrences, weekly periodicity has been learned.
  24. Step 4: Scoring of Unusualness 24

  25. Sequence Diagram 25 _index: models _index: results _index: data get

    data get model update model save results get results
  26. 26 The Demo

  27. I, for one, am wary of our new AI overlords

    27 “This mission is too important for me to allow you to jeopardize it.” “General, you are listening to a machine! Do the world a favor and don't act like one.” Copyright © 1966 Metro-Goldwyn-Mayer, Inc. All Rights Reserved Copyright © 1982 United Artists Corporation All Rights Reserved
  28. For More Information 28 YouTube tutorials:

  29. THANK YOU @elastic