Anomaly Detection With NuPIC

OSCON 2015 presentation: http://www.oscon.com/open-source-2015/public/schedule/detail/41265

Transcript

1. Anomaly Detection with NuPIC Matt Taylor @rhyolight OS Community Flag-Bearer Numenta, Inc. @numenta
2. What is NuPIC? Numenta Platform for Intelligent Computing http://numenta.org • HTM: Hierarchical Temporal Memory • Based on neuroscience • Sequence Memory • Online Learning
3. NuPIC • One layer of one region of neocortical cells • Highly realistic implementation, very close to neuroscience • Sophisticated neuron model with 1000s of synapses • Finds temporal patterns in data just like your brain https://github.com/numenta/nupic
5. NuPIC Community Trends
6. What are Anomalies? Spa$al&(sta$c)&anomalies& Temporal&anomalies&
7. Challenges with Temporal Anomalies Tolerance)to)noise) Con.nuous)learning)
8. Thresholds vs Pattern Recognition • Thresholds fail when the amplitude of the data stream changes • dynamic thresholds can help • NuPIC does anomaly detection the same way your neocortex does: temporal pattern recognition
9. Example App: Grok for IT Unpredictable  Data Highly  Predictable  Data Con6nuous  Learning http://numenta.com/grok/
10. Example: Grok for Stocks Companies sorted by unusual activity Stock price Stock volume Twitter volume Tweets reveal cause http://numenta.com/grok-for-stocks/
11. All of this is Open Source • https://github.com/numenta/numenta-apps • NuPIC integration • Android code • server-side code • HTM Engine for scaling many models • You can build stuff with this!
12. Scalar and Geospatial Temporal Anomalies
16. Geospatial Anomalies Deviation in path Change in direction
17. Geospatial Anomalies Unusual speed
18. Sample Geospatial App • https://github.com/numenta/nupic.geospatial • Load your own GPS tracks • Uses NuPIC directly
19. Live Minecraft Demo Extracting live X/Y/Z coordinates from the Minecraft world Identifying geospatial anomalies using NuPIC
20. Minecraft Demo Source Code • https://github.com/nupic-community/mine-hack • Java code for Forge (Minecraft mod loader) • Python code for NuPIC • Only 1 NuPIC model
21. HTM Engine Traffic Demo Creating NuPIC models for 153 different traffic paths in New York City. Displaying anomaly indications for each path, associating with traffic alerts.
22. Correlation with NYCDOT Alerts I-495 (LIE) W/B Accident at the Cross Island Parkway in Queens, the left and center lanes are blocked, est. 1 hr. Issued By: NYS - DOT Affected Jurisdictions: Nassau, Queens Counties
23. Correlation with NYCDOT Alerts I-495 (LIE) W/B Accident at the Cross Island Parkway in Queens, the left and center lanes are blocked, est. 1 hr. Issued By: NYS - DOT Affected Jurisdictions: Nassau, Queens Counties
24. Correlation with NYCDOT Alerts Grand Central Pkwy W/B approaching Jewel Ave blocking all lanes due to an accident @ 1200 PM for up to 2 hrs Queens County Issued By: NYS - DOT Affected Jurisdictions: Nassau, Queens Counties
25. Correlation with NYCDOT Alerts Grand Central Pkwy W/B approaching Jewel Ave blocking all lanes due to an accident @ 1200 PM for up to 2 hrs Queens County Issued By: NYS - DOT Affected Jurisdictions: Nassau, Queens Counties
26. HTM Engine Traffic Demo Architecture MySQL Rabbit MQ anomaly_service metric_listener metric_storer model_scheduler HTM Engine HTTP API Client App Traffic Data Pusher Traffic Data Service Data Server (web) https://github.com/oxtopus/skeleton-htmengine-app https://github.com/rhyolight/htmengine-traffic-tutorial
27. HTM Engine vs NuPIC Capabilities Setup Difficulty Scales Requires NuPIC anomaly detection prediction classification Hard manually Numpy HTM Engine anomaly detection Easy automatically MySQL RabbitMQ NuPIC Supervisord
28. In Closing • This is one layer of one region of cortical cells • We are working on other layers for sensorimotor integration and feedback • This is just the beginning
29. http://numenta.org matt@numenta.org @rhyolight @numenta Contact