Semi-Supervised Anomaly Detection

Presentation by Eric Kramer. data scientists @dataiku from data science track at #StrataHadoop London 2016

Transcript

1. Semi-Supervised Anomaly Detection Use cases, theory and hands-on
2. Dataiku The Numbers x 50 + x 1 + +  48
3. Me The Numbers 1/2 1/2 + = ! > df = data_scientists %>% inner_join(r_users) %>% filter(speaker_at == “meetup”) > df$name [1] “Eric Kramer”
4. Agenda 30 minutes Introduction Definition Use Cases Formalization Building an anomaly detector in R Theory R code Result Summary Improvements Concerns Questions } Iterate 3 times, increasingly complexity each time
5. Code Everything is on github https://github.com/erickramer/anomaly_detection_demo
6. Introduction
7. What is an anomaly? Global anomalies are unexpected based on the entire dataset Local anomalies are unexpected given their context Anomalies are “unexpected” observations
8. Use Case: Bank Fraud
9. Use Case: EEG Is this an anomaly?
10. Key Principles Anomalies are rare Anomalies have little in common with each other
11. Formalization P ( y | x ) < ⇢ An observation is an anomaly if where y x ⇢ represents the observation represents the context is some arbitrary threshold
12. Formalization P ( y | x ) < ⇢ Observation Context
13. Use Case: Anomalous Weather P ( y | x ) < ⇢ Current weather: • Temperature • Humidity • Pressure Current Location: • Lat, Long • Altitude • Date • Time
14. Use case: Bank Fraud P ( y | x ) < ⇢ Financial transaction: • Origin • Destination • Amount • Medium Account history: • Past transactions • Account address • Account balance • Account flux
15. Use case: EEG P ( y | x ) < ⇢ EEG reading Patient History: • Diagnoses • Interventions / Surgeries • Current medications
16. Questions P ( y | x ) < ⇢ How do I choose ? P ( y | x ) < ⇢ Find your tolerance for false positives e.g. 100 false positives is equivalent to stopping one fraud How do I calculate ? Density Estimation P ( y | x ) < ⇢
17. Density Estimation We’re going to use Gaussian Mixture Models. Alternatives: • Kernel density estimators • Histograms • K-means • Bayesian methods
18. Our data Weather in Paris, London and NYC for 2010-2015
19. Goals Weather in Paris, London and NYC for 2010-2015 Can we find days with anomalousweather? Can we controlfor the location of the measurement? Can we controlfor the date?
20. Gaussian Mixture Models
21. Theory Gaussian Mixture Models The probability density is the sum of a small number of Gaussian distributions = +
22. Theory Gaussian Mixture Models The probability density is the sum of a small number of Gaussian distributions Number of Gaussian distributions to use Gaussian distribution Mean of ith Gaussian distribution P(y) = 1 n n X i=1 N(µi, 2 i ) Variance of ith Gaussian distribution
23. Questions Gaussian Mixture Models P(y) = 1 n n X i=1 N(µi, 2 i ) How do I find and ? Maximum likelihood optimization P(y) = 1 n n X i=1 N(µi, 2 i ) P(y) = 1 n n X i=1 N(µi, 2 i ) How do I choose ? Fit models for several and choose one with best BIC P(y) = 1 n n X i=1 N(µi, 2 i )
24. Our first model
25. Fitting a GMM Using the mclust package library(mclust) load(”./data/weather_data.Rdata”) gmm = Mclust(df[c(“temperature”, “humidity”], G=seq(1,6)) plot(gmm, what=“classification”) Try anywhere from 1 to 6 Gaussians in the mixture Just two dimensions for now Load package and data
26. Fitting a GMM Using the mclust package
27. Fitting a GMM Using the mclust package
28. Getting a density from a GMM Using the mclust package Mclust(…) => densityMclust(…)
29. What are the most anomalous days? Using the mclust package library(dplyr) df %>% mutate(score = gmm$density) %>% arrange(score) %>% head(3) City Temperature Humidity New  York 5 20 New  York -­‐12 39 New  York 29 27
30. Choosing a Threshold Using the mclust package
31. What about controlling for the location of measurement?
32. Controlling for City
33. P(temperature, humidity|city) Option 1: Option 2: Train one GMM for each city Regress temperature and humidity on city Build GMM on residuals of model
34. Fitting multiple GMMs train_gmm = function(df){ densityMclust(df[c(“temperature”, “humidity”)], G=seq(1,6)) } gmms = df %>% nest(-city) %>% mutate(gmm = map(data, train_gmm)) map(gmms$gmm, plot, what=“density”) } Wrap training in a function } Train one GMM per city
35. Raw Data Weather in NYC is much more variable London NYC Paris
36. Multiple GMMs Weather in NYC is much more variable London NYC Paris
37. What are the most anomalous days? Controlling for location gmms %>% mutate(score = map(gmm, "density")) %>% select(-gmm) %>% unnest() %>% arrange(score) %>% head(3) City Temperature Humidity Paris -­‐6 51 Paris -­‐6 51 London 29 40
38. Choosing a Threshold Using the mclust package
39. Increasing the Dimensionality
40. Fitting multiple GMMs train_gmm = function(df){ densityMclust(df[c("temperature", "humidity", "visibility", "wind_speed")], G=seq(1,6)) } gmms = df %>% nest(-city) %>% mutate(gmm = map(data, train_gmm)) map(gmms$gmm, plot, what=“density”) } Include four columns this time
41. High-dimensional densities Visualizing more than 2 dimensions
42. High-dimensional densities Visualizing more than 2 dimensions London NYC Paris
43. High-dimensional densities Paris
44. High-dimensional densities London
45. High-dimensional densities New York
46. What are the most anomalous days? Controlling for location gmms %>% mutate(score = map(gmm, "density")) %>% select(-gmm) %>% unnest() %>% arrange(score) %>% head(3) City Temperature Humidity Wind  Speed Pressure New York 23 49 159 1015 London 29 40 16 1012 New  York 14 80 30 980 Hurricane winds Insanely hot for London Crazy low pressure
47. Choosing a Threshold Using the mclust package
48. Summary
49. Summary • Semi-supervised learning attemps to learn the distribution of data. Anomalies are then low-probability observations • GMMs provide a quick-and-easy way to estimate probability densities • Mclust is an awesome GMM package for R • Anomalies highly dependent on context (i.e. city) and the variables included in detector (i.e. temperature, humidity, wind speed, pressure). Semi-supervised learning
50. Moving to Production • Train GMM on “normal” data, updated weekly • REST API scores incoming data based on last weeks data • Threshold chosen based on false positive tolerance Semi-supervised learning