anomaly detection with python

Automatically detecting anomalies and their causes in business-metric time-series. A fairly simple and configurable anomaly detection method that adjusts quickly to changing distributions.

Transcript

1. Automatically detecting anomalies and their causes in business-metric time-series. by Rosie Bloxsom
2. Hi, I’m Rosie Bloxsom
3. Who am I? Employed at Lyst Fashion startup heavily into data Data Analyst On the data-science end not the BI end Background Mathematics, fraud detection, software engineering
4. We collect the world of
 fashion into a customisable
 shopping experience.
5. What is Lyst? We attract web traffic … so we care about web data of people who want to buy clothes … so we care about customer origin data and show them clothes they can buy … so we care about customer browsing habits and we get commission. so we care about order data
6. People spend a lot of time watching the data. It’s hard to manually spot when something has changed enough to care about. If something has changed, it’s hard to identify why. The problem
7. Implement a system which watches for unexpected changes. When a change occurs, offer likely explanations for the change so people can investigate. The solution
8. Easy to understand, quick to implement, not spammy. The constraints
9. The data Business metrics Number of orders Total value of orders Average value of order Number of web visits Order rate of web visits Bounce rate Business metrics we want to detect anomalies in. Origins of change Country Device type (e.g. mobile) Web landing page Web traffic source Retailer
10. One retailer forgets to send us order data. The number of orders that we make is lower than expected. Checkout is broken for mobile web. The number of orders that we make is lower than expected. One retailer is having a sale. The average value of an order falls. It’s sale season in a particular country. The average value of an order falls. A common landing point of our website is broken. Bounce rate increases. Examples
11. So… to work.
12. An anomaly is a point which deviates from our expectation by a significant margin. What is an anomaly?
13. So we can totally just use mean +/- 2*std, right?
14. Wrong Wrong.
15. Things change over time.
16. Recent data should carry more weight.
17. Other things you should know. The data sometimes breaks so the algorithm must be robust to slightly dodgy history. Explainability is important so we can’t use super-complicated models. We only care if the current value is anomalous so we don’t need a historic report on all anomalies.
18. An anomaly is a point which deviates from our expectation by a significant margin. Our expectation should be more dependent on the recent past than the whole history. Our definition of significant should be more dependent on the recent past than the whole history. So what is an anomaly again?
19. Predict the future by exponentially weighted mean of the past. This takes all the past into account, but weights the most recent past as more predictive. This is called Holt method, or sometimes Holt-Winters method (definitions vary). Choose alpha between 0 and 1. Lower values of alpha adapt to changes slower, so lead to more stable predictions, but don’t adapt so quickly to genuine change. Idea
20. Measure the exponentially weighted mean-squared-error of previous predictions from the actual values. This gives an expected range of current deviation from predicted value. Measure prediction error
21. It works !!!
22. Or does it?
23. Not great.
25. Incorporate slope Beta is between 0 and 1
26. Fixed.
27. Parameter choosing alpha - value stickiness How quickly you forget past values beta - slope stickiness How quickly you forget past slope standard deviation window How much deviation you allow standard deviation half-life How quickly standard deviation measure adjusts to change
28. Great, so we can detect anomalies. Now we need to find the causes.
29. The data Business metrics Number of orders Total value of orders Average value of order Number of web visits Order rate of web visits Bounce rate Business metrics we want to detect anomalies in. Origins of change Country Device type (e.g. mobile) Web landing page Web traffic source Retailer
30. One retailer forgets to send us order data. The number of orders that we make is lower than expected. Checkout is broken for mobile web. The number of orders that we make is lower than expected. One retailer is having a sale. The average value of an order falls. It’s sale season in a particular country. The average value of an order falls. A common landing point of our website is broken. Bounce rate increases. Examples
32. What might affect AOV? Country Which country was the order placed from. Device type What did the customer use? Desktop / mobile …? Traffic source How did we attract the customer to the website? Retailer Which retailer’s products were bought?
33. Let’s look at country US UK DE Change explained Fix the other countries at their original data. Allow one country to change to the new data. How much of the total change has still happened? Is this enough?
34. Let’s look at country Population US UK DE Change explained US UK DE
35. Are we surprised? % change explained % population covered “level of surprise” =
36. Let’s look at country 1 1 1 Population US UK DE Change explained US UK DE Surprise:
37. Let’s look at retailer 1.5 Population R1 R2 R3 Change explained R1 R2 R3 Surprise: 0.8 0.5
38. Retailer’s top change explainer is more surprising. Retailer is the more likely cause. 1.5 R1 R2 R3 Change explained Surprise: 0.8 0.5 1 1 1 US UK DE Change explained Surprise:
39. The business analyst is informed that there was an unusual drop in AOV, and the cause is likely retailer-driven (by R1). Useful stats are handed over (all calculated as described earlier). The business analyst can begin a useful investigation. The outcome
40. psycopg2 All the data is stored in a Redshift DB pandas Minor data manipulation and statistics numpy Some maths cron jobs Code runs and sends alerts automatically. **Mandatory tech stack slide:
41. thank you