Anomaly Detection by Mean and Standard Deviation

Transcript

1. Anomaly Detection iwanaga
2. Who am I @quake_alert @quake_alert_en @quake_alert_fr @quake_alert_kr Yoshihiro Iwanaga
3. Motivation for detecting anomaly Traditional system monitoring •  process existence •  ping, http, tcp response •  disk usage → “fixed” rule / threshold
4. Motivation for detecting anomaly Notice something out of ordinary •  network traffic is heavier than usual •  number of login try is obviously larger •  a colleague is strangely gracious today → Unusual behaviors; Indications of fault. Such info helps preventing service degrading in advance!! but rule/threshold vary with service, host, client, time…
5. key to detect anomaly usual unusual Watch differences b/w
6. e.g. Network Traffic Mon Tue Wed Thu Fri traffic time
7. Superimpose 24 hour plot Traffic at 15:00 on workday is about 1.2 Gbps traffic time Periodicity!!
8. mean mean － 3σ mean + 3σ amount of dispersion from mean = 1 N v u u t N X i=1 (¯ x xi)2 Acceptable “range” → e.g. Acceptable range of traffic at 15:00 on workday is 1.01 to 1.38 Gbps
9. Case examples
10. DDoS partial hardware failure Traffic
11. number of mail passed spam filter spam rate e-mail Applied a wrong spam rule
12. However Reality is not that simple… 人生楽ありゃ苦もあるさ 涙の後には虹も出る 歩いてゆくんだしっかりと 自分の道をふみしめて 山上路夫
13. downloading large files mass e-mail sending “Traffic spike” happens so frequently Frequent false-positive alerting will be “cry-wolf” system…
14. heuristic filtering In usual, traffic gets cool down within 15 minutes notify engineers if anomaly continues more than 15 minutes Engineers’ knowledge is gold mine for better algorithm J → one practical example: