Authors: Becca Lynch and Stefano Meschiari, presented during DEFCON AI Village 29
Typical machine learning models in the security space use labels (annotations that describe whether a certain action is benign or malicious) in order to learn how to discriminate between threats and normal activity.
In practice, however, many systems in the security space that would benefit from machine learning models are critically hampered by a scarcity of labels. This may be due to many factors, such as low coverage of the collected labels, long latency between threat events and receiving the corresponding label, and noise in the feedback from domain experts and the system's users. New systems may have to be bootstrapped in the complete absence of established historical data (cold starts). Human behavior, being intrinsically difficult to quantifiably predict, often leaves us with benign activity that is constantly shifting and attack techniques that are constantly improving.
In this talk, we will discuss how we addressed the issues stemming from this complex ecosystem in the detection of two-factor authentication anomalies. We will be describing some of the algorithms, heuristics, and systems we developed to both understand user behavior and detect attack vectors, as well as discussing the many ways in which we fail miserably (and -- sometimes -- enjoy small successes).