Measuring Security Uncertainty

Using confusion matrices to provide figures of merit for security solutions, given improved measurements of performance matched to needs at each stage of the development to production lifecycle.

Transcript

1. MEASURING SECURIT Y UNCERTAINT Y C O N F U S I O N M AT R I C E S A S F I G U R E S O F M E R I T F O R S E C U R I T Y S O L U T I O N S
2. JAMES BORE Tyred of all this Security consultant @coffee_fueled https://linkedin.com/in/jbore james@coffeefueled.org
3. FIGURES OF MERIT (FOM) R E D U C I N G T H E Q U A L I T Y O F S O L U T I O N S D OW N TO S I M P L E N U M B E R S
4. WHERE IS A FOM USEFUL? Anywhere we want to understand what risk we are exposed to by acting or not acting on outputs of a system. • AV • DAST • DLP • EDR • IDS/IPS • MSSP • SAST • SIEM • SOC • TI • UBA • WAF
5. S E C U R I T Y T E S T I N G L I F E C Y C L E J U S T A Q U I C K R E F R E S H E R , I P R O M I S E
6. LIFECYCLE STAGES DEFINITION DESIGN DEVELOPMENT DEPLOYMENT MAINTENANCE
7. FOREACH STAGE PLAN DO CHECK ACT
8. T R U T H , B E A U T Y, A N D T E S T S T RY I N G TO G E T T H E T E R M I N O L O G Y R I G H T
9. WHAT IS TRUTH? Not a philosophical issue What is true is what is true regardless of our tests, or at least what we say is true We can only take a sample of ‘truth’
10. TESTS, POSITIVES AND NEGATIVES Our tests will say that a vulnerability does or does not exist. Where our test claims there’s a vulnerability, this is a positive. Anything which is not a positive given the way our tests work, is assumed to test negative. Where our test reflects the underlying truth, this is a true positive or negative.
11. NO CONFUSION AT ALL To keep it simple, this isn’t confusing at all. A test will give either a positive or negative for a finding that is or is not true. Bear with me.
12. TRUTH POSITIVE NEGATIVE TEST POSITIVE TRUE POSITIVE A finding from our test (e.g. SQL injection, which can be actively exploited) FALSE POSITIVE A finding from our test (e.g. cross-site scripting, which does not exist and is not exploitable.) NEGATIVE FALSE NEGATIVE An exploitable vulnerability which is not found by our test (e.g. unpatched software not detected by monitoring tools) TRUE NEGATIVE Anything which is not exploitable, and is not detected by our tests.
13. WHAT IS A CONFUSION MATRIX? O T H E R T H A N H I G H LY C O N F U S I N G W I T H L I M I T E D S TA N D A R D I S E D T E R M I N O L O G Y
14. ASSUMPTIONS AND LIMITS Ignore uncertainty Our test sample is representative
15. ACCURACY + +
16. ERROR RATE Also known as misclassification. + +
17. TRUE POSITIVE RATE (TPR) Also known as sensitivity, recall rate or hit rate. +
18. TRUE NEGATIVE RATE (TNR) Also known as selectivity or specificity. +
19. POSITIVE PREDICTIVE VALUE (PPV) Also known as precision +
20. NEGATIVE PREDICTIVE VALUE (NPV) +
21. WHICH MEASURES MATTER? At different lifecycle stages, different aspects matter more. When remediation costs are highest, we care most about false positives. When remediation costs are lowest, we care most about false negatives. When determining ground truth is difficult, we care less about truth. When determining ground truth is easy, we care more about truth.
22. WORKED EXAMPLE C A L C U L AT I N G T H E D I F F E R E N T M E T R I C S
23. SAMPLE REPORT Hypothetical sample system with known findings. Any additional findings can be examined and determined as false or true through testing. Different ‘sample’ systems are applicable at each lifecycle stage, but we can calculate FoMs in the same way. With vulnerability findings, we usually will not have true negatives as testing outcomes. False negatives include any positives not identified. Actual Positive Actual Negative Test Positive 62 38 Test Negative 14 0
24. Actual Positive Actual Negative Test Positive 62 38 PPV: 0.62 Test Negative 14 0 NPV: 0.00 TPR: 0.82 TNR: 0.00
25. USE CASES W O R K E D E X A M P L E S F O R D I F F E R E N T S TA G E S
26. DEFINITION At this stage: • Remediation costs are low • Identifying true positives is expensive So: • False positives are cheap • False negatives may be caught in later testing We want: • High sensitivity over all
27. DESIGN At this stage: • Remediation costs are low • Identifying true positives is expensive So: • False positives are cheap • False negatives may be caught in later testing We want: • High sensitivity over all
28. DEVELOPMENT At this stage: • Remediation costs are middling • Identifying true positives is middling So: • False positives require effort and create backlog • False negatives may be caught in later testing We want: • Balance of sensitivity and overall accuracy
29. DEPLOYMENT At this stage: • Remediation costs are high • Identifying true positives is easy So: • False positives are expensive • False negatives create unknown residual risk We want: • Negative and positive predictive value to be high
30. MAINTENANCE At this stage: • Identifying true positives is easy, but undesirable • Inappropriate responses can have significant impact So: • False positives are potentially disastrous • False negatives mean an extended breach We want: • As few false positives and negatives as possible
31. RECAP 1.Security assessment and monitoring can be measured. 2.Measurement does not require a lot of work. 3.Which measure is important varies with lifecycle. If you’ve enjoyed this, donate to TMHC Isolation Con fundraiser for MSF! and get in touch @coffee_fueled https://linkedin.com/in/jbore james@coffeefueled.org