"Блеск и нищета SAST", Андрей Ковалев, Яндекс

Transcript

1. None

2. Andrey Kovalev, Security Engineer Splendors an Miseries of the SAST

3. Agenda 1 SAST at Yandex 2 Appropriate cases and true

   positive examples 3 Inappropriate cases and false positive examples 4 Statistics and summary

4. SAST at Yandex Splendors an Miseries of the SAST

5. SAST as a process — Yandex approach 5 Dashboards SAST

   Code analysis Configuration check Dependency inspection Search of secrets Project tracker Data storage Product team

6. Why SAST? 6 At Yandex: › Solve SAST-specific cases: dependency

   inspection or search of secrets › Perform analysis for platforms which don’t have good DAST tools (for example on Windows) › Protect the products with poor fuzzing’s or unit-test’s coverage

7. Why SAST? 7 In other cases › Perform regular analysis

   without powerful testing infrastructure › One-time security check without writing additional fuzz-tests › Security assessment without writing additional code (ideal case)

8. Code analyzers we use 8 › For C\C++ : Clang

   Static Analyzer, Coverity › For Java: Findbugs and Find security bugs › JavaScript: Coverity › For Python: Bandit

9. Main search targets for code static analysis 9 For web

   applications: › Parsing dangerous serialization data formats › Insecure platform API or technology usage › Unsafe processing of data in SQL queries › Improper encoding or escaping of output

10. Main search targets for code static analysis 10 For binary

    applications: › Race conditions: time-of-check-time-of-use problem, unsafe thread access problem (Coverity) › Memory corruptions: out of bounds, use after free, type confusion issues

11. Appropriate and true positive examples Brilliance and poverty of the

    SAST

12. Coverity detects simple memory corruptions 12 Char array overrun 


    
 
 


13. Bandit warns about potentially XXE 13 Unsafe XML Parsing 


    
 
 


14. Inappropriate and false positive examples Brilliance and poverty of the

    SAST

15. Bandit warns about dangerous serialization 15 Is the usage of

    the Pickle always bad? It depends… 
 
 
 


16. Output escaping false positive 16 Coverity doesn’t recognize custom escaping

    implementation 
 
 
 


17. Output escaping false positive 17 Coverity doesn’t understand shared pointer

    semantics 
 
 
 


18. Statistics and summary Splendors an Miseries of the SAST

19. Statistics §for vulnerabilities 19 0 7,5 15 22,5 30 Coverity

    Bandit FindSecBugs High Medium

20. Statistics for true positives / false positives 20 0 10

    20 30 40 Coverity Bandit FindSecBugs True False

21. Where is SAST good enough? 21 Good precision (70%+ ):

    › Dangerous format data parsing: yaml, lxml, etc › Insecure API usage: shell commands, insecure randoms, debug modes, etc. › SQL injections › Unitialized variables and members for C++ Average precision (55% - 70%) › Improper encoding or escaping of output

22. Where is SAST have lot’s of false positives? 22 Awful

    precision (20%): › Memory corruption in modern C++ applications › Data leakage — wrapper escaping

23. Implementation best practice 23 › Analyze only relevant code: exclude

    tests and third-party › Tune checking rules, switch off false positive checkers › Divide analysis output in two parts: one (with tuned proven checker) for developers, second — for security team › Get feedback from developers: try to find enthusiasts (or security champions) and ask them to help to classify results › Don’t create tickets for developers if issue is not critical and checker is not proven

24. Our point of view (general conclusions) 24 › Some of

    SAST checkers (Bandit) are regexp-based analyzers of AST › Some analyzers (Coverity, FindSecBugs) try to do taint analysis, but it’s not very good - they still can’t recognize user-controlled input › In general, SAST is good enough for insecure API usage, dangerous format parsing, input validation etc., but only for standard cases › For memory corruptions in modern applications — use DAST (sanity testing, fuzzing), these technics are more effective

25. Andrey Kovalev Security engineer @L1kvID avkov@yandex-team.ru Thank you! Questions?