The Long Con of Application Scanning - Security B Sides ATL

Presented on October 8th 2010

Transcript

1. The Long Con of
   Automated Web
   Application Security
   Testing
   Erik Peterson / erik@silvexis.com / @silvexis
   ATL
   

   View Slide

2. Who is Erik Peterson
   erik@silvexis.com
   @silvexis (twitter)
   I am a Application Security Evangelist for Veracode with 15 years of application
   and information security industry experience and a contributing member of the
   Cloud Security Alliance.
   I’ve worked for the following organizations:
   Most of past has been spent as a product
   manager for Application & Information Security
   Companies
   silvexis.com (blog)
   

   View Slide

3. A Few Quick Definitions
   • Long Con
   
   ‣ A long con is a confidence trick that takes place over a long period of time
   
   • Dynamic Analysis
   
   ‣ Black box, external automated pen testing performed against a running
   application. Sometimes called Dynamic Application Security Testing or
   DAST
   
   • Static Analysis
   
   ‣ White box, source/binary automated analysis performed against a non-
   running collection of source code or compiled binaries
   

   View Slide

4. What is The Long Con?
   • There are 27 dynamic web
   application security scanning
   products available today1
   
   ‣ 17 are commercial products, the rest open
   source
   
   • Most claim to fully automate the
   process of finding web vulnerabilities
   
   ‣ An educated guess says that all of the Fortune
   500 own or use one of these tools
   
   • Yet, Web Sites are still vulnerable,
   people are still getting hacked, and
   we don’t seem to be any better off
   1. The History of Web Application Scanning Project, http://silvexis.com/research/hwas/
   

   View Slide

5. The Cake is a Lie
   But what about Dynamic Scanning?
   • Just what have we been trying to do for 10
   years?
   
   • Why is it so damn hard?
   
   • Is there a better approach?

   These are todays questions
   

   View Slide

6. 10 years of history in 60 seconds
   Attempts at Dynamic
   Web Scanning
   

   View Slide

7. It was 1999 and the web was young...
   • The first web application scanner
   “Whisker”, created by Rain Forest Puppy
   
   ‣ “CGI Scanner” back when CGI was how it was done
   
   ‣ Focused on known attacks and misconfigurations
   
   ‣ Whisker was discontinued eventually and the core
   features became libwhisker which was used by tools
   like Nikto
   
   ‣ Both of which are still maintained which makes Nikto
   the longest running web application scanner in
   existence
   

   View Slide

8. A year later, the Start of the Commercial Era
   AppScan 1.0
   Released July 25th 2000
   WebInspect 1.0
   Released Feb 1st 2001
   

   View Slide

9. Two dramatically different starting points
   With one common ending
   AppScan v1
   • Focused on ease of use and the
   less experienced user
   
   • Linux Server Hosted Web Proxy,
   Web Based
   
   • No Automated Crawler
   
   • Searched for unknown
   vulnerabilities only
   
   WebInspect v1
   • Focused on the expert user/Pen
   tester
   
   • Windows Desktop Application
   
   • Fully Automated Crawler
   
   • Searched for known
   Both products rapidly converged into
   two very similar products...and got
   acquired by giants
   

   View Slide

10. A lot of other tools arrived since then
    • Acunetx, Cenzic, KavaDo, NT Objectives, Rapid7,
    nCircle, Qualys, Veracode, Whitehat etc..
    
    ‣ They all work more or less the same way
    
    ‣ Crawl something, Audit something, Try to be smart about it
    
    ‣ Some work better than others, but it doesn’t seem to be solving the
    problem
    
    • Veracode and Whitehat have a managed service model,
    which just might be on to something
    
    ‣ Managed service means when the scan fails, it’s their fault, not yours
    
    ‣ Not to be confused with Qualys, which is hosted not managed
    
    ‣ Disclosure: I work for Veracode, so don’t just take my word for it
    

    View Slide

11. So after 10
    years, Why is
    Dynamic Web
    Scanning still
    so Damn Hard?
    

    View Slide

12. A Never Ending Battle
    A scanning tool has to keep up with two things
    
    ‣ The technology platform (e.g. “standards”)
    
    ‣ The attacks and weaknesses
    
    The Web technology platform is moving as fast or faster than the
    attacks and weaknesses
    
    ‣ Other than the web, Is there any other area in security technology where the platform has
    consistently moved faster than the threat landscape?
    
    ‣ This is also why web applications are consistently the source of the majority of the defects1,2
    
    
    Nobody can exactly agree on just what a Web Application is
    
    ‣ It’s a moving target, yesterday it was a collection of web pages, today its multi-headed
    monster
    1
    2
    1. Majority of vulnerabilities found were web vulnerabilities, Veracode State of Software Security Report, Vol 2
    2. 54% of hacking breaches we targeted at web applications, Verizon data breach report, 2010
    3
    

    View Slide

13. False Positives False Negatives
    Pick One
    

    View Slide

14. How Would You Expect an Automated Crawl of
    this Application to Perform?
    

    View Slide

15. And Just What is a Web Application Anyway?
    Web Application
    * P.S. - This is also your new “perimeter”
    

    View Slide

16. Web Standards will save us?
    • The first rule of the Internet is that there are no
    standards
    
    • The second rule of the Internet is that THERE ARE NO
    STANDARDS
    
    • Anything even remotely resembling a standard is only
    loosely enforced
    
    • Internet Explorer is the “compiler for web”

    If it works in IE, ship it
    rfc-3514
    HTML5
    rfc-3093
    AJAX
    JSON
    

    View Slide

17. What about the People and Process part?
    • That’s broken too
    
    ‣ An expert with nothing is still better than a
    novice with a tool, yet we are giving novices
    tools and expecting the same result
    
    ‣ We are too obsessed with trying to control
    change as a means to improve security
    
    ‣ We are not thinking about scalability
    
    ‣ We are too focused on scanning
    

    View Slide

18. Suggestions for a better
    approach End the madness
    

    View Slide

19. Accept that Automated
    Crawling is Dead
    • Really, it’s dead.
    • If you are a vendor, stop offering this as an viable option, you are lying to your
    customers
    
    ‣ Using only an automated crawler is just asking for it, demand that they record the important test cases
    
    • Ultimately, Web security testing will look a lot like desktop application testing
    
    ‣ There is no such thing as a automated crawler for desktop applications
    
    ‣ This requires you must get to know the application in advance
    
    ‣ BTW, The application owner would be much better at this than you
    

    View Slide

20. Stop Scanning
    Support A Continuous Assessment Model
    • Abandon the scan centric mode of thinking
    
    ‣ Unless you are only interested in compliance, ugh.
    
    • This will be your only hope for keeping track of your
    application perimeter
    

    View Slide

21. Don’t expect the web to stand still
    Be Ready to Evolve with the Technology
    A constant feedback loop is required
    

    View Slide

22. Figure out how to get out of the testing business
    Unless you happen to be in the Testing Business
    • You have 3000 applications, your
    small team of 3 will never scale1
    
    • You want to manage the process,
    not be the process
    
    • If you or your application
    development org can’t handle
    security testing consider a
    managed security testing service
    1. Actual size of one Fortune 100 company’s application security team
    

    View Slide

23. Go/No Go Authority is a Myth
    Accept that Change is Constant
    • If your process requires that you
    can control change, you have
    already failed
    
    • Even if you have it, you are hurting
    your business by keeping it
    
    • Set the expectation for continuous
    anytime anywhere assessment
    
    ‣ If you can’t do it, find a company that can
    do it for you
    
    ‣ Hackers don’t schedule their pen tests
    

    View Slide

24. My Ultimate Recommendation
    Give up on Stand Alone Tools
    • They will never scale
    
    • There will always be false
    positives or negatives
    
    • They can’t improve as
    fast as the web evolves
    
    • They are too complex to
    operate consistently
    
    • They are not capable of
    continuous assessment
    

    View Slide

25. Final Thought
    Your Applications are the new Perimeter
    • A perimeter must protect itself, can’t rely on external
    controls
    
    • Must be resilient to a continuous assessment of it’s
    strength
    
    • Rugged Software Manifesto
    
    ‣ http://www.ruggedsoftware.org/
    

    View Slide

26. silvexis.com
    erik@silvexis.com
    @silvexis (twitter)
    ATL Thank You
    BSides ATL!
    

    View Slide