Building Offensive Web Security Framework in Python

Building Offensive Web Security Framework in Python

A draft version of slides for a proposed talk at PyCon India 2015


Bharadwaj Machiraju

July 14, 2015


  1. 2.

    subprocess.check_output(“whoami”).decode(“utf-8”) • Senior @ IIT (BHU) Varanasi • Application Security

    Engineer @Yodlee • Project Leader of OWASP OWTF (along with Abraham Aranguren) • Loves writing security tools • Blogs @
  2. 3.

    import Agenda; print(Agenda.pycon_in) ▪ Introduction to OWASP and OWTF ▪

    Do we really need another web security tool? ▪ Basic architecture & dependencies ▪ Test categorization and implementation ▪ Some interesting modules & their implementations – HTTP(S) MiTM proxy – Snooping on what all the tools are doing – Ajax web crawler – Crawling dynamic sites for better idea of attack surface – Botnet Mode – Attacking targets with botnet style – Pentester’sTools Parser (PTP) – A python library to parse tool outputs ▪ Extending with own tests and integration ▪ Pre-requisites to know if planning on building a similar one
  3. 4.

    Open Web Application Security Project • OWASP is an open

    community dedicated to enabling organizations to conceive, develop, acquire, operate, and maintain applications that can be trusted. • All of the OWASP tools, documents, forums, and chapters are free and open to anyone interested in improving application security. • OWASP advocates approaching application security as a people, process, and technology problem because the most effective approaches to application security include improvements in all of these areas. • Popular projects include OWASP Top 10 & OWASP ZAP •
  4. 6.

    Offensive Web Testing Framework • OWTF!! • A project aimed

    to penetration testing efficiency and alignment of security tests to standards like OWASP v4, NIST etc.. • Core framework is written in python2. Openhub stats below!
  5. 7.

    Is OWTF really needed? • There are many good security

    tools out there, but none of them serve all our purposes. It is practically impossible to create one tool for everything. • So Abraham decided to write a framework which let you run all those tools along with your custom tests. • The main goal was to present all the output in an organised fashion. • Some similar frameworks are Golismero Project (Open Source) & Faraday IDE (Community, Pro & Corp versions)
  6. 8.

    Basic OWTF Architecture Interface & API Server Proxy Server Worker

    Worker Database Transaction Logger (P) HTTP TRAFFIC API & WEB UI
  7. 9.

    CORE DB Handler Plugin Helper File Handler Config Handler Plugin

    Handler Developer Perspective HTTP Requester
  8. 10.

    Python libraries used! • Tornado • pycurl • SQLAlchemy •

    Selenium • BeautifulSoup4 • lxml • markdown • pexpect • psycopg2 • pyOpenSSL • python –owasp-zap-v2 • rdflib • PyVirtualDisplay
  9. 12.

    Extensibility in OWTF • Tests are classified into three main

    categories • WEB • Active • Passive • Semi-Passive • Grep • External • NET • AUX
  10. 13.

    So what we do? • Provide Core object to the

    plugins, so they have all the features of the framework available. • Import the plugin source using python “imp” module to get the plugin code. • A sample plugin
  11. 15.

    HTTP(S) Proxy • Built in Tornado! Why?? Because our pre-implementation

    research said so! • We wanted to build a really fast proxy but at the same time, not implement all the request parsing code. So, decided to use tornado Application instance. • Few of our crazy requirements were • Caching • SSL MiTM • Serving HTTP, HTTPS, WS & WSS on the same port!! • Change the outbound proxy randomly for different requests • Lets dig a bit into internals of this module!
  12. 16.

    File System as Cache Instance 1 Instance 2 Instance 3

    Instance 4 File System When writing to the file system, a file lock is used and this prevents race conditions. A shared memory for all the instances also exists which is mostly read only.
  13. 17.
  14. 18.

    SSL MiTM Web Server 2 Web Server 1 Proxy 8OO8

    Worker X HTTP Traffic Worker Y SSL Session 1 SSL Session 2 Cert. Store (File System) Self-Signed CA
  15. 19.

    Supporting HTTP & WebSocket • Tornado has “RequestHandler” and “WebSocketHandler”

    classes and instances of these are used to handle http requests and websockets accordingly. • So, when a request arrives at tornado, it will create one of the instances according to the request type and call certain methods of that instance. • Since we are using the application aspect of tornado to build our proxy there was a small problem. • As a proxy we never know on which path you will get a websocket request, so we had the requirement of changing the Handler class once tornado calls for a new object. There is one popular way :P
  16. 20.