Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Offensive Web Security Framework in Python

Building Offensive Web Security Framework in Python

A draft version of slides for a proposed talk at PyCon India 2015


Bharadwaj Machiraju

July 14, 2015


  1. Building Offensive Web Security Framework in Python Bharadwaj `tunnelshade` Machiraju

  2. subprocess.check_output(“whoami”).decode(“utf-8”) • Senior @ IIT (BHU) Varanasi • Application Security

    Engineer @Yodlee • Project Leader of OWASP OWTF (along with Abraham Aranguren) • Loves writing security tools • Blogs @ blog.tunnelshade.in
  3. import Agenda; print(Agenda.pycon_in) ▪ Introduction to OWASP and OWTF ▪

    Do we really need another web security tool? ▪ Basic architecture & dependencies ▪ Test categorization and implementation ▪ Some interesting modules & their implementations – HTTP(S) MiTM proxy – Snooping on what all the tools are doing – Ajax web crawler – Crawling dynamic sites for better idea of attack surface – Botnet Mode – Attacking targets with botnet style – Pentester’sTools Parser (PTP) – A python library to parse tool outputs ▪ Extending with own tests and integration ▪ Pre-requisites to know if planning on building a similar one
  4. Open Web Application Security Project • OWASP is an open

    community dedicated to enabling organizations to conceive, develop, acquire, operate, and maintain applications that can be trusted. • All of the OWASP tools, documents, forums, and chapters are free and open to anyone interested in improving application security. • OWASP advocates approaching application security as a people, process, and technology problem because the most effective approaches to application security include improvements in all of these areas. • Popular projects include OWASP Top 10 & OWASP ZAP • www.owasp.org.
  5. Openhub Stats!

  6. Offensive Web Testing Framework • OWTF!! • A project aimed

    to penetration testing efficiency and alignment of security tests to standards like OWASP v4, NIST etc.. • Core framework is written in python2. Openhub stats below!
  7. Is OWTF really needed? • There are many good security

    tools out there, but none of them serve all our purposes. It is practically impossible to create one tool for everything. • So Abraham decided to write a framework which let you run all those tools along with your custom tests. • The main goal was to present all the output in an organised fashion. • Some similar frameworks are Golismero Project (Open Source) & Faraday IDE (Community, Pro & Corp versions)
  8. Basic OWTF Architecture Interface & API Server Proxy Server Worker

    Worker Database Transaction Logger (P) HTTP TRAFFIC API & WEB UI
  9. CORE DB Handler Plugin Helper File Handler Config Handler Plugin

    Handler Developer Perspective HTTP Requester
  10. Python libraries used! • Tornado • pycurl • SQLAlchemy •

    Selenium • BeautifulSoup4 • lxml • markdown • pexpect • psycopg2 • pyOpenSSL • python –owasp-zap-v2 • rdflib • PyVirtualDisplay
  11. Well most developers are careful about XSS, SQLi etc.. But

    does OWTF check for all these??
  12. Extensibility in OWTF • Tests are classified into three main

    categories • WEB • Active • Passive • Semi-Passive • Grep • External • NET • AUX
  13. So what we do? • Provide Core object to the

    plugins, so they have all the features of the framework available. • Import the plugin source using python “imp” module to get the plugin code. • A sample plugin
  14. Modules!!!

  15. HTTP(S) Proxy • Built in Tornado! Why?? Because our pre-implementation

    research said so! • We wanted to build a really fast proxy but at the same time, not implement all the request parsing code. So, decided to use tornado Application instance. • Few of our crazy requirements were • Caching • SSL MiTM • Serving HTTP, HTTPS, WS & WSS on the same port!! • Change the outbound proxy randomly for different requests • Lets dig a bit into internals of this module!
  16. File System as Cache Instance 1 Instance 2 Instance 3

    Instance 4 File System When writing to the file system, a file lock is used and this prevents race conditions. A shared memory for all the instances also exists which is mostly read only.
  17. None
  18. SSL MiTM Web Server 2 Web Server 1 Proxy 8OO8

    Worker X HTTP Traffic Worker Y SSL Session 1 SSL Session 2 Cert. Store (File System) Self-Signed CA
  19. Supporting HTTP & WebSocket • Tornado has “RequestHandler” and “WebSocketHandler”

    classes and instances of these are used to handle http requests and websockets accordingly. • So, when a request arrives at tornado, it will create one of the instances according to the request type and call certain methods of that instance. • Since we are using the application aspect of tornado to build our proxy there was a small problem. • As a proxy we never know on which path you will get a websocket request, so we had the requirement of changing the Handler class once tornado calls for a new object. There is one popular way :P
  20. None
  21. Performance Improvements