Building Offensive Web Security Framework in Python

A draft version of slides for a proposed talk at PyCon India 2015

Transcript

1. Building Offensive Web Security Framework in Python Bharadwaj `tunnelshade` Machiraju
2. subprocess.check_output(“whoami”).decode(“utf-8”) • Senior @ IIT (BHU) Varanasi • Application Security Engineer @Yodlee • Project Leader of OWASP OWTF (along with Abraham Aranguren) • Loves writing security tools • Blogs @ blog.tunnelshade.in
3. import Agenda; print(Agenda.pycon_in) ▪ Introduction to OWASP and OWTF ▪ Do we really need another web security tool? ▪ Basic architecture & dependencies ▪ Test categorization and implementation ▪ Some interesting modules & their implementations – HTTP(S) MiTM proxy – Snooping on what all the tools are doing – Ajax web crawler – Crawling dynamic sites for better idea of attack surface – Botnet Mode – Attacking targets with botnet style – Pentester’sTools Parser (PTP) – A python library to parse tool outputs ▪ Extending with own tests and integration ▪ Pre-requisites to know if planning on building a similar one
4. Open Web Application Security Project • OWASP is an open community dedicated to enabling organizations to conceive, develop, acquire, operate, and maintain applications that can be trusted. • All of the OWASP tools, documents, forums, and chapters are free and open to anyone interested in improving application security. • OWASP advocates approaching application security as a people, process, and technology problem because the most effective approaches to application security include improvements in all of these areas. • Popular projects include OWASP Top 10 & OWASP ZAP • www.owasp.org.
5. Openhub Stats!
6. Offensive Web Testing Framework • OWTF!! • A project aimed to penetration testing efficiency and alignment of security tests to standards like OWASP v4, NIST etc.. • Core framework is written in python2. Openhub stats below!
7. Is OWTF really needed? • There are many good security tools out there, but none of them serve all our purposes. It is practically impossible to create one tool for everything. • So Abraham decided to write a framework which let you run all those tools along with your custom tests. • The main goal was to present all the output in an organised fashion. • Some similar frameworks are Golismero Project (Open Source) & Faraday IDE (Community, Pro & Corp versions)
8. Basic OWTF Architecture Interface & API Server Proxy Server Worker Worker Database Transaction Logger (P) HTTP TRAFFIC API & WEB UI
9. CORE DB Handler Plugin Helper File Handler Config Handler Plugin Handler Developer Perspective HTTP Requester
10. Python libraries used! • Tornado • pycurl • SQLAlchemy • Selenium • BeautifulSoup4 • lxml • markdown • pexpect • psycopg2 • pyOpenSSL • python –owasp-zap-v2 • rdflib • PyVirtualDisplay
11. Well most developers are careful about XSS, SQLi etc.. But does OWTF check for all these??
12. Extensibility in OWTF • Tests are classified into three main categories • WEB • Active • Passive • Semi-Passive • Grep • External • NET • AUX
13. So what we do? • Provide Core object to the plugins, so they have all the features of the framework available. • Import the plugin source using python “imp” module to get the plugin code. • A sample plugin
14. Modules!!!
15. HTTP(S) Proxy • Built in Tornado! Why?? Because our pre-implementation research said so! • We wanted to build a really fast proxy but at the same time, not implement all the request parsing code. So, decided to use tornado Application instance. • Few of our crazy requirements were • Caching • SSL MiTM • Serving HTTP, HTTPS, WS & WSS on the same port!! • Change the outbound proxy randomly for different requests • Lets dig a bit into internals of this module!
16. File System as Cache Instance 1 Instance 2 Instance 3 Instance 4 File System When writing to the file system, a file lock is used and this prevents race conditions. A shared memory for all the instances also exists which is mostly read only.
18. SSL MiTM Web Server 2 Web Server 1 Proxy 8OO8 Worker X HTTP Traffic Worker Y SSL Session 1 SSL Session 2 Cert. Store (File System) Self-Signed CA
19. Supporting HTTP & WebSocket • Tornado has “RequestHandler” and “WebSocketHandler” classes and instances of these are used to handle http requests and websockets accordingly. • So, when a request arrives at tornado, it will create one of the instances according to the request type and call certain methods of that instance. • Since we are using the application aspect of tornado to build our proxy there was a small problem. • As a proxy we never know on which path you will get a websocket request, so we had the requirement of changing the Handler class once tornado calls for a new object. There is one popular way :P
21. Performance Improvements